Methods, devices, equipment and media for recommending special literature

Abstract

The present application relates to the field of text processing technology, and proposes a method, device, equipment and medium for recommending special documents, wherein the method includes: obtaining document information under a user preset topic; determining a feature set corresponding to the special document based on the document information; performing dimensionality reduction processing on the feature set to obtain a special document feature set after dimensionality reduction; performing subject community mining on the subject document feature set after dimensionality reduction to determine the subject community mining results; and recommending documents to the user based on the subject community mining results. Through this technical solution, the research direction of the topic can be more accurately represented, avoiding inaccurate statistical information caused by feature representation diversity and poor clustering effect caused by feature divergence, which can not only make users more clear about the document distribution structure, but also allow users to selectively learn recommended documents.

Description

[Technical field]

[0001] The present application relates to the field of text processing technology, and in particular to a method, device, equipment and medium for recommending special documents. [Background technology]

[0002] With more and more learning software, users use learning software to obtain documents, manage documents, take notes, etc. During the learning process, users will manage learning documents and collect them into corresponding document categories or upload local documents to corresponding document categories. The document category name may be a subject, multiple subjects, issuing institutions, issuing authors, publishing units, a long string of simple English letters, numbers, or more abstract representations, etc. The setting of document categories represents that users believe that some documents belong to the same category in a certain dimension, level or granularity, or that they will manage common documents in a certain dimension, level or granularity. Document categories represent the user's interest direction or range. The user's interest direction or range represented by the document category may be a very specific direction or range, or it may be a very large direction or range, which contains many sub-directions or ranges.

[0003] The document category set by the user is called "topic" here.

[0004] The collection and acquisition of special topic literature is an important step in user learning. After setting up the topic, the conventional method of obtaining literature is to search for literature by keywords or obtain special topic recommended literature, and then add the literature to the corresponding topic. The conventional recommendation of special topic literature has the following problems:

[0005] 1) Problems faced in recommending literature based on user topic names: This type of method uses topic names to indicate the literature topics that users are interested in, but the user's topic name may be a relatively abstract representation or an invalid content representation, such as "classic literature", "Baidu Company", "excellent paper", "Graham Neubig 2023", etc., so the topic name may not necessarily represent the user's topic needs.

[0006] 2) Problems faced in recommending documents based on user topics: By analyzing the high-frequency keywords in the document collection under the user topic, recommendations are made based on keywords. However, the diverse representations of keywords (for example, generative artificial intelligence, generative AI) make the statistical results of keywords inaccurate, and there is a lack of correlation between high-frequency keywords. The document direction represented by a single keyword is relatively broad and not granular enough, so the keyword cannot fully meet and cover the user's research direction.

[0007] 3) Problems faced in displaying recommended content: The current method adopted by the subject literature recommendation system is to push all the information to users at one time, which lacks a structured presentation format. This approach is not conducive to users' more refined selective learning. [Summary of the invention]

[0008] The embodiments of the present application provide a method, device, equipment and medium for recommending special documents, aiming to solve the technical problems existing in the related technologies.

[0009] In a first aspect, the present application provides a method for recommending special documents, including:

[0010] Get the literature information under the user's preset topic;

[0011] Determine a feature set corresponding to the special document according to the document information;

[0012] Performing dimensionality reduction processing on the feature set to obtain a feature set of special documents after dimensionality reduction;

[0013] Performing subject community mining on the feature set of thematic documents after dimension reduction to determine the subject community mining result;

[0014] According to the subject community mining results, literature recommendations are made for the user.

[0015] In one embodiment, optionally, the document information includes at least one of the following: document title, abstract and document keywords, and determining the feature set corresponding to the special document according to the document information includes:

[0016] When there are no document keywords in the document information, the document title and / or abstract is segmented to obtain a preliminary segmentation result;

[0017] Performing secondary processing on the preliminary word segmentation result to obtain a final word segmentation result;

[0018] Extracting important keywords that meet the first preset condition from the final word segmentation result as target keywords;

[0019] When there are document keywords in the document information, all the document keywords are sorted by importance, and the document keywords that meet the second preset condition are selected as target keywords;

[0020] The feature set corresponding to the subject document is determined according to the target keywords corresponding to all the documents.

[0021] In one embodiment, optionally, the document title and / or abstract is segmented to obtain a preliminary segmentation result, including:

[0022] Performing word segmentation processing on the document title and / or abstract, and performing part-of-speech tagging to obtain preliminary word segmentation results and part-of-speech tagging results;

[0023] The preliminary word segmentation result is subjected to secondary processing to obtain a final word segmentation result, including:

[0024] According to the part-of-speech tagging result, selecting pre-selected keywords that meet preset requirements from the preliminary word segmentation results as the final word segmentation results;

[0025] Extracting important keywords that meet the first preset condition from the final word segmentation result as target keywords includes:

[0026] Calculate the weight value of each pre-selected keyword and sort them in descending order by weight value;

[0027] A third preset number of keywords are selected from the sorted pre-selected keywords as the target keywords.

[0028] In one embodiment, optionally, all the document keywords are sorted by importance, and the document keywords that meet the second preset condition are selected as target keywords, including:

[0029] The first calculation formula is used to calculate the key score corresponding to each document keyword;

[0030] When the key score is greater than or equal to a preset value, the importance value corresponding to the document keyword is calculated using a second calculation formula;

[0031] When the key score is less than a preset value, the importance value corresponding to the document keyword is calculated using a third calculation formula;

[0032] According to the importance value, the document keywords corresponding to each document are sorted in descending order of importance;

[0033] The first preset number of keywords ranked in front are determined as the target keywords.

[0034] In one embodiment, optionally, the feature set is subjected to dimensionality reduction processing to obtain a reduced-dimensional feature set of special documents, including:

[0035] The feature set is processed into blocks to obtain a plurality of feature blocks;

[0036] Calculate the similarity between different feature blocks to obtain the similarity calculation result;

[0037] When the similarity calculation result between any two feature blocks is greater than a preset similarity threshold, the features corresponding to the any two feature blocks are merged;

[0038] The feature set of the subject documents after dimensionality reduction is determined based on the merged feature blocks.

[0039] In one embodiment, optionally, subject community mining is performed on the feature set of thematic documents after dimension reduction to determine the subject community mining result, including:

[0040] Constructing a feature co-occurrence network based on the document features in the thematic document feature set after dimension reduction, wherein each document feature is used as a node of the feature co-occurrence network, the node value corresponding to each node is the number of first documents where the document feature appears, and two document features appearing in the same document are connected as an edge of the feature co-occurrence network, and the value of the edge is the number of second documents where the two document features co-occur;

[0041] The modularity algorithm is used to perform topic community mining on the feature co-occurrence network to determine the topic community mining results.

[0042] In one embodiment, optionally, the document information also includes the most recent learning time, and the modularity algorithm is used to perform topic community mining on the feature co-occurrence network to determine the topic community mining results, including:

[0043] Using a modularity algorithm to perform topic community mining on the feature co-occurrence network to determine all topic communities;

[0044] Sort all the subject communities in descending order according to the user's most recent learning time to obtain the subject community sorting result;

[0045] Selecting a second preset number of target subject community ranking results ranked first from the subject community ranking results;

[0046] The target subject community ranking result is determined as the subject community mining result.

[0047] In one embodiment, optionally, making a document recommendation for the user based on the subject community mining result includes:

[0048] Finding target documents to be recommended according to each feature representation in the subject community mining result;

[0049] Arrange the target documents to be recommended in descending order according to the number of feature representations contained therein to obtain a target document ranking result;

[0050] Documents are recommended to the user in sequence according to the ranking results of the target documents.

[0051] In a second aspect, an embodiment of the present application provides a device for recommending special documents, including:

[0052] The acquisition module is used to obtain the literature information under the user's preset topic;

[0053] A first determination module is used to determine a feature set corresponding to a special document according to the document information;

[0054] A dimension reduction module is used to perform dimension reduction processing on the feature set to obtain a feature set of special documents after dimension reduction;

[0055] The second determination module is used to perform subject community mining on the feature set of thematic documents after dimension reduction and determine the subject community mining results;

[0056] The recommendation module is used to recommend documents to the user based on the subject community mining results.

[0057] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for recommending special documents when executing the computer program.

[0058] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for recommending special documents are implemented.

[0059] In the scheme implemented by the above-mentioned method, device, equipment and medium for recommending special documents, the document information under the topic preset by the user is obtained; the feature set corresponding to the special document is determined according to the document information; the feature set is subjected to dimensionality reduction processing to obtain the special document feature set after dimensionality reduction; the subject community mining is performed on the subject document feature set after dimensionality reduction to determine the subject community mining results; and the document recommendation is performed for the user according to the subject community mining results. In the present invention, special documents are used to represent the topic set by the user instead of the topic name, which can more accurately represent the research direction of the topic, and the feature set of the special document is subjected to dimensionality reduction to avoid inaccurate statistical information caused by the diversity of feature representation and poor clustering effect caused by feature divergence. At the same time, the subject community mining is performed based on the feature set after dimensionality reduction of the special document, and the special document of the user is further granularized, which can not only make the document distribution structure clearer to the user, but also allow the user to selectively learn recommended documents.

Brief Description of the Drawings

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1A schematic flow chart of a method for recommending special documents according to an embodiment of the present application is shown.

[0062] Figure 2 A schematic flow chart of step S102 in a method for recommending special documents according to an embodiment of the present application is shown.

[0063] Figure 3 A schematic flow chart of step S204 in a method for recommending special documents according to an embodiment of the present application is shown.

[0064] Figure 4 A schematic flow chart of step S103 in a method for recommending special documents according to an embodiment of the present application is shown.

[0065] Figure 5 A schematic flow chart of step S1053 in a method for recommending special documents according to an embodiment of the present application is shown.

[0066] Figure 6 A schematic diagram of learning materials under the topic of user A according to an embodiment of the present application is shown.

[0067] Figure 7 and Figure 8 A schematic diagram showing a result of recommending special documents according to an embodiment of the present application is shown.

[0068] Fig. 9 A block diagram of a device for recommending special documents according to an embodiment of the present application is shown. [Specific implementation method]

[0069] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0070] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0071] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0072] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0073] See also Figure 1, Figure 1 A schematic flow chart of a method for recommending special documents according to an embodiment of the present application is shown.

[0074] like Figure 1 As shown, according to a method for recommending special documents in one embodiment of the present application, the process includes:

[0075] Step S101, obtaining document information under a topic preset by a user;

[0076] The document information includes, but is not limited to, the title, keywords, author, issuing organization, publishing unit, and the most recent study time of the document.

[0077] Step S102, determining a feature set corresponding to a subject document according to the document information;

[0078] like Figure 2 As shown, in one embodiment, optionally, the document information includes at least one of the following: document title, abstract and document keywords, and step S102 includes:

[0079] Step S201, when there is no document keyword in the document information, segment the document title and / or abstract to obtain a preliminary segmentation result;

[0080] In one embodiment, optionally, the document title and / or abstract is segmented to obtain a preliminary segmentation result, including:

[0081] Performing word segmentation processing on the document title and / or abstract, and performing part-of-speech tagging to obtain preliminary word segmentation results and part-of-speech tagging results;

[0082] In this step, the document title and abstract are obtained as the text sequence text of the keywords to be extracted. If the document has no abstract, the title is taken as the text. The subsequent process is introduced by taking text = "Digital X-ray Detection Technology Gives Pressure Tanks a "Physical Examination" as an example; segmentation and part-of-speech tagging are performed for the text to be analyzed: the jieba word segmentation tool is used to segment and tag the text, and the word and part-of-speech tagging results segp_s = [(seg_1,pos_tag_1),......,(seg_i,pos_tag_i),......], where seg_i and pos_tag_i are s_tag_i is the i-th word and part-of-speech tagging result after word segmentation. The word segmentation tool can also use other word segmentation tools, such as HanLP, FoolNLTK and other Chinese word segmentation tools. Take text = "Digital ray detection technology gives pressure tanks a "physical examination" as an example, segp_s = [("digital", "n"), ("ray", "n"), ("detection", "v"), ("technology", "n"), ("give", "p"), ("pressure", "n"), ("tank", "n"), ("do", "v"), (""", "x"), ("physical examination", "v")].

[0083] Step S202, performing secondary processing on the preliminary word segmentation result to obtain a final word segmentation result;

[0084] The preliminary word segmentation result is subjected to secondary processing to obtain a final word segmentation result, including:

[0085] According to the part-of-speech tagging result, selecting pre-selected keywords that meet preset requirements from the preliminary word segmentation results as the final word segmentation results;

[0086] In this step, the preliminary word segmentation results are processed twice to obtain the final word segmentation results. Taking the above example, the processing process includes:

[0087] 1) Remove words with the parts of speech of x (non-morpheme words), c (conjunction), p (preposition), u (particle), e (interjection), o (onomatopoeia), q (quantifier), r (pronoun), and y (modal particle), and use the remaining words as the phrase to be merged merge_segs. For example, text = "Digital ray detection technology gives pressure tanks a "physical examination", merge_segs = ["digital", "ray", "detection", "technology", "pressure", "tank", "do", "physical examination"];

[0088] 2) Extract valid 1-gram sequences: select words with part of speech "n" (noun) and length greater than 1 as the 1-ngram sequences to be selected. For example, if text = "Digital ray detection technology gives pressure tanks a "physical examination", the valid 1-ngram sequence is ["digital", "ray", "technology", "pressure"];

[0089] 3) Extract valid n-gram sequences (n>1). The extraction process is as follows:

[0090] a) Extract the sequence ngram_seqs from merge_segs in sequence according to the length of n i , extract formula:

[0091] ngram_seqs i =merge_segs[i:i+n-1]+merge_segs[i+n-1]

[0092] in:

[0093] len(merge_segs[i:i+n-1]) <avg_k,

[0094] len(merge_segs[i+n-1] <avg_k,

[0095] avg_k=4, taken from the average word length of keywords in HowNet documents.

[0096] 0= <i<=len(merge_segs)-n,

[0097] len(merge_segs) is the number of words in merge_segs,

[0098] n<=len(merge_segs)

[0099] n<=avg_k.

[0100] Take text="Digital ray detection technology performs "physical examination" on pressure tanks" as an example, the n-gram sequence extracted here is ["pressure tank does", "tank does physical examination", "digital ray", "ray detection", "detection technology", "technical pressure", "pressure tank", "tank does", "do physical examination"]

[0101] b) Judge each sequence ngram_seqs in turn i Whether it exists in the text, take text = "Digital ray detection technology gives pressure tanks a "physical examination" as an example, remove the n-gram sequence ["tank physical examination", "technical pressure", "physical examination"] that does not exist in the text;

[0102] c) Take all the remaining sequences in text as valid n-gram (n>1) sequences. Take text = "Digital ray detection technology gives pressure tanks a "physical examination" as an example. The valid n-gram (n>1) sequence is ["pressure tank", "digital ray", "ray detection", "detection technology", "pressure tank", "tank"];

[0103] 4) Construct pre-selected keywords pre_keywords of the text to be analyzed: Here, the pre-selected keywords of the document are composed of valid 1-gram valid sequences and valid n-gram (n>1) valid sequences. Taking text="Digital ray detection technology gives pressure tanks a "physical examination" as an example, the pre-selected keywords pre_keywords = ["pressure tank", "digital ray", "ray detection", "detection technology", "pressure tank", "tank", "digital", "ray", "technology", "pressure"].

[0104] Step S203, extracting important keywords that meet the first preset condition from the final word segmentation result as target keywords;

[0105] Extracting important keywords that meet the first preset condition from the final word segmentation result as target keywords includes:

[0106] Calculate the weight value of each pre-selected keyword and sort them in descending order by weight value;

[0107] The weight value calculation formula of the pre-selected keywords is as follows:

[0108] kw i =w1*k_count i +w2*k_len i +w3*k_index i

[0109]

[0110] Where: k_count i Indicates the frequency of occurrence of the i-th pre-selected keyword in the text to be analyzed, k_len i Indicates the length of the i-th pre-selected keyword; k_index i Indicates the position weight of the i-th pre-selected keyword in the text to be analyzed; index i Indicates the position of the i-th pre-selected keyword in the text to be analyzed;

[0111] Taking text="Digital X-ray detection technology to perform a "physical examination" on pressure tanks" as an example, the order of pre-selected keywords after sorting by weight is ["digital ray", "ray detection", "detection technology", "pressure tank", "digital", "ray", "technology", "pressure"].

[0112] A third preset number of keywords are selected from the sorted pre-selected keywords as the target keywords. For example, the top-3 valid words can be selected in sequence from the sorted pre-selected keywords as the important features segs_im of the document. The specific steps include:

[0113] a) Compare the pre-selected keyword with the selected keyword in turn. If the semantic similarity is not higher than the threshold of 0.7 (this threshold is the Multi-sBert adaptation threshold of the model, and the threshold can also be adjusted according to needs), then add the pre-selected keyword as the selected keyword;

[0114] b) Loop in sequence until the number of selected keywords is greater than or equal to 3 (top-3 keywords are taken), or the pre-selected keywords are traversed;

[0115] c) Take the selected keywords as important features of the document.

[0116] Step S204, when there are document keywords in the document information, all the document keywords are sorted by importance, and document keywords that meet the second preset condition are selected as target keywords;

[0117] like Figure 3 As shown, in one embodiment, optionally, step S204 includes:

[0118] Step S301, using the first calculation formula to calculate the key score corresponding to each document keyword;

[0119] Each document generally has 3 to 5 document keywords. The default order of the keywords is the order filled in by the author. In order to highlight the importance of the keywords represented by the keyword order, the document keywords of each document are reordered.

[0120] The first calculation formula includes:

[0121]

[0122] Where: S i represents the key score, T represents the title of the document; k i ∩T represents the document keyword k i and the character intersection of the document title T; len(k i ) represents the document keyword k i The number of characters to include.

[0123] Step S302: when the key score is greater than or equal to a preset value, the importance value corresponding to the document keyword is calculated using a second calculation formula;

[0124] In this step, a preset value may be set as needed, such as 0.5.

[0125] If S i >=0.5 then keyword k i Importance of i The following second calculation formula is used:

[0126] I i =S i +level(k i )

[0127] level(k i )=len(keywordss)-index(k i )

[0128] Among them: len(keywordss) represents the number of keywords in the document; index(k i ) represents the document keyword k i The default order of all keywords in this document.

[0129] Step S303, when the key score is less than a preset value, the importance value corresponding to the document keyword is calculated using a third calculation formula;

[0130] When the key score is less than the preset value, such as S i <0.5, then the literature keyword k i Importance of i The cosine calculation method is adopted, that is, the following third calculation formula is adopted:

[0131] I i =cosine(vector(t),vector(k i )),

[0132] Among them, vector(t) and vector(k i ) are the document title and document keyword k i The semantic vector is converted using a pre-trained language model (e.g., Multi-sBert, multi-E5).

[0133] Step S304, sorting the document keywords corresponding to each document in descending order of importance according to the importance value;

[0134] Step S305: determine a first preset number of keywords ranked in front as the target keywords.

[0135] In this step, the keywords of each document are sorted according to the importance of the keywords, and the top-n keywords are taken as the features of this document. For example, n can be 3, or it can be set according to the needs or application scenarios. For example, the keywords of a document are "higher education, medical education, teaching reform, artificial intelligence, personalized education, experimental teaching", and the keywords and order after sorting are "medical education, teaching reform, artificial intelligence, experimental teaching, personalized education, higher education". Here, the top-3 keywords are taken as the features of this document segs_im = ["medical education", "teaching reform", "artificial intelligence"].

[0136] The above respectively explains the method of determining the target keywords based on the title or abstract, and the literature keywords. Of course, the two can also be combined, that is, both are ranked in importance at the same time, and the target keywords are selected from them.

[0137] Step S205, determining a feature set corresponding to the subject document according to target keywords corresponding to all documents.

[0138] In this step, the union of segs_im of all documents is taken as the feature set features of the subject documents.

[0139] Step S103, performing dimension reduction processing on the feature set to obtain a feature set of special documents after dimension reduction;

[0140] like Figure 4 As shown, in one embodiment, optionally, step S103 includes:

[0141] Step S401, performing block processing on the feature set to obtain multiple feature blocks;

[0142] Wherein step S401 comprises:

[0143] Step S4011, obtain the feature set features: Here, the elements in segs_im of all documents are a feature feature, features = [feature0, feature1, ..., feature n-1 ],n>1, features are n-dimensional.

[0144] Step S4012, obtain the vector representation features_embedding of features: obtain the semantic representation of features, use PreTrained Language Model (for example: sBert, E5, other PTLMs can also be used) to convert each f feature into a high-dimensional vector, features_embedding = [f_emb0, f_emb1, ..., f_emb j ,........,f_emb n-1 ],n>1,features_embedding is n-dimensional, where f_emb j It is m-dimensional (m is determined by the PTLMs model parameters, and different dimensions can be selected as needed), so features_embedding is a matrix with n rows and m columns, f_emb n-1 Represents feature n-1 The vector representation of .

[0145] Step S4012, features dimensionality reduction: according to features and features_embedding, features are quickly reduced in dimensionality, that is, the value of n is reduced, and finally the feature set features_reduction after dimensionality reduction and the mapping relationship map(features,features_reduction) between features and features_reduction are obtained. The dimensionality reduction process is as follows:

[0146] 1) Divide features into blocks of a certain length L: L <n

[0147] features=[fch0,fch1,.......,fch p-1 ],

[0148] in:

[0149] f i =[feature0,feature1,.......,feature L-1 ],L <n,i<p-1;

[0150] f p-1 =[feature0,feature1,.......,feature j-1 ],j<=L;

[0151] 2) Divide features_embedding into blocks by rows according to a certain length L:

[0152] features_embedding = [fembch0, fembch1,......., fembch p-1 ,

[0153] where:

[0154] fembch i = [femb0, femb1,......., femb L-1 , L < n, i < p - 1;

[0155] fembch p-1 = [femb0, femb1,......., femb j-1 , j <= L;

[0156] Step S402, calculate the similarity between different feature blocks to obtain the similarity calculation result;

[0157] Calculate the similarity S i,j between the i-th block and the j-th block in features in sequence, and merge and update features and features_embedding:

[0158] Traversal order: Traverse the diagonal and the upper triangle by rows, from left to right, from top to bottom, that is, i <= j < p,

[0159]

[0160] where:

[0161]

[0162] fembch i and fembch j correspond to the vector representations on fch i and fch j respectively.

[0163] Further expand:

[0164]

[0165] Take i and j in sequence. If i <= j < p is satisfied, then calculate S i,j ;

[0166] Step S403, when the similarity calculation result between any two feature blocks is greater than the preset similarity threshold, merge the features corresponding to the any two feature blocks;

[0167] According to the set similarity threshold T (0 < T < 1.0) and S i,j Perform feature merging, and the merging process is as follows:

[0168] Step S4031, for If x ≥ y, continue to traverse the next element, otherwise execute Step S4032;

[0169] Step S4032, if Then fch j Merge the y-th feature in the block into the x-th feature in fch i Block;

[0170] Step S4033, loop Step S4031 and Step S4032 until all elements in S i,j Are traversed.

[0171] Update features and features_embedding:

[0172] According to the features merged in S recorded in Step S4032 i,j Delete the corresponding feature in the j-th block fch in features, and the dimension of fch j Is reduced; j According to the features merged in S recorded in Step S4032

[0173] Delete the corresponding feature vector representation in the j-th block fch in features_embedding, and the dimension of fembch i,j Is reduced; j j Continue Step S4031 until all blocks in S are traversed. Since the dimension of features_embedding is reduced, S i,j The calculation speed is accelerated, and the speed will become faster and faster as the number of iterations increases.

[0174] i,j The calculation speed is accelerated, and the speed will become faster and faster as the number of iterations increases.

[0175] Step S404, determine the dimensionality-reduced collection of topic literature features according to the merged feature blocks.

[0176] When the features dimension no longer decreases, the training process ends, and the reduced feature set features_reduction and the mapping relationship between features and features_reduction map(features,features_reduction) are obtained;

[0177] According to map(features,features_reduction), the feature segs_im of each document is converted into the reduced-dimensional feature segs_im_reduc.

[0178] Step S104, performing subject community mining on the feature set of thematic documents after dimension reduction to determine the subject community mining result;

[0179] In one embodiment, optionally, step S104 includes:

[0180] Step S1041, constructing a feature co-occurrence network based on the document features in the feature set of thematic document after dimension reduction, wherein each document feature is used as a node of the feature co-occurrence network, the node value corresponding to each node is the number of first documents in which the document feature appears, and two document features appearing in the same document are connected as an edge of the feature co-occurrence network, and the value of the edge is the number of second documents in which the two document features co-occur;

[0181] Step S1042: Use a modularity algorithm to perform topic community mining on the feature co-occurrence network to determine a topic community mining result.

[0182] In one embodiment, optionally, the document information also includes the most recent learning time, and step S1042 includes:

[0183] Using a modularity algorithm to perform topic community mining on the feature co-occurrence network to determine all topic communities;

[0184] Sort all the subject communities in descending order according to the user's most recent learning time to obtain the subject community sorting result;

[0185] Selecting a second preset number of target subject community ranking results ranked first from the subject community ranking results;

[0186] The target subject community ranking result is determined as the subject community mining result.

[0187] In this step, the Louvain algorithm can be used to mine topic communities, or other modularity algorithms such as Girvan-Newman can be used to mine topic communities. The Louvain algorithm process is as follows:

[0188] A) Local Optimization

[0189] a) Initialization: Each node in the network is considered as an independent community.

[0190] b) Iteration: For each node in the network, try to move it to the community where its neighbor nodes are located, and calculate the change in modularity before and after the move.

[0191] c) Selection: If moving a node can result in the largest increase in modularity, then the node is moved to the new community.

[0192] Repeat steps b) and c) until no node movement can increase modularity.

[0193] B) Community Merger

[0194] 1) Build a new network: Treat the communities formed in the previous step as new nodes and build a new network. The edge weights in the new network are equal to the total weights of the edges between the corresponding communities in the original network.

[0195] 2) Use the method in the first step in the newly constructed network to continue searching for community structure.

[0196] 3) Repeat the above two steps until the modularity of the entire network no longer increases significantly, and each independent network is regarded as a theme community.

[0197] 4) Output all subject communities: Here, the features corresponding to the top-n nodes with the largest median value (the frequency of literature corresponding to the feature) in each subject community are used to represent the subject community. n is 3 here, and can also be set according to needs and application scenarios.

[0198] communities = {com_0:[feature_0_0,feature_0_1,feature_0_2],......,com_i:[feature_i_0,feature_i_1,feature_i_2],......}, where i represents the i-th topic.

[0199] 5) Select the topic community: All topic communities are sorted in descending order by the user's most recent learning time, and the top-n topics with the latest most recent learning time are selected. Here, n is 5 by default and can be set according to needs and application scenarios. The most recent learning time of a topic community is the time when all documents corresponding to the topic feature were most recently learned by the user.

[0200] Step S105: recommending documents to the user based on the subject community mining results.

[0201] like Figure 5As shown, in one embodiment, optionally, step S105 includes:

[0202] Step S501, searching for a target document to be recommended according to each feature representation in the subject community mining result;

[0203] Step S502, arranging the target documents to be recommended in descending order according to the number of feature representations contained therein, to obtain a target document ranking result;

[0204] In this step, the target documents to be recommended are sorted in descending order according to the number of feature representations contained therein to obtain a target document sorting result, for example, based on the representation com_i of each subject community: [feature_i_0, feature_i_1, feature_i_2].

[0205] The ranking result of the target document is:

[0206] (1) Recommend documents that contain the three features feature_i_0, feature_i_1, and feature_i_2;

[0207] (2) Recommend documents that contain the three features feature_i_0 and feature_i_1;

[0208] (3) The recommended document contains feature_i_0.

[0209] Step S503: recommending documents to the user in sequence according to the ranking results of the target documents.

[0210] According to the results of the target literature sorting in the previous step, the priority of the recommended literature is as follows:

[0211] (1) Recommend documents that contain the three features feature_i_0, feature_i_1, and feature_i_2;

[0212] (2) Recommend documents that contain the three features feature_i_0 and feature_i_1;

[0213] (3) The recommended document contains feature_i_0.

[0214] In this way, the literature is recommended in order of priority, which facilitates users to conduct refined selective learning.

[0215] In the present invention, the topic documents are used to represent the topic set by the user instead of the topic name, which can more accurately represent the research direction of the topic, reduce the dimension of the feature set of the topic documents, avoid inaccurate statistical information caused by feature representation diversity and poor clustering effect caused by feature divergence, and at the same time, perform subject community mining based on the reduced-dimensional feature set of the topic documents to further granularize the user's topic documents, which can not only make the user more clear about the document distribution structure, but also allow the user to selectively learn recommended documents.

[0216] The above technical solution of the present invention is described in detail below with a specific embodiment.

[0217] For example, for user A's topic "Paper Writing-2024", there are 133 learning materials under this topic, such as Figure 6 shown.

[0218] The process of recommending special literature is as follows:

[0219] 1. Obtain the user's special literature:

[0220] groupdocsinfo={["Title":"Artificial Intelligence Explainability: Development and Application","Keywords":"Artificial Intelligence; Machine Learning; Deep Learning; Explainability","Last Learning Time":"2024-10-25 16:07"},["Title":"Characteristics and Topic Selection of Academic Papers - Research on Academic Paper Writing Part 1","Keywords":"Humanities and Social Sciences; Academic Papers; Characteristics; Topic Selection","Last Learning Time":"2024-07-11 10:07"}......}

[0221] 2. Construct a feature set of special documents:

[0222] After the keyword extraction and sorting process, a total of 265 special document features are obtained, which constitute the feature set of special documents features = ["artificial intelligence", "generative artificial intelligence", "generative AI", "academic papers", "dissertations", "academic research", "course papers", "academic journals", "master's thesis", "teaching papers", "academic paper writing", "bibliometrics", "bibliometrics", "topic planning", "topic selection", "planning", "topic selection steps", "big language model", "big language model materials", "big language model evidence,......],

[0223] 3. Dimensionality reduction of the feature set of thematic documents: Set the similarity threshold to reduce the dimension of the document feature set, and obtain the feature set after dimensionality reduction and the mapping relationship between and . After the feature set dimensionality reduction, the 263 features are reduced to 203 features, so there are 203 features in , = {"Artificial Intelligence":"Artificial Intelligence","Generative Artificial Intelligence":"Artificial Intelligence","Generative AI":"Artificial Intelligence","Academic Paper":"Academic Paper","Dissertation":"Academic Paper","Academic Paper Writing":"Academic Paper","Hot Topic Identification":"Hot Topic Identification","Topic Identification":"Hot Topic Identification","Visualization Analysis":"Visualization Analysis","Visualization":"Visualization Analysis",......}, the features of each document under the topic are replaced with the features after dimensionality reduction.

[0224] 4. Mining thematic communities of special documents: Use the document features after dimensionality reduction to build a feature co-occurrence network, and use Louvain for topic mining to obtain the top-5 topic community sets and the representation of this topic community = [0, ['artificial intelligence', 'graduate education', 'ChatGPT',], [1, ['academic papers, 'paper writing', 'scientific and technological papers'], [2, ['hot topic identification', 'Doc2Vec', 'digital government'], [3, ['community question and answer', 'expert recommendation', 'PageRank'], [4, ['co-word analysis', 'comparative education', 'doctoral dissertation topic selection']]

[0225] 5. Recommend literature based on the results of subject community mining: Recommend literature based on each subject community. The recommended results are as follows: Figure 7 and Figure 8 As shown (only the literature recommendation results of two subject communities are shown).

[0226] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0227] Fig. 9 A block diagram of a device for recommending special documents according to an embodiment of the present application is shown.

[0228] like Fig. 9 As shown, in a second aspect, the embodiment of the present application provides a device 90 for recommending special documents, including:

[0229] The acquisition module 91 is used to acquire the literature information under the topic preset by the user;

[0230] A first determination module 92, used to determine a feature set corresponding to a special document according to the document information;

[0231] A dimension reduction module 93 is used to perform dimension reduction processing on the feature set to obtain a feature set of special documents after dimension reduction;

[0232] The second determination module 94 is used to perform subject community mining on the feature set of thematic documents after dimension reduction and determine the subject community mining result;

[0233] The recommendation module 95 is used to recommend documents to the user based on the subject community mining results.

[0234] In one embodiment, optionally, the document information includes at least one of the following: document title, abstract and document keywords, and the first determination module includes:

[0235] A word segmentation unit is used to perform word segmentation processing on the document title and / or abstract to obtain a preliminary word segmentation result when there are no document keywords in the document information;

[0236] A processing unit, used for performing secondary processing on the preliminary word segmentation result to obtain a final word segmentation result;

[0237] An extraction unit, configured to extract important keywords that meet a first preset condition from the final word segmentation result as target keywords;

[0238] A first sorting unit is used to sort all the document keywords by importance when there are document keywords in the document information, and select the document keywords that meet the second preset condition as target keywords;

[0239] The first determining unit is used to determine the feature set corresponding to the subject document according to the target keywords corresponding to all the documents.

[0240] In one embodiment, optionally, the word segmentation unit is used to:

[0241] Performing word segmentation processing on the document title and / or abstract, and performing part-of-speech tagging to obtain preliminary word segmentation results and part-of-speech tagging results;

[0242] The processing unit is used to:

[0243] According to the part-of-speech tagging result, selecting pre-selected keywords that meet preset requirements from the preliminary word segmentation results as the final word segmentation results;

[0244] The extraction unit is used for:

[0245] Calculate the weight value of each pre-selected keyword and sort them in descending order by weight value;

[0246] A third preset number of keywords are selected from the sorted pre-selected keywords as the target keywords.

[0247] In one embodiment, optionally, the first sorting unit is used to:

[0248] The first calculation formula is used to calculate the key score corresponding to each document keyword;

[0249] When the key score is greater than or equal to a preset value, the importance value corresponding to the document keyword is calculated using a second calculation formula;

[0250] When the key score is less than a preset value, the importance value corresponding to the document keyword is calculated using a third calculation formula;

[0251] According to the importance value, the document keywords corresponding to each document are sorted in descending order of importance;

[0252] The first preset number of keywords ranked in front are determined as the target keywords.

[0253] In one embodiment, optionally, the dimension reduction module includes:

[0254] A block division unit, used for performing block processing on the feature set to obtain a plurality of feature blocks;

[0255] A calculation unit, used for calculating the similarity between different feature blocks and obtaining a similarity calculation result;

[0256] A merging unit, configured to merge features corresponding to any two feature blocks when a similarity calculation result between the two feature blocks is greater than a preset similarity threshold;

[0257] The second determining unit is used to determine the feature set of the subject documents after dimension reduction according to the merged feature blocks.

[0258] In one embodiment, optionally, the second determining module includes:

[0259] A construction unit is used to construct a feature co-occurrence network according to the document features in the thematic document feature set after dimensionality reduction, wherein each document feature is used as a node of the feature co-occurrence network, the node value corresponding to each node is the number of first documents in which the document feature appears, and two document features appearing in the same document are connected as an edge of the feature co-occurrence network, and the value of the edge is the number of second documents in which the two document features co-occur;

[0260] The mining unit is used to perform topic community mining on the feature co-occurrence network using a modularity algorithm to determine a topic community mining result.

[0261] In one embodiment, optionally, the document information further includes the most recent learning time, and the mining unit is used to:

[0262] Using a modularity algorithm to perform topic community mining on the feature co-occurrence network to determine all topic communities;

[0263] Sort all the subject communities in descending order according to the user's most recent learning time to obtain the subject community sorting result;

[0264] Selecting a second preset number of target subject community ranking results ranked first from the subject community ranking results;

[0265] The target subject community ranking result is determined as the subject community mining result.

[0266] In one embodiment, optionally, the recommendation module includes:

[0267] A search unit, used for searching target documents to be recommended according to each feature representation in the subject community mining result;

[0268] A second sorting unit is used to sort the target documents to be recommended in descending order according to the number of feature representations contained therein, to obtain a sorting result of the target documents;

[0269] The document recommendation unit is used to recommend documents to the user in sequence according to the ranking results of the target documents.

[0270] The specific definition of the subject document recommendation device can be found in the definition of the subject document recommendation method mentioned above, which will not be repeated here. Each module in the above-mentioned subject document recommendation device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0271] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0272] Perform layout analysis on the document to be translated to obtain layout analysis result information;

[0273] Dividing the layout analysis result information into text information, non-text information and attribute information according to the document type;

[0274] Respectively translating and parsing the text information and the non-text information to obtain parsing result information;

[0275] Generate an initial translation according to each parsing result information, the attribute information and a preset translation template;

[0276] The position of the text content in the initial translation is detected, and the target text with abnormal position is relocated to obtain the target translation.

[0277] In one embodiment, optionally, performing layout analysis on the file to be translated to obtain layout analysis result information includes:

[0278] Analyze the layout of each page of the file to be translated in order according to the page number of the file to be translated, so as to obtain the metadata XML information corresponding to each page;

[0279] The metadata XML information corresponding to each page is divided into hierarchical nodes according to the XML hierarchical structure, wherein the hierarchical nodes include at least one of the following: page node, segment node, row node, picture node, table node and formula node, as well as attribute information corresponding to each node.

[0280] In one embodiment, optionally, the layout analysis result information is divided into text information, non-text information and attribute information according to the document type, including:

[0281] According to the hierarchical nodes, the first metadata XML information corresponding to the segment nodes and row nodes is divided into text information, the second metadata XML information corresponding to the picture nodes, table nodes and formula nodes is divided into non-text information, and the third metadata XML information corresponding to the attribute information of each node is divided into attribute information.

[0282] In one embodiment, optionally, the text information and the non-text information are translated and parsed respectively to obtain parsing result information, including:

[0283] For the text information, parse the metadata XML information of all row nodes corresponding to each segment node in the first metadata XML information, and add them into a text array to obtain a text array set corresponding to the entire page of text;

[0284] Performing text translation on the text array set to obtain a text translation;

[0285] For the non-text information, parse the metadata XML information of the picture node, the table node and the formula node in the second metadata XML information, determine the position information and the coordinate range information corresponding to each node in the form of an image tag, and capture the corresponding target image;

[0286] For the attribute information, parse the third metadata XML information and generate corresponding structure tags in HTML.

[0287] In one embodiment, optionally, generating an initial translation according to each parsing result information, the attribute information and a preset translation template includes:

[0288] According to the text translation, the image tag and the structure tag are marked in the preset translation template, and the corresponding text translation and target image are inserted into the preset translation template to generate the initial translation.

[0289] In one embodiment, optionally, performing position detection on the text content in the initial translation and relocating the target text with abnormal position to obtain the target translation includes:

[0290] According to the attribute information corresponding to each segment node, each segment node of each page in the initial translation is sorted to obtain a sorted result;

[0291] Determine whether the target areas corresponding to any two segment nodes intersect in order from top to bottom and from left to right;

[0292] When it is determined that there is an intersection, the segment nodes sorted later are shifted downward according to the intersection height until all segment nodes have no intersection.

[0293] In one embodiment, optionally, the method further comprises:

[0294] Obtaining paragraph format information and column layout information of the file to be translated;

[0295] The paragraph format and column layout of the target translation are adjusted according to the paragraph format information and column layout information.

[0296] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or electronic device can refer to the relevant description in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0297] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0298] It should be understood that, although the terms first, second, etc. may be used to describe the setting unit in the embodiments of the present application, these setting units should not be limited to these terms. These terms are only used to distinguish the setting units from each other. For example, without departing from the scope of the embodiments of the present application, the first setting unit may also be referred to as the second setting unit, and similarly, the second setting unit may also be referred to as the first setting unit.

[0299] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0300] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0301] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0302] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0303] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for recommending special documents, characterized in that: include: Get the literature information under the user's preset topic; Determine a feature set corresponding to the special document according to the document information; Performing dimensionality reduction processing on the feature set to obtain a feature set of special documents after dimensionality reduction; Performing subject community mining on the feature set of thematic documents after dimension reduction to determine the subject community mining result; According to the subject community mining results, literature recommendations are made for the user.

2. The method according to claim 1, characterized in that The document information includes at least one of the following: document title, abstract and document keywords. The step of determining a feature set corresponding to a special document based on the document information includes: When there are no document keywords in the document information, the document title and / or abstract is segmented to obtain a preliminary segmentation result; Performing secondary processing on the preliminary word segmentation result to obtain a final word segmentation result; Extracting important keywords that meet the first preset condition from the final word segmentation result as target keywords; When there are document keywords in the document information, all the document keywords are sorted by importance, and the document keywords that meet the second preset condition are selected as target keywords; The feature set corresponding to the subject document is determined according to the target keywords corresponding to all the documents.

3. The method according to claim 2, characterized in that Sort all the literature keywords by importance and select the literature keywords that meet the second preset condition as the target keywords, including: The first calculation formula is used to calculate the key score corresponding to each document keyword; When the key score is greater than or equal to a preset value, the importance value corresponding to the document keyword is calculated using a second calculation formula; When the key score is less than a preset value, the importance value corresponding to the document keyword is calculated using a third calculation formula; According to the importance value, the document keywords corresponding to each document are sorted in descending order of importance; The first preset number of keywords ranked in front are determined as the target keywords.

4. The method according to claim 1, characterized in that The feature set is subjected to dimensionality reduction processing to obtain a feature set of special documents after dimensionality reduction, including: The feature set is processed into blocks to obtain a plurality of feature blocks; Calculate the similarity between different feature blocks to obtain the similarity calculation result; When the similarity calculation result between any two feature blocks is greater than a preset similarity threshold, the features corresponding to the any two feature blocks are merged; The feature set of the subject documents after dimensionality reduction is determined based on the merged feature blocks.

5. The method according to claim 1, characterized in that Performing subject community mining on the feature set of the subject documents after dimension reduction to determine the subject community mining results includes: Constructing a feature co-occurrence network based on the document features in the thematic document feature set after dimension reduction, wherein each document feature is used as a node of the feature co-occurrence network, the node value corresponding to each node is the number of first documents where the document feature appears, and two document features appearing in the same document are connected as an edge of the feature co-occurrence network, and the value of the edge is the number of second documents where the two document features co-occur; The modularity algorithm is used to perform topic community mining on the feature co-occurrence network to determine the topic community mining results.

6. The method according to claim 5, characterized in that The document information also includes the most recent learning time. The feature co-occurrence network is mined using a modularity algorithm to determine the topic community mining results, including: Using a modularity algorithm to perform topic community mining on the feature co-occurrence network to determine all topic communities; Sort all the subject communities in descending order according to the user's most recent learning time to obtain the subject community sorting result; Selecting a second preset number of target subject community ranking results ranked first from the subject community ranking results; The target subject community ranking result is determined as the subject community mining result.

7. The method according to claim 1, characterized in that According to the subject community mining results, literature recommendations are made for the user, including: Finding target documents to be recommended according to each feature representation in the subject community mining result; Arrange the target documents to be recommended in descending order according to the number of feature representations contained therein to obtain a target document ranking result; Documents are recommended to the user in sequence according to the ranking results of the target documents.

8. A device for recommending special documents, characterized in that: include: The acquisition module is used to obtain the literature information under the user's preset topic; A first determination module is used to determine a feature set corresponding to a special document according to the document information; A dimension reduction module is used to perform dimension reduction processing on the feature set to obtain a feature set of special documents after dimension reduction; The second determination module is used to perform subject community mining on the feature set of thematic documents after dimension reduction and determine the subject community mining results; The recommendation module is used to recommend documents to the user based on the subject community mining results.

9. A computer device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, wherein the instructions are configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are used to execute the method according to any one of claims 1 to 7.

Images