Text processing method and device, and storage medium

Abstract

The present disclosure relates to a text processing method, device and storage medium. The method comprises: comparing the text content in the to-be-labeled document with the text content corresponding to each initial label in the preset database, selecting a candidate labeling label corresponding to the to-be-labeled document from the initial labels according to the matching degree of the to-be-labeled document and each initial label; extracting the label vector of the candidate labeling label from the preset database; obtaining the text vector of the to-be-labeled document based on a target vector conversion model; determining the vector distance between the text vector of the to-be-labeled document and the label vector of each candidate labeling label; and determining the target labeling label for labeling the to-be-labeled document based on each vector distance. In this way, the response speed when determining the target labeling label can be improved, the target labeling label can be accurately and quickly obtained, and the accuracy of determining the target labeling label can be improved on the basis of improving the accuracy of text semantic representation.

Description

Technical Field

[0001] This disclosure relates to natural language processing technology, and more particularly to a text processing method, apparatus and storage medium. Background Technology

[0002] With the widespread use of social media, a huge number of documents are published on social communication platforms every day. For original documents to be annotated, it is necessary to accurately identify the tags mentioned in the documents and determine the target tags for annotating the documents from the initial tags. This ensures that when the recommendation service determines tags based on user profiles and recommends relevant content to users, the documents corresponding to the tags are content with high similarity to the tags.

[0003] In related technologies, when it is necessary to annotate a document and determine the target label, a combination of matching recall and relevance ranking can be used for modeling. In the relevance ranking module, text classification tasks can be used for modeling, i.e., binary classification of the document to be annotated and the initial label text to determine whether there is a relevance between the document to be annotated and the initial label. However, since each pair of documents to be annotated and the initial label for binary classification needs to be input into the model once, the same document to be annotated needs to be processed through the model multiple times, which greatly reduces the response speed when determining the target label.

[0004] In the relevance ranking module, text semantic matching modeling can also be used. That is, the model is used to calculate the text semantic representation (e.g., feature vector) of the document to be labeled and the initial label respectively, and the relevance is measured by the similarity of the two text semantic representations in high-dimensional space. However, due to the design defects of the model itself, the accuracy of the text semantic representation is not high, and the target label cannot be accurately determined. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this disclosure provides a text processing method, apparatus, and storage medium. Firstly, it can directly extract corresponding tag vectors from a preset database, which can improve the response speed when determining target annotation tags. Secondly, by determining the target annotation tags through vector distance, the similarity between documents and tags can be vectorized, enabling more accurate and faster acquisition of target annotation tags. Thirdly, by using a pre-trained target vector transformation model to determine the text vector of the document to be annotated and the tag vector of the initial tags, it can improve the accuracy of determining target annotation tags while enhancing the accuracy of text semantic representation.

[0006] According to a first aspect of the present disclosure, a text processing method is provided, comprising:

[0007] The text content in the document to be annotated is compared with the text content corresponding to each initial tag in the preset database. Based on the matching degree between the document to be annotated and each initial tag, candidate annotation tags corresponding to the document to be annotated are selected from the initial tags. The preset database stores each initial tag and the tag vector corresponding to each initial tag. The tag vector is obtained by vector conversion of the initial tags by the target vector conversion model.

[0008] Extract the label vectors of the candidate labels from the preset database;

[0009] Based on the target vector transformation model, the document to be labeled is transformed into a vector to obtain the text vector of the document to be labeled;

[0010] Determine the vector distance between the text vector of the document to be annotated and the tag vector of each of the candidate annotation tags;

[0011] Based on the distances of each vector, a target label is determined from the candidate labels; wherein the target label is used to label the document to be labeled.

[0012] In some embodiments, comparing the text content in the document to be annotated with the text content corresponding to each initial tag in a preset database includes:

[0013] The non-text content in the document to be annotated is filtered out to obtain an intermediate document with text content;

[0014] The target document is generated based on the text content in the intermediate document that meets the preset filtering conditions;

[0015] The text content in the target document is compared with the text content corresponding to each of the initial tags.

[0016] In some embodiments, the method further includes:

[0017] Identify training documents from historical document data;

[0018] The initial labels are preprocessed to obtain candidate training labels corresponding to the initial labels;

[0019] The text content in the training document is compared with the text content corresponding to each of the candidate training labels. Based on the matching degree between the training document and each of the candidate training labels, the target training label corresponding to the training document is selected from the candidate training labels.

[0020] The training document and the target training label are input into the initial vector transformation model for vector transformation processing to obtain the text vector of the training document and the label vector of the target training label;

[0021] The text vector of the training document and the label vector of the target training label are input into the contrastive learning model to obtain the contrastive loss value;

[0022] The model parameters of the initial vector transformation model are adjusted based on the contrastive loss value to obtain the target vector transformation model.

[0023] In some embodiments, preprocessing the initial label to obtain candidate training labels corresponding to the initial label includes:

[0024] Based on the tag type of the initial tag, at least one associated tag is determined that is related to the initial tag; wherein the associated tag is the same as the object labeled by the initial tag, and the labeling range of the associated tag is greater than the labeling range of the initial tag;

[0025] Based on the annotation range of the initial label and the annotation range of each associated label, determine the splicing order of the initial label and each associated label;

[0026] According to the splicing order, the initial label and each of the associated labels are spliced ​​together to obtain the candidate training label corresponding to the initial label.

[0027] In some embodiments, the target training label corresponding to the training document includes: positive sample label and negative sample label; the step of inputting the text vector of the training document and the label vector of the target training label into the contrastive learning model to obtain the contrastive loss value includes:

[0028] Identify similar documents from the historical document data that have a similarity greater than a preset similarity threshold with the training documents;

[0029] Using the similarity sub-model in the contrastive learning model, the first similarity between the training document and the positive sample label is determined based on the text vector of the training document and the label vector of the positive sample label.

[0030] Using the similarity sub-model, a second similarity between the training document and the negative sample label is determined based on the text vector of the training document and the label vector of the negative sample label;

[0031] The contrast loss value is determined based on the first similarity, the second similarity, and the third similarity between the training document and the similar document; wherein the contrast loss value is negatively correlated with the third similarity and positively correlated with the sum of the first similarity and the second similarity.

[0032] In some embodiments, the method further includes:

[0033] Candidate training tags with a matching degree greater than a preset matching degree to the training document are identified as visible tags;

[0034] Candidate training labels whose matching degree with the training document is less than or equal to the preset matching degree are determined as invisible labels;

[0035] According to a preset labeling strategy, a first number of visible labels are determined as positive sample labels, and a second number of visible labels are determined as negative sample labels;

[0036] Based on the first quantity and the second quantity, a third quantity of the negative sample labels is determined from the invisible labels;

[0037] The ratio between the sum of the second quantity and the third quantity and the first quantity satisfies a preset proportional relationship.

[0038] In some embodiments, determining the target label from the candidate labels based on each of the vector distances includes:

[0039] The candidate labels whose vector distance is greater than a preset distance threshold are determined as the target labels;

[0040] The vector distance is used to characterize the similarity between the document to be labeled and the candidate labels.

[0041] According to a second aspect of the present disclosure, a text processing apparatus is provided, comprising:

[0042] The first comparison module is configured to compare the text content in the document to be annotated with the text content corresponding to each initial tag in a preset database, and select candidate annotation tags corresponding to the document to be annotated from the initial tags based on the matching degree between the document to be annotated and each initial tag; wherein, the preset database pre-stores: each initial tag and the tag vector corresponding to each initial tag, and the tag vector is obtained by vector conversion of the initial tags by a target vector conversion model;

[0043] The extraction module is configured to extract the label vectors of the candidate labeled labels from the preset database;

[0044] The first conversion module is configured to perform vector conversion on the document to be annotated based on the target vector conversion model to obtain the text vector of the document to be annotated;

[0045] The first determining module is configured to determine the vector distance between the text vector of the document to be annotated and the tag vector of each of the candidate annotation tags;

[0046] The second determining module is configured to determine a target label from the candidate labeling labels based on the distances between the vectors; wherein the target labeling label is used to label the document to be labeled.

[0047] In some embodiments, the first comparison module is configured as follows:

[0048] The non-text content in the document to be annotated is filtered out to obtain an intermediate document with text content;

[0049] The target document is generated based on the text content in the intermediate document that meets the preset filtering conditions;

[0050] The text content in the target document is compared with the text content corresponding to each of the initial tags.

[0051] In some embodiments, the apparatus further includes:

[0052] The third determination module is configured to determine training documents from historical document data;

[0053] The processing module is configured to preprocess the initial label to obtain candidate training labels corresponding to the initial label;

[0054] The second comparison module is configured to compare the text content in the training document with the text content corresponding to each of the candidate training tags, and select the target training tag corresponding to the training document from the candidate training tags based on the matching degree between the training document and each of the candidate training tags.

[0055] The second conversion module is configured to input the training document and the target training label into an initial vector conversion model for vector conversion processing to obtain the text vector of the training document and the label vector of the target training label;

[0056] The input module is configured to input the text vector of the training document and the label vector of the target training label into the contrastive learning model to obtain the contrastive loss value;

[0057] The adjustment module is configured to adjust the model parameters of the initial vector transformation model based on the contrast loss value to obtain the target vector transformation model.

[0058] In some embodiments, the processing module is configured to:

[0059] Based on the tag type of the initial tag, at least one associated tag is determined that is related to the initial tag; wherein the associated tag is the same as the object labeled by the initial tag, and the labeling range of the associated tag is greater than the labeling range of the initial tag;

[0060] Based on the annotation range of the initial label and the annotation range of each associated label, determine the splicing order of the initial label and each associated label;

[0061] According to the splicing order, the initial label and each of the associated labels are spliced ​​together to obtain the candidate training label corresponding to the initial label.

[0062] In some embodiments, the target training labels corresponding to the training document include: positive sample labels and negative sample labels; the input module is configured to:

[0063] Identify similar documents from the historical document data that have a similarity greater than a preset similarity threshold with the training documents;

[0064] Using the similarity sub-model in the contrastive learning model, the first similarity between the training document and the positive sample label is determined based on the text vector of the training document and the label vector of the positive sample label.

[0065] Using the similarity sub-model, a second similarity between the training document and the negative sample label is determined based on the text vector of the training document and the label vector of the negative sample label;

[0066] The contrast loss value is determined based on the first similarity, the second similarity, and the third similarity between the training document and the similar document; wherein the contrast loss value is negatively correlated with the third similarity and positively correlated with the sum of the first similarity and the second similarity.

[0067] In some embodiments, the apparatus further includes:

[0068] The fourth determining module is configured to determine candidate training tags that have a matching degree greater than a preset matching degree with the training document as visible tags;

[0069] The fifth determining module is configured to determine candidate training labels whose matching degree with the training document is less than or equal to the preset matching degree as invisible labels;

[0070] The sixth determining module is configured to determine a first number of visible labels as positive sample labels and a second number of visible labels as negative sample labels according to a preset labeling strategy;

[0071] The seventh determining module is configured to determine a third number of negative sample labels from the invisible labels based on the first number and the second number;

[0072] The ratio between the sum of the second quantity and the third quantity and the first quantity satisfies a preset proportional relationship.

[0073] In some embodiments, the second determining module is configured to:

[0074] The candidate labels whose vector distance is greater than a preset distance threshold are determined as the target labels;

[0075] The vector distance is used to characterize the similarity between the document to be labeled and the candidate labels.

[0076] According to a third aspect of the present disclosure, a text processing apparatus is provided, comprising:

[0077] processor;

[0078] Memory configured to store processor-executable instructions;

[0079] The processor is configured to implement the steps of any of the text processing methods in the first aspect above during execution.

[0080] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of a text processing apparatus, the apparatus is enabled to perform steps in any of the text processing methods of the first aspect described above.

[0081] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0082] In this embodiment, firstly, by pre-storing candidate label vectors in a preset database, when the target label needs to be determined, the corresponding label vector can be directly extracted from the preset database. Furthermore, before determining the vector distance between the text vector of the document to be labeled and the label vectors of each candidate label, the document to be labeled only needs to determine its text vector through a single target vector transformation model, which improves the response speed when determining the target label. Secondly, since vector distance can be used to characterize the similarity between a document and a label, determining the target label through vector distance can vectorize the similarity between the document and the label. Compared to using an additional similarity calculation model for similarity calculation, this method can obtain the target label more accurately and quickly. Thirdly, by using a pre-trained target vector transformation model to determine the text vector of the document to be labeled and the label vector of the initial label, the accuracy of determining the target label can be improved while simultaneously increasing the accuracy of text semantic representation.

[0083] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0084] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0085] Figure 1 This is a flowchart illustrating a text processing method according to an exemplary embodiment;

[0086] Figure 2A This is a structure of a Transformer-based bidirectional encoding representation model illustrated according to an exemplary embodiment. Figure 1 ;

[0087] Figure 2B This is a structural diagram of a Transformer-based bidirectional encoding representation model, illustrated according to an exemplary embodiment.

[0088] Figure 3A This is a structure of a sentence embedding model using a Siamese BERT network, as illustrated in an exemplary embodiment. Figure 1 ;

[0089] Figure 3B This is a structural diagram of a sentence embedding model using a Siamese BERT network, illustrated according to an exemplary embodiment;

[0090] Figure 4 This is a structural diagram illustrating the correspondence between training documents and target training labels according to an exemplary embodiment;

[0091] Figure 5 This is a schematic diagram illustrating the determination of visible labels and positive sample labels among candidate training labels according to an exemplary embodiment;

[0092] Figure 6 This is a block diagram of a text processing apparatus according to an exemplary embodiment;

[0093] Figure 7 This is a block diagram illustrating a text processing apparatus 800 according to an exemplary embodiment;

[0094] Figure 8 This is a hardware structure block diagram of a text processing device according to an exemplary embodiment. Detailed Implementation

[0095] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0096] This disclosure provides a text processing method. Figure 1 This is a flowchart illustrating a text processing method according to an exemplary embodiment, such as... Figure 1 As shown. This method mainly includes the following steps:

[0097] In step 101, the text content in the document to be annotated is compared with the text content corresponding to each initial tag in the preset database. Based on the matching degree between the document to be annotated and each initial tag, candidate annotation tags corresponding to the document to be annotated are selected from the initial tags. The preset database stores each initial tag and the tag vector corresponding to each initial tag. The tag vector is obtained by vector transformation of the initial tags by the target vector transformation model.

[0098] In step 102, the tag vectors of the candidate labels are extracted from the preset database;

[0099] In step 103, the document to be annotated is vectorized based on the target vector conversion model to obtain the text vector of the document to be annotated;

[0100] In step 104, the vector distance between the text vector of the document to be annotated and the tag vector of each of the candidate annotation tags is determined.

[0101] In step 105, a target label is determined from the candidate labels based on each of the vector distances; wherein the target label is used to label the document to be labeled.

[0102] It should be noted that the text processing method proposed in this disclosure can be applied to electronic devices or servers. Here, electronic devices can include terminal devices, such as mobile terminals or fixed terminals. Mobile terminals can include devices such as mobile phones, tablets, and laptops. Fixed terminals can include desktop computers, smart TVs, etc. A server, as a type of computer, can provide computing or application services to other client machines (such as computers, smartphones, ATMs, and even large equipment like train systems) on a network.

[0103] The text processing method in this embodiment can be configured in a text processing device, which can be located in a server or an electronic device. This embodiment does not limit this.

[0104] It should be noted that the execution entity of the embodiments disclosed herein may be, in hardware, a central processing unit (CPU) in a server or electronic device, and in software, a related background service in a server or electronic device, without limitation.

[0105] Here, the document to be annotated can be an original document, such as an original blog post; it can also be a newly generated document based on an original one, such as a newly generated original blog post obtained from an application, or a real-time generated original news article. The text content in the document to be annotated can be any text content other than non-text content. Non-text content can include Uniform Resource Locator (URL), special characters (characters used infrequently and difficult to input directly), emoticons, etc. The text content can be a sentence or a paragraph, including text, numbers, etc.

[0106] The preset database is a cache space pre-created in memory for data storage when the database is offline. In some embodiments, the preset database can be local memory space; in other embodiments, it can be cloud-based memory space. The initial tag can be a raw, unprocessed tag. Here, the initial tag can be a tag of any level, such as a third-level tag, a second-level tag, or a first-level tag. The text content corresponding to the initial tag can be any text content other than the main text content of the initial tag.

[0107] In some embodiments, when it is necessary to annotate a document, the text content in the document to be annotated can be compared with the text content corresponding to each initial tag in the preset database to obtain the matching degree between the document to be annotated and each initial tag. Based on the obtained matching degree between the document to be annotated and each initial tag, candidate annotation tags corresponding to the document to be annotated can be selected from the initial tags. The preset database stores each initial tag and the tag vector corresponding to each initial tag, and the tag vector is obtained by the target vector conversion model to convert the initial tags.

[0108] In some embodiments, a matching recall module can be used to compare the document to be labeled with the initial tags. This matching recall module can be a module that compares the text content of the document to be labeled with the text content corresponding to the initial tags, or it can be a module that compares the semantic information of the document to be labeled with the semantic information of the initial tags. Here, the matching recall module can be embedded in the target vector transformation model, or it can be configured as a separate module together with the target vector transformation model in an electronic device or server. There are no specific limitations on the model and configuration of the matching recall module, as long as the comparison function can be implemented.

[0109] In some embodiments, the matching recall module can employ either content-based recall or collaborative filtering-based recall. Taking content-based recall as an example, during the recall process, the text content of the document to be labeled is matched with the text content corresponding to the initial tags. Based on the matching results, initial tags with a matching degree greater than a preset text content matching degree are recalled, and these recalled initial tags are determined as candidate labeling tags. Here, the matching degree can characterize the repetition rate between the text content in the document to be labeled and the text content corresponding to the initial tags.

[0110] For example, if the preset text content matching degree is 50%, and one of the initial tags has a text content matching degree of 60% with the document to be annotated, since the text content matching degree of 60% is greater than the preset text content matching degree of 50%, then the initial tag can be determined as the candidate tag corresponding to the document to be annotated.

[0111] In some embodiments, when candidate labeling tags that match the document to be labeled are determined from the initial tags, the label vectors corresponding to each initial tag are pre-stored in the preset database, and the label vectors are obtained by vector transformation by the target vector transformation model. Therefore, the label vectors of the candidate labeling tags can be directly extracted and used from the preset database.

[0112] In some embodiments, when offline, the label vectors corresponding to the initial labels can be determined in advance based on the target vector transformation model, and all initial labels and the label vectors corresponding to each initial label can be stored in memory as key-value pairs.

[0113] In other embodiments, when it is necessary to determine the target annotation tags for annotating the document to be annotated, the matching recall module can first compare the text content of the document to be annotated with the text content corresponding to a preset number of initial tags. After the matching recall module, at least one candidate annotation tag can be determined. Then, the tag vector corresponding to the candidate annotation tag can be searched in the preset database based on the candidate annotation tag.

[0114] Given the label vectors of the candidate labels and the text vector of the document to be labeled, the vector distance between the text vector of the document to be labeled and the label vectors of each candidate label can be determined. Then, the target label can be quickly determined based on the vector distance.

[0115] In some embodiments, the vector distance between the text vector of the document to be annotated and the label vectors of each candidate label can be determined using different distance calculation models. For example, the cosine distance formula can be used to calculate the vector distance between the text vector of the document to be annotated and the label vectors of each candidate label. Another example is the Euclidean distance formula. Yet another example is the Manhattan distance formula.

[0116] Taking the calculation of the vector distance between the text vector of the document to be annotated and the label vectors of each candidate label using the cosine distance formula as an example, the vector distance can be represented by the cosine distance. The formula for calculating the cosine distance is as follows:

[0117]

[0118] In formula (1), cos(θ) is the cosine distance, A is the text vector of the document to be annotated, B is the label vector of the candidate label, ||A|| is the magnitude of the text vector of the document to be annotated, and ||B|| is the magnitude of the label vector of the candidate label. i For each dimension of the text vector of the document to be annotated, B i Each dimension in the label vector of the candidate labels.

[0119] Given the vector distance between the text vector of the document to be annotated and the label vectors of each candidate label, the target label can be determined from the candidate labels based on the vector distances; the target label can be used to annotate the document to be annotated.

[0120] In some embodiments, once the target annotation tag is determined, the association between the document to be annotated and the target annotation tag can be directly established, thus completing the annotation of the document to be annotated by the target annotation tag. In this way, when the recommendation service determines the target annotation tag for a user based on the user profile, the documents to be annotated recommended in association with the target annotation tag are all content with a high degree of relevance to the target annotation tag. In other embodiments, the association between the document to be annotated and the target annotation tag can be established upon receiving an annotation instruction for the document to be annotated, thus completing the annotation of the document to be annotated.

[0121] In some embodiments, the vector distances can be sorted according to their magnitude, and the target label can be determined from the candidate labels based on the sorting results. For example, the vector distances can be sorted in descending order, and the candidate labels preceding a preset number can be determined as the target labels. Taking a preset number of two as an example, the first two candidate labels in the sorting results can be determined as the target labels.

[0122] For example, sort the vector distances in descending order, and determine the target label from the candidate labels according to a preset percentage. Taking 10 candidate labels and a preset percentage of 10% as an example, the first candidate label in the sorting result can be determined as the target label.

[0123] In some embodiments, determining the target label from the candidate labels based on each of the vector distances includes:

[0124] The candidate labels whose vector distance is greater than a preset distance threshold are determined as the target labels;

[0125] The vector distance is used to characterize the similarity between the document to be labeled and the candidate labels.

[0126] Here, candidate labels whose vector distance is greater than a preset distance threshold can be identified as target labels. The vector distance is used to characterize the similarity between the document to be labeled and the candidate labels. For example, with a preset distance threshold of 0.5, candidate labels whose preset distance threshold is greater than 0.5 can be identified as target labels.

[0127] In this embodiment, vectors are used to represent the document to be labeled and the candidate labels, and vector distance is determined by these vectors. The vector distance is used to represent the similarity between the document to be labeled and the candidate labels. This allows for the reuse of the target vector transformation model and enables faster similarity determination, effectively improving the query-per-second (QPS) rate. Based on each vector distance and a preset distance threshold, the target label is determined from the candidate labels, making the determination of the target label more convenient. Furthermore, the preset distance threshold can be adaptively adjusted according to the actual application scenario, so that the target label determined based on the preset distance threshold can meet the actual needs.

[0128] In this embodiment, firstly, by pre-storing candidate label vectors in a preset database, when the target label needs to be determined, the corresponding label vector can be directly extracted from the preset database. Furthermore, before determining the vector distance between the text vector of the document to be labeled and the label vectors of each candidate label, the document to be labeled only needs to determine its text vector through a single target vector transformation model, which improves the response speed when determining the target label. Secondly, since vector distance can be used to characterize the similarity between a document and a label, determining the target label through vector distance can vectorize the similarity between the document and the label. Compared to using an additional similarity calculation model for similarity calculation, this method can obtain the target label more accurately and quickly. Thirdly, by using a pre-trained target vector transformation model to determine the text vector of the document to be labeled and the label vector of the initial label, the accuracy of determining the target label can be improved while simultaneously increasing the accuracy of text semantic representation.

[0129] In some embodiments, comparing the text content in the document to be annotated with the text content corresponding to each initial tag in a preset database includes:

[0130] The non-text content in the document to be annotated is filtered out to obtain an intermediate document with text content;

[0131] The target document is generated based on the text content in the intermediate document that meets the preset filtering conditions;

[0132] The text content in the target document is compared with the text content corresponding to each of the initial tags.

[0133] Since the documents to be annotated can be original and may contain punctuation marks, emoticons, and other special characters, inaccurate comparison results may occur when comparing text with tags if the document contains non-text content. Furthermore, excessively long documents may lead to the identification of candidate tags that deviate significantly from the key content of the document during text content comparison. Additionally, non-text content may affect the accuracy of text vectors during vector conversion.

[0134] In this embodiment of the disclosure, non-text content in the document to be annotated can be filtered out. That is, the non-text content in the document to be annotated is removed, and the text content in the document to be annotated is integrated in its original order to obtain an intermediate document with text content. The non-text content may include Uniform Resource Locator (URL), special characters, emoticons, etc.

[0135] After obtaining the intermediate document, the target document can be generated based on the text content in the intermediate document that meets preset filtering criteria. In some embodiments, the text content that meets the preset filtering criteria can be extracted according to the order of the text content in the intermediate document to obtain the target document. Here, the preset filtering criteria can be pre-defined conditions. For example, the filtering criteria can be set according to the number of characters to be extracted, or according to the semantics of the text content to be extracted. Taking setting the filtering criteria according to the number of characters to be extracted as an example, the first 256 characters in the intermediate document can be extracted, or the first 192 characters can be extracted, etc. The preset filtering criteria can be adaptively adjusted according to the average number of characters or the maximum number of characters in the document to be annotated, etc., and are not specifically limited here.

[0136] For example, the document to be annotated might be: "Today my friends and I went to the food court to eat hot pot. The hot pot at this restaurant was delicious, and everyone was very satisfied." After filtering out non-standard content, the intermediate document would be: "Today my friends and I went to the food court to eat hot pot. The hot pot at this restaurant was delicious, and everyone was very satisfied." If the preset filter condition is set to extract the first 32 characters, where one Chinese character corresponds to two characters, then according to the preset filter condition, the generated target document could be: "Today my friends and I went to the food court to eat hot pot."

[0137] In this embodiment of the disclosure, once a target document is identified, the text content in the target document can be compared with the text content corresponding to the initial tags. Here, a matching recall module can be used to implement the function of comparing the target document with the initial tags, thereby determining candidate tags that match the target document.

[0138] In this embodiment of the disclosure, by preprocessing the document to be labeled, the problem of inaccurate comparison results caused by non-text content contained in the document or excessive document length can be avoided as much as possible, thereby reducing the impact on the model performance and improving the accuracy when determining text vectors.

[0139] In some embodiments, the method further includes:

[0140] Identify training documents from historical document data;

[0141] The initial labels are preprocessed to obtain candidate training labels corresponding to the initial labels;

[0142] The text content in the training document is compared with the text content corresponding to each of the candidate training labels. Based on the matching degree between the training document and each of the candidate training labels, the target training label corresponding to the training document is selected from the candidate training labels.

[0143] The training document and the target training label are input into the initial vector transformation model for vector transformation processing to obtain the text vector of the training document and the label vector of the target training label;

[0144] The text vector of the training document and the label vector of the target training label are input into the contrastive learning model to obtain the contrastive loss value;

[0145] The model parameters of the initial vector transformation model are adjusted based on the contrastive loss value to obtain the target vector transformation model.

[0146] Here, historical document data refers to accumulated data, which can be historical data stored in memory or historical data retrieved according to preset retrieval criteria. For example, historical document data can be historically published blog posts or historically shared information. Training documents can be randomly selected from the historical document data or determined according to the order of the historical data.

[0147] In some embodiments, the training document can be an original document or a preprocessed document. Preprocessing can involve filtering out non-text content from the training document, extracting content that meets preset filtering criteria from the training document, or extracting content that meets preset filtering criteria after filtering out non-text content from the training document. The preprocessed document can be an intermediate document with text content or a target document. By using the preprocessed document as the training document, the training document can be directly used when comparing the text content in the training document with the text content corresponding to the candidate training labels. Furthermore, since the training document is an original document that has been preprocessed, it can save memory resources when stored in memory as historical document data.

[0148] Here, the initial labels can be preprocessed by expanding them according to preset rules to obtain candidate training labels. For example, the text content corresponding to the initial labels can be expanded by adding modifiers to generate candidate training labels with a fixed number of characters. Alternatively, the initial labels can be expanded based on their semantics to generate candidate training labels containing richer related semantics. In practical applications, the preprocessing method can be adaptively adjusted based on the number of characters and semantics of the initial labels.

[0149] In some embodiments, when candidate training labels are determined, the text content in the training document can be compared with the text content corresponding to each candidate training label to obtain the matching degree between the training document and each candidate training label. Based on the obtained matching degree between the training document and each candidate training label, the target training label corresponding to the training document can be selected from the candidate training labels.

[0150] In some embodiments, a matching recall module can be used to compare training documents with candidate training labels. This matching recall module can be a module that compares the text content of the training document with the text content corresponding to the candidate training labels, or it can be a module that compares the semantic information of the training document with the semantic information of the candidate training labels. Here, the matching recall module can be embedded in the target vector transformation model, or it can be configured as a separate module together with the target vector transformation model in an electronic device or server. There are no specific limitations on the model and configuration of the matching recall module, as long as the comparison function can be achieved.

[0151] In some embodiments, once the target training labels are determined, the training document and the target training labels can be input into an initial vector transformation model for vector transformation processing to obtain the text vector of the training document and the label vector of the target training labels. Here, the initial vector transformation model can be a pre-trained model, which may include: a Transformer-based Bidirectional Encoder Representations from Transformer (BERT) model, a Sentence Embeddings using Siamese BERT Networks (Sentence-BERT) model, a Masked Sequence to Sequence Pre-training (MASS) model, etc.

[0152] For example, Figure 2A This is a structure of a Transformer-based bidirectional encoding representation model illustrated according to an exemplary embodiment. Figure 1 , Figure 2B Figure 2 illustrates the structure of a Transformer-based bidirectional encoding representation model according to an exemplary embodiment. This Transformer-based bidirectional encoding representation model is a natural language processing model containing 12 Transformer encoders and consists of two pre-training tasks: Masked Language Model (MLM) and Next Sentence Prediction (NSP). The Masked Language Model randomly masks a portion of words in a sentence and then simultaneously uses contextual information to predict the masked words, thus improving the understanding of word meaning within the entire text. The Next Sentence Prediction task primarily aims to enable the model to better understand the relationships between sentences.

[0153] like Figure 2A As shown in the diagram, this structure diagram illustrates the pre-training process using a masked language model and a next-sentence prediction task. [CLS] is the identifier used for classification, [SEP] is the identifier used for sentence segmentation, Tok represents different words, E represents the input embedding vector, C represents the vector output by the identifier [CLS], and TN represents the vector output by the Nth word after model processing. During pre-training, sentence pairs are input, including sentence 1 and sentence 2, which are segmented using [SEP].

[0154] like Figure 2BAs shown in the diagram, this structure diagram illustrates the process of fine-tuning a natural language processing (NLP) task. Fine-tuning the model for a specific NLP task means that the pre-trained model can be applied to various NLP tasks for simple adjustments. Here, NLP tasks can include sentence pair classification, single sentence annotation, question answering, etc. In the question answering task, the input is a question-answer pair, which consists of a question and text containing the answer; the output is a start / end sequence. In some embodiments, the model obtained after fine-tuning the pre-trained model can be used as the initial vector transformation model to fully utilize the knowledge of the pre-trained model.

[0155] For example, Figure 3A This is a structure of a sentence embedding model using a Siamese BERT network, as illustrated in an exemplary embodiment. Figure 1 , Figure 3B Figure 2 illustrates the structure of a sentence embedding model using a Siamese BERT network, according to an exemplary embodiment. This model can modify a pre-trained BERT network to generate textual semantic representations (e.g., feature vectors) using a Siamese or triple network. Figure 3A As shown in the diagram, this structure represents the training phase. Feature vectors u and v are extracted from two texts (which can be sentence 1 and sentence 2) using BERT (which may include embedding layers, encoders, pooling layers, etc.). Then, the features of u and v are concatenated and input into the binary classification model. Figure 3B As shown in the diagram, this structure represents the prediction phase, which is the phase where the model is used to determine the similarity of the feature vectors of two texts using the cosine distance formula, i.e., cosine-sim(u,v). In some embodiments, the sentence embedding model using the Siamese BERT network can be used as the initial vector transformation model.

[0156] Given the text vectors of the training documents and the label vectors of the target training labels, the text vectors and label vectors can be input into a contrastive learning model to obtain a contrastive loss value. Based on this contrastive loss value, the model parameters of the initial vector transformation model are adjusted to obtain the target vector transformation model. The contrastive learning model can include: a Simple Contrastive Learning of Sentence Embeddings (SimCSE) model, a Self-supervised learning implemented by contrastive learning (SimCLR) model, etc. The goal of contrastive learning is to bring similar samples closer together and separate dissimilar samples. The target vector transformation model can be obtained by further training a Transformer-based bidirectional encoding representation model using a contrastive learning training approach.

[0157] It should be noted that different contrastive learning models have different contrastive loss functions and different determined contrastive loss values. When adjusting the model parameters of the initial vector transformation model based on the contrastive loss value, it is also necessary to combine the corresponding contrastive learning model.

[0158] In some embodiments, the Transformer-based bidirectional encoding representation model can be fine-tuned using historical document data and initial tags, and the fine-tuned Transformer-based bidirectional encoding representation model can be determined as the initial vector transformation model. In this way, when using the fine-tuned initial vector transformation model, the text vectors and tag vectors obtained through the initial vector transformation model can be more accurate and more suitable for actual use scenarios.

[0159] Methods for obtaining vectors include directly inserting the output with a [CLS] symbol before the text, summing the outputs of all words, or averaging the outputs of all words. However, all of these methods suffer from anisotropy, which can lead to two phenomena: word frequency affecting the distribution of the word vector space and sparsity in the word vector space. The sparsity of low-frequency word vectors can cause errors in the determined vector distances. In scenarios where the initial labels change very frequently, multiple new labels will be added, significantly impacting the robustness of the model and severely hindering system stability.

[0160] The Simple Contrastive Sentence Vector Representation Framework (SCNVFR) is a novel vector representation model that employs contrastive learning. By inputting text vectors and label vectors into the SCNVFR, a contrastive loss value is obtained. Based on this loss value, the Transformer-based bidirectional encoding representation model can be adjusted to obtain the target vector transformation model. Adjusting the initial vector transformation model based on the contrastive loss value obtained from the contrastive learning model effectively addresses the anisotropy problem and reduces the impact on the model's robustness when initial labels change frequently.

[0161] In this embodiment of the disclosure, the text classification problem of determining the target annotation labels for annotating documents is modeled as determining a suitable target vector transformation model, which can effectively improve the processing efficiency of the model. By leveraging the advantages of a deep pre-trained initial vector transformation model, combined with the idea of ​​contrastive learning, and by preprocessing the initial labels to customize the input format, the text semantic representation ability of the model can be improved. It can also effectively improve the processing speed and robustness of the target vector transformation model. At the same time, the idea of ​​contrastive learning can effectively reduce the anisotropy problem that occurs when using a Transformer-based bidirectional encoding representation model as an initial vector transformation model in vector representation, and can significantly improve the accuracy of the target vector transformation model.

[0162] In some embodiments, preprocessing the initial label to obtain candidate training labels corresponding to the initial label includes:

[0163] Based on the tag type of the initial tag, at least one associated tag is determined that is related to the initial tag; wherein the associated tag is the same as the object labeled by the initial tag, and the labeling range of the associated tag is greater than the labeling range of the initial tag;

[0164] Based on the annotation range of the initial label and the annotation range of each associated label, determine the splicing order of the initial label and each associated label;

[0165] According to the splicing order, the initial label and each of the associated labels are spliced ​​together to obtain the candidate training label corresponding to the initial label.

[0166] In some embodiments, a tag type can represent the level of a tag, meaning tags can be categorized according to their level, where the tag level represents different hierarchical levels of the tag. For example, if the initial tag is determined to be a first tag type, it can be identified as a first-level tag; if the initial tag is determined to be a second tag type, it can be identified as a second-level tag. In other embodiments, a tag type can also represent the semantics of a tag, meaning tags can be categorized according to their semantics. Associated tags and initial tags can be associated with each other at the tag level or at the tag semantic level.

[0167] In some embodiments, the object labeled by the tag can be any object. For example, it can be a document, a thing, a person, or a phenomenon. In this embodiment of the disclosure, the object labeled by the associated tag is the same as the object labeled by the initial tag. For example, if the object labeled by the initial tag is a first document, then the object labeled by the associated tag is also the first document. If the object labeled by the initial tag is a second document, then the object labeled by the associated tag is also the second document. As another example, if the object labeled by the initial tag is an apple, then the object labeled by the associated tag is also an apple.

[0168] The annotation range can be either the size of the tag level or the scope of the semantic representation of the tag. Taking the annotation range as the size of the tag level, and the annotation range of the initial tag being smaller than the annotation range of the associated tags, as an example, the annotation range of the initial tag is a specific type of fruit, while the annotation range of the associated tags can be all fruits. In some embodiments, the annotation range of the initial tag can also be larger than the annotation range of the associated tags. The size relationship between the annotation range of the initial tag and the annotation range of the associated tags can be adaptively adjusted according to actual needs.

[0169] Here, taking the tag type to represent the tag level as an example, if the tag level of the initial tag is determined to be a level 3 tag based on the tag type of the initial tag, since the text of the level 3 tag is usually very short, such as 3-5 characters, at least one associated tag can be determined based on the tag type of the level 3 tag.

[0170] In some embodiments, an associated label can be determined based on the initial label. The associated label can be a secondary label associated with the tertiary label. For example, based on the tertiary label being A (car), the secondary label can be determined to be a car brand. Both the secondary label and the initial label refer to cars, and the labeling range of the secondary label is greater than that of the initial label.

[0171] In other embodiments, two associated labels can be determined based on the initial label. These two associated labels can be a second-level label and a first-level label associated with the third-level label. For example, based on the third-level label "A car", the second-level label can be determined as "car brand" and the first-level label as "car". The objects labeled by the first-level label, the second-level label and the initial label are all cars, and the labeling range of the first-level label is larger than that of the second-level label, and the labeling range of the second-level label is larger than that of the third-level label.

[0172] It should be noted that after determining the annotation range of the initial label and the annotation range of each associated label, the concatenation order of the initial label and each associated label can be determined based on the annotation range of the initial label and each associated label. Then, the initial label and each associated label are concatenated according to the concatenation order to obtain the candidate training label corresponding to the initial label.

[0173] Taking an initial label as a third-level label and associated labels including first-level and second-level labels as an example, in some embodiments, if the initial label and associated labels are concatenated in ascending order of annotation range, since the annotation range of the first-level label is larger than that of the second-level label, and the annotation range of the second-level label is larger than that of the third-level label, the second-level label can be placed before the first-level label, and the third-level label can be placed before the second-level label. For example, if the third-level label is "A Car", the second-level label is "Car Brand", and the first-level label is "Car", and underscores are used to separate the first-level, second-level, and third-level labels, the candidate training label obtained after concatenation can be "A Car_Car Brand_Car".

[0174] In other embodiments, if the initial label and associated labels are concatenated in descending order of annotation range, since the annotation range of the first-level label is larger than that of the second-level label, and the annotation range of the second-level label is larger than that of the third-level label, the first-level label can be placed before the second-level label, and the second-level label can be placed before the third-level label. For example, if the third-level label is "Car A", the second-level label is "Car Brand", and the first-level label is "Car", and underscores are used to separate the first-level, second-level, and third-level labels, the resulting candidate training label could be "Car_Car Brand_Car A".

[0175] In some embodiments, the method of preprocessing the initial labels can be adaptively adjusted according to the number and characteristics of the initial labels in the actual business scenario, so that the candidate labels obtained after preprocessing can be more accurately and effectively applied to the actual application scenario.

[0176] In some embodiments, candidate training labels obtained after preprocessing the initial labels can be stored in memory. In this way, when the same candidate training label needs to be used multiple times, it can be directly obtained from memory without processing the initial labels again to obtain candidate training labels.

[0177] In this embodiment of the disclosure, candidate training labels are obtained by concatenating the initial labels according to the label type of the initial labels. The initial labels can be appropriately expanded to introduce richer semantic information, which can more accurately identify the labels corresponding to the documents. When the candidate training labels are input into the model, the accuracy of the model can be improved.

[0178] In some embodiments, the target training label corresponding to the training document includes: positive sample label and negative sample label; the step of inputting the text vector of the training document and the label vector of the target training label into the contrastive learning model to obtain the contrastive loss value includes:

[0179] Identify similar documents from the historical document data that have a similarity greater than a preset similarity threshold with the training documents;

[0180] Using the similarity sub-model in the contrastive learning model, the first similarity between the training document and the positive sample label is determined based on the text vector of the training document and the label vector of the positive sample label.

[0181] Using the similarity sub-model, a second similarity between the training document and the negative sample label is determined based on the text vector of the training document and the label vector of the negative sample label;

[0182] The contrast loss value is determined based on the first similarity, the second similarity, and the third similarity between the training document and the similar document; wherein the contrast loss value is negatively correlated with the third similarity and positively correlated with the sum of the first similarity and the second similarity.

[0183] Here, we take a simple contrastive sentence vector representation framework model as an example, which can include both unsupervised and supervised contrastive learning methods. In unsupervised contrastive learning, dropout technology is used to augment the original text (i.e., the training document) to construct positive examples for subsequent contrastive learning training. In supervised contrastive learning, a premise assumption in the natural language inference dataset is that the text has corresponding implied and contradictory texts. Using the natural language inference dataset, the implied text can be used as positive examples, and the contradictory text as difficult negative examples for subsequent contrastive learning training.

[0184] Figure 4 This is a structural diagram illustrating the correspondence between training documents and target training labels according to an exemplary embodiment, such as... Figure 4 As shown, target training label 1 corresponding to training document 1 is the implied text, and target training label 2, target training label 3, ..., target training label N are contradictory texts obtained after filtering target training label 1. Among them, training document 2, ..., training document N are training documents to be processed. The training documents and target training labels can be vector-converted using the target vector conversion model 301 to obtain the text vector of the training document and the label vector of the target training label.

[0185] Taking a contrastive learning model as an example of supervised contrastive learning, in some embodiments, similar documents (i.e., implied text) with a similarity greater than a preset similarity threshold to the training document can be identified from historical document data as positive examples. The preset similarity threshold can be 90%, and can be adaptively adjusted according to actual conditions. In other embodiments, a positive sample label can be selected from the positive sample labels according to preset instructions as a positive example.

[0186] In some embodiments, using a similarity sub-model in a contrastive learning model, a first similarity between a training document and positive sample labels can be determined based on the text vector of the training document and the label vector of the positive sample labels; using the similarity sub-model, a second similarity between a training document and negative sample labels can be determined based on the text vector of the training document and the label vector of the negative sample labels; wherein, positive sample labels can be target training labels with a high degree of matching with the training document, or target training labels determined using a natural language inference dataset; negative sample labels can be target training labels with a low degree of matching with the training document, or target training labels determined using a natural language inference dataset.

[0187] For example, if the matching degree between the target training label and the training document reaches 65%, it can be determined that the matching degree between the target training label and the training document is high, and the target training label with a matching degree of 65% with the training document is identified as a positive sample label. If the matching degree between the target training label and the training document is less than 15%, it can be determined that the matching degree between the target training label and the training document is low, and the target training label with a matching degree of less than 15% with the training document is identified as a negative sample label. In some embodiments, the matching degree corresponding to the positive sample label and the matching degree corresponding to the negative sample label can be adaptively adjusted according to the actual situation.

[0188] It should be noted that here, positive and negative sample labels can represent the matching degree between the training document and the target document. Negative sample labels can serve as difficult negative examples in this supervised contrastive learning model, and positive sample labels can serve as negative examples in the same model. The first similarity between the training document and the positive sample labels can include: after removing positive examples from the positive sample labels (assuming a positive sample label is selected as a positive example according to a preset instruction), the first similarity between the remaining positive sample labels and the training document. The second similarity between the training document and the negative sample labels can include: the second similarity between the target training label and the training document with a low matching degree, for example, the second similarity between the target training label and the training document with a matching degree of less than 15%.

[0189] The contrastive loss value can be determined based on the first similarity between the training document and positive sample labels, the second similarity between the training document and negative sample labels, and the third similarity between the training document and similar documents. The third similarity is determined by comparing the text vectors of the training document and the text vectors of similar documents. The contrastive loss value is negatively correlated with the third similarity and positively correlated with the sum of the first and second similarities. Taking this contrastive learning model as a supervised, simple contrastive sentence vector representation framework model as an example, the formula for calculating the contrastive loss value is as follows:

[0190]

[0191] In formula (2), loss is the contrast loss value, sim is the similarity sub-model, τ is the temperature hyperparameter, and h i h is the text vector of the training document. i + For text vectors of similar documents, sim(h) i ,h i + ) represents the third similarity, h j + h is the label vector of the positive sample labels. j - The label vector of the negative sample labels, sim(h) i ,h j + ) represents the first similarity, sim(h) i ,h j - The second similarity is 0.

[0192] In some embodiments, the similarity between training documents and positive examples can be maximized, while the similarity between training documents and negative examples and difficult negative examples can be minimized, thereby achieving the goal of minimizing the contrastive loss value.

[0193] In this embodiment of the disclosure, by minimizing the contrast loss value, that is, maximizing the third similarity and minimizing the sum between the first similarity and the second similarity, the model parameters of the initial vector transformation model are adjusted according to the contrast loss value. The vector distribution obtained based on the target vector transformation model is more uniform, which can also effectively solve the anisotropy problem.

[0194] In some embodiments, the method further includes:

[0195] Candidate training tags with a matching degree greater than a preset matching degree to the training document are identified as visible tags;

[0196] Candidate training labels whose matching degree with the training document is less than or equal to the preset matching degree are determined as invisible labels;

[0197] According to a preset labeling strategy, a first number of visible labels are determined as positive sample labels, and a second number of visible labels are determined as negative sample labels;

[0198] Based on the first quantity and the second quantity, a third quantity of the negative sample labels is determined from the invisible labels;

[0199] The ratio between the sum of the second quantity and the third quantity and the first quantity satisfies a preset proportional relationship.

[0200] In some embodiments, candidate training labels that satisfy a preset annotation strategy can be used as positive sample labels, while other candidate training labels that do not satisfy the preset annotation strategy can be used as negative sample labels. However, directly using all candidate training labels that do not satisfy the preset annotation strategy as negative sample labels will increase the training difficulty of the model and cause the model to fail to converge.

[0201] The preset annotation strategy can be a pre-configured annotation strategy in the matching recall module or a pre-configured annotation strategy in the annotation module. The preset annotation strategy can be based on matching degree or on percentage. For example, with a preset annotation strategy of 70% matching degree, candidate training labels with a matching degree of 70% or higher with the training document can be identified as positive sample labels, and candidate training labels with a matching degree lower than 70% can be identified as negative sample labels. As another example, with a preset annotation strategy of 30% percentage, the matching degrees between each candidate training label and the training document can be sorted in descending order, with the top 30% of candidate training labels identified as positive sample labels and the bottom 30% identified as negative sample labels.

[0202] Here, candidate training labels with a matching degree greater than a preset matching degree to the training document can be identified as visible labels, while candidate training labels with a matching degree less than or equal to the preset matching degree to the training document can be identified as invisible labels. In some embodiments, an online filtering module can be used to determine visible and invisible labels. The preset matching degree can be pre-configured in the online filtering module, or it can be adaptively adjusted during actual application based on the number of candidate training labels, the matching degree between candidate training labels and training documents, etc.

[0203] In other embodiments, when the matching recall module selects the target training label corresponding to the training document from the candidate training labels based on the matching degree between the obtained training document and each candidate training label, the target training labels with a matching degree greater than the matching threshold are determined as visible labels, and the target training labels with a matching degree less than or equal to the matching threshold are determined as invisible labels.

[0204] In some embodiments, a matching threshold can be configured in the matching recall module, or it can be configured through the online filtering module after the matching recall module has run, to determine visible and invisible tags. The matching threshold can be pre-configured in the matching recall module or the online filtering module, or it can be adaptively adjusted during actual application based on the number of target training tags, the matching degree between the target training tags and the training documents, etc. It should be noted that the preset matching degree can be equal to or less than the matching threshold, and the magnitude of the preset matching degree and the matching threshold can be adjusted according to actual needs.

[0205] For example, after running the matching recall module, 10 target training labels can be identified. The online filtering module can output the 5 labels among the target training labels whose matching degree with the training document is greater than the matching threshold, and these 5 output labels are identified as visible labels; the other 5 labels whose matching degree is less than or equal to the preset matching degree are identified as invisible labels.

[0206] Given visible and invisible labels, a first number of visible labels can be designated as positive sample labels, and a second number as negative sample labels, according to a preset labeling strategy. This preset labeling strategy can include conditions for labeling visible labels based on matching degree, and can also include the number of positive and negative sample labels to be determined. For example, according to the preset labeling strategy, two visible labels out of five that satisfy the strategy can be designated as positive sample labels, and the three visible labels out of five that do not satisfy the strategy can be designated as negative sample labels.

[0207] In some embodiments, after determining a first number of positive sample labels and a second number of negative sample labels, a third number of negative sample labels are further determined from the invisible labels; wherein the ratio between the sum of the second and third numbers and the first number satisfies a preset ratio relationship. For example, if the first number is 2 and the second number is 3, that is, 2 positive sample labels and 3 negative sample labels can be determined, and the preset ratio relationship is 3:1, then the third number can be determined as 3, that is, 3 negative sample labels are determined from the invisible labels. Here, negative sample labels can be determined from the invisible labels by random sampling.

[0208] The preset ratio can be determined by combining the number of initial labels and the number of target training labels. It can be used to reduce the problem of excessive negative labels caused by the uncontrollable number of negative labels, which increases the difficulty of model training.

[0209] Figure 5 This is a schematic diagram illustrating the determination of visible labels and positive sample labels among candidate training labels according to an exemplary embodiment, such as... Figure 5 As shown, candidate training labels 401 include visible labels 402, and visible labels 402 include positive sample labels 403.

[0210] In this embodiment of the disclosure, positive and negative sample labels are determined by a preset matching degree and a preset labeling strategy. A third number of target training labels are randomly selected from the invisible labels as negative sample labels. The ratio between negative and positive sample labels is adjusted to meet a preset ratio relationship. The negative and positive sample labels that meet the preset ratio relationship are determined as training data. In this way, the training difficulty of the model can be reduced and the problem of model non-convergence caused by the construction of training data can be avoided.

[0211] Figure 6 This is a block diagram illustrating a text processing apparatus according to an exemplary embodiment. Figure 6 As shown, the text processing device 500 mainly includes:

[0212] The first comparison module 501 is configured to compare the text content in the document to be annotated with the text content corresponding to each initial tag in a preset database, and select candidate annotation tags corresponding to the document to be annotated from the initial tags based on the obtained matching degree between the document to be annotated and each initial tag; wherein, the preset database pre-stores: each initial tag and the tag vector corresponding to each initial tag, and the tag vector is obtained by vector conversion of the initial tags by a target vector conversion model;

[0213] Extraction module 502 is configured to extract the tag vector of the candidate labeled tags from the preset database;

[0214] The first conversion module 503 is configured to perform vector conversion on the document to be annotated based on the target vector conversion model to obtain the text vector of the document to be annotated.

[0215] The first determining module 504 is configured to determine the vector distance between the text vector of the document to be annotated and the tag vector of each of the candidate annotation tags;

[0216] The second determining module 505 is configured to determine a target label from the candidate labeling labels based on the distances between the vectors; wherein the target labeling label is used to label the document to be labeled.

[0217] In some embodiments, the first comparison module 501 is configured as follows:

[0218] The non-text content in the document to be annotated is filtered out to obtain an intermediate document with text content;

[0219] The target document is generated based on the text content in the intermediate document that meets the preset filtering conditions;

[0220] The text content in the target document is compared with the text content corresponding to each of the initial tags.

[0221] In some embodiments, the apparatus further includes:

[0222] The third determination module is configured to determine training documents from historical document data;

[0223] The processing module is configured to preprocess the initial label to obtain candidate training labels corresponding to the initial label;

[0224] The second comparison module is configured to compare the text content in the training document with the text content corresponding to each of the candidate training tags, and select the target training tag corresponding to the training document from the candidate training tags based on the matching degree between the training document and each of the candidate training tags.

[0225] The second conversion module is configured to input the training document and the target training label into an initial vector conversion model for vector conversion processing to obtain the text vector of the training document and the label vector of the target training label;

[0226] The input module is configured to input the text vector of the training document and the label vector of the target training label into the contrastive learning model to obtain the contrastive loss value;

[0227] The adjustment module is configured to adjust the model parameters of the initial vector transformation model based on the contrast loss value to obtain the target vector transformation model.

[0228] In some embodiments, the processing module is configured to:

[0229] Based on the tag type of the initial tag, at least one associated tag is determined that is related to the initial tag; wherein the associated tag is the same as the object labeled by the initial tag, and the labeling range of the associated tag is greater than the labeling range of the initial tag;

[0230] Based on the annotation range of the initial label and the annotation range of each associated label, determine the splicing order of the initial label and each associated label;

[0231] According to the splicing order, the initial label and each of the associated labels are spliced ​​together to obtain the candidate training label corresponding to the initial label.

[0232] In some embodiments, the target training labels corresponding to the training document include: positive sample labels and negative sample labels; the input module is configured to:

[0233] Identify similar documents from the historical document data that have a similarity greater than a preset similarity threshold with the training documents;

[0234] Using the similarity sub-model in the contrastive learning model, the first similarity between the training document and the positive sample label is determined based on the text vector of the training document and the label vector of the positive sample label.

[0235] Using the similarity sub-model, a second similarity between the training document and the negative sample label is determined based on the text vector of the training document and the label vector of the negative sample label;

[0236] The contrast loss value is determined based on the first similarity, the second similarity, and the third similarity between the training document and the similar document; wherein the contrast loss value is negatively correlated with the third similarity and positively correlated with the sum of the first similarity and the second similarity.

[0237] In some embodiments, the apparatus further includes:

[0238] The fourth determining module is configured to determine candidate training tags that have a matching degree greater than a preset matching degree with the training document as visible tags;

[0239] The fifth determining module is configured to determine candidate training labels whose matching degree with the training document is less than or equal to the preset matching degree as invisible labels;

[0240] The sixth determining module is configured to determine a first number of visible labels as positive sample labels and a second number of visible labels as negative sample labels according to a preset labeling strategy;

[0241] The seventh determining module is configured to determine a third number of negative sample labels from the invisible labels based on the first number and the second number;

[0242] The ratio between the sum of the second quantity and the third quantity and the first quantity satisfies a preset proportional relationship.

[0243] In some embodiments, the second determining module 505 is configured to:

[0244] The candidate labels whose vector distance is greater than a preset distance threshold are determined as the target labels;

[0245] The vector distance is used to characterize the similarity between the document to be labeled and the candidate labels.

[0246] Figure 7 This is a block diagram illustrating a text processing apparatus 800 according to an exemplary embodiment. For example, apparatus 800 may be a mobile phone, a mobile computer, etc.

[0247] Reference Figure 7 The device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0248] Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0249] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of such data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0250] Power supply component 806 provides power to various components of device 800. Power supply component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 800.

[0251] Multimedia component 808 includes a screen that provides an output interface between device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0252] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0253] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0254] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of device 800, changes in the position of device 800 or a component of device 800, the presence or absence of user contact with device 800, the orientation or acceleration / deceleration of device 800, and temperature changes of device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0255] Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices. Device 800 can access wireless networks based on communication standards, such as Wi-Fi, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0256] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0257] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of the device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0258] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a text processing device, enables the text processing device to perform a text processing method, the method comprising:

[0259] The text content in the document to be annotated is compared with the text content corresponding to each initial tag in the preset database. Based on the matching degree between the document to be annotated and each initial tag, candidate annotation tags corresponding to the document to be annotated are selected from the initial tags. The preset database stores each initial tag and the tag vector corresponding to each initial tag. The tag vector is obtained by vector conversion of the initial tags by the target vector conversion model.

[0260] Extract the label vectors of the candidate labels from the preset database;

[0261] Based on the target vector transformation model, the document to be labeled is transformed into a vector to obtain the text vector of the document to be labeled;

[0262] Determine the vector distance between the text vector of the document to be annotated and the tag vector of each of the candidate annotation tags;

[0263] Based on the distances of each vector, a target label is determined from the candidate labels; wherein the target label is used to label the document to be labeled.

[0264] Figure 8 This is a hardware structure block diagram of a text processing device according to an exemplary embodiment. For example, electronic device 1900 may be provided as a server. (Refer to...) Figure 8 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform a text processing method, including:

[0265] The text content in the document to be annotated is compared with the text content corresponding to each initial tag in the preset database. Based on the matching degree between the document to be annotated and each initial tag, candidate annotation tags corresponding to the document to be annotated are selected from the initial tags. The preset database stores each initial tag and the tag vector corresponding to each initial tag. The tag vector is obtained by vector conversion of the initial tags by the target vector conversion model.

[0266] Extract the label vectors of the candidate labels from the preset database;

[0267] Based on the target vector transformation model, the document to be labeled is transformed into a vector to obtain the text vector of the document to be labeled;

[0268] Determine the vector distance between the text vector of the document to be annotated and the tag vector of each of the candidate annotation tags;

[0269] Based on the distances of each vector, a target label is determined from the candidate labels; wherein the target label is used to label the document to be labeled.

[0270] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0271] It should be understood that the phrase "an embodiment" or "one embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this disclosure. Therefore, "in one embodiment" or "one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this disclosure, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure. The sequence numbers of the above-described embodiments are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0272] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0273] In the several embodiments provided in this disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0274] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0275] In addition, each functional unit in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0276] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0277] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A text processing method, characterized in that, include: The text content in the document to be annotated is compared with the text content corresponding to each initial tag in the preset database. Based on the matching degree between the document to be annotated and each initial tag, candidate annotation tags corresponding to the document to be annotated are selected from the initial tags. The preset database stores each initial tag and the tag vector corresponding to each initial tag. The tag vector is obtained by vector conversion of the initial tags by the target vector conversion model. Extract the label vectors of the candidate labels from the preset database; Based on the target vector transformation model, the document to be labeled is transformed into a vector to obtain the text vector of the document to be labeled; Determine the vector distance between the text vector of the document to be annotated and the tag vector of each of the candidate annotation tags; Based on the vector distances, a target label is determined from the candidate labels; wherein the target label is used to label the document to be labeled. The method further includes: Identify training documents from historical document data; The initial labels are preprocessed to obtain candidate training labels corresponding to the initial labels; The text content in the training document is compared with the text content corresponding to each of the candidate training labels. Based on the matching degree between the training document and each of the candidate training labels, the target training label corresponding to the training document is selected from the candidate training labels. The training document and the target training label are input into the initial vector transformation model for vector transformation processing to obtain the text vector of the training document and the label vector of the target training label; The text vector of the training document and the label vector of the target training label are input into the contrastive learning model to obtain the contrastive loss value; The model parameters of the initial vector transformation model are adjusted based on the contrastive loss value to obtain the target vector transformation model.

2. The method according to claim 1, characterized in that, The step of comparing the text content in the document to be annotated with the text content corresponding to each initial tag in the preset database includes: The non-text content in the document to be annotated is filtered out to obtain an intermediate document with text content; The target document is generated based on the text content in the intermediate document that meets the preset filtering conditions; The text content in the target document is compared with the text content corresponding to each of the initial tags.

3. The method according to claim 1, characterized in that, The step of preprocessing the initial label to obtain candidate training labels corresponding to the initial label includes: Based on the tag type of the initial tag, at least one associated tag is determined that is related to the initial tag; wherein the associated tag is the same as the object labeled by the initial tag, and the labeling range of the associated tag is greater than the labeling range of the initial tag; Based on the annotation range of the initial label and the annotation range of each associated label, determine the splicing order of the initial label and each associated label; According to the splicing order, the initial label and each of the associated labels are spliced ​​together to obtain the candidate training label corresponding to the initial label.

4. The method according to claim 1, characterized in that, The target training labels corresponding to the training document include: positive sample labels and negative sample labels; the step of inputting the text vector of the training document and the label vector of the target training labels into the contrastive learning model to obtain the contrastive loss value includes: Identify similar documents from the historical document data that have a similarity greater than a preset similarity threshold with the training documents; Using the similarity sub-model in the contrastive learning model, the first similarity between the training document and the positive sample label is determined based on the text vector of the training document and the label vector of the positive sample label. Using the similarity sub-model, a second similarity between the training document and the negative sample label is determined based on the text vector of the training document and the label vector of the negative sample label; The contrast loss value is determined based on the first similarity, the second similarity, and the third similarity between the training document and the similar document; wherein the contrast loss value is negatively correlated with the third similarity and positively correlated with the sum of the first similarity and the second similarity.

5. The method according to claim 4, characterized in that, The method further includes: Candidate training tags with a matching degree greater than a preset matching degree to the training document are identified as visible tags; Candidate training labels whose matching degree with the training document is less than or equal to the preset matching degree are determined as invisible labels; According to a preset labeling strategy, a first number of visible labels are determined as positive sample labels, and a second number of visible labels are determined as negative sample labels; Based on the first quantity and the second quantity, a third quantity of the negative sample labels is determined from the invisible labels; The ratio between the sum of the second quantity and the third quantity and the first quantity satisfies a preset proportional relationship.

6. The method according to any one of claims 1 to 5, characterized in that, The step of determining the target label from the candidate labels based on the distances of each of the vectors includes: The candidate labels whose vector distance is greater than a preset distance threshold are determined as the target labels; The vector distance is used to characterize the similarity between the document to be labeled and the candidate labels.

7. A text processing device, characterized in that, include: The first comparison module is configured to compare the text content in the document to be annotated with the text content corresponding to each initial tag in a preset database, and select candidate annotation tags corresponding to the document to be annotated from the initial tags based on the matching degree between the document to be annotated and each initial tag; wherein, the preset database pre-stores: each initial tag and the tag vector corresponding to each initial tag, and the tag vector is obtained by vector conversion of the initial tags by a target vector conversion model; The extraction module is configured to extract the label vectors of the candidate labeled labels from the preset database; The first conversion module is configured to perform vector conversion on the document to be annotated based on the target vector conversion model to obtain the text vector of the document to be annotated; The first determining module is configured to determine the vector distance between the text vector of the document to be annotated and the tag vector of each of the candidate annotation tags; The second determining module is configured to determine a target label from the candidate labeling labels based on the distances between the vectors; wherein the target labeling label is used to label the document to be labeled; The third determination module is configured to determine training documents from historical document data; The processing module is configured to preprocess the initial label to obtain candidate training labels corresponding to the initial label; The second comparison module is configured to compare the text content in the training document with the text content corresponding to each of the candidate training tags, and select the target training tag corresponding to the training document from the candidate training tags based on the matching degree between the training document and each of the candidate training tags. The second conversion module is configured to input the training document and the target training label into an initial vector conversion model for vector conversion processing to obtain the text vector of the training document and the label vector of the target training label; The input module is configured to input the text vector of the training document and the label vector of the target training label into the contrastive learning model to obtain the contrastive loss value; The adjustment module is configured to adjust the model parameters of the initial vector transformation model based on the contrast loss value to obtain the target vector transformation model.

8. A text processing device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the steps of any one of the text processing methods of claims 1 to 6 during execution.

9. A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a text processing device, the device is able to perform the steps of any one of the text processing methods of claims 1 to 6.

Images