Text data processing method, device and equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a text data processing method and device, equipment and a computer readable storage medium. The method comprises the steps that a first to-be-verified text containing a plurality of first texts is obtained, each first text corresponds to a first text label, and the first text labels are used for representing correlation between the first texts and a preset theme; a second to-be-verified text containing a plurality of second texts is obtained, each second text corresponds to a second text label, and the second text labels are used for representing correlation between the second texts and a preset theme; extracting a target first text from the plurality of first texts based on the first text label, and extracting a target second text from the plurality of second texts based on the second text label; and based on the similarity between the target first text and the target second text, determining the similarity between the first to-be-verified text and the second to-be-verified text. Through the method, the calculation accuracy and reliability of the text similarity can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and specifically to a text data processing method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the rapid rise of internet technology, users or media can publish text (such as articles) on different platforms on the internet. In order to facilitate the management of various texts on the internet, it is usually necessary to calculate the similarity between different texts.

[0003] When calculating the similarity between different texts, the relevant technologies perform similarity matching on different texts, which results in low accuracy and efficiency in text similarity calculation.

[0004] Therefore, improving the accuracy and efficiency of text similarity calculation to enhance the reliability of text data processing is an urgent problem to be solved. Summary of the Invention

[0005] To address the aforementioned technical problems, embodiments of this application provide a text data processing method, apparatus, device, and computer-readable storage medium.

[0006] The technical solution adopted in this application is as follows:

[0007] A text data processing method, comprising:

[0008] Obtain a first text to be verified containing multiple first texts, each first text corresponding to a first text tag, the first text tag being used to characterize the relevance of the first text to a preset topic;

[0009] Obtain a second text to be verified that contains multiple second texts, each of which has a corresponding second text tag. The second text tag is used to characterize the relevance of the second text to the preset topic.

[0010] The target first text is extracted from the plurality of first texts based on the first text tag, and the target second text is extracted from the plurality of second texts based on the second text tag;

[0011] Based on the similarity between the first target text and the second target text, the similarity between the first text to be verified and the second text to be verified is determined.

[0012] A text data processing device, comprising:

[0013] The acquisition unit is used to acquire a first text to be verified containing multiple first texts, each first text corresponding to a first text tag, and the first text tag is used to characterize the relevance of the first text to a preset topic;

[0014] The acquisition unit is further configured to acquire a second text to be verified containing multiple second texts, each second text corresponding to a second text tag, the second text tag being used to characterize the relevance between the second text and the preset topic;

[0015] An extraction unit is configured to extract target first text from the plurality of first texts based on the first text tag, and to extract target second text from the plurality of second texts based on the second text tag;

[0016] The processing unit is configured to determine the similarity between the first text to be verified and the second text to be verified based on the similarity between the first target text and the second target text.

[0017] In one embodiment of this application, based on the foregoing scheme, the calculation unit is configured to calculate the similarity between each first text and the preset topic, obtain a first similarity corresponding to each first text, and generate a first text tag corresponding to each first text based on the first similarity of each first text; and to calculate the similarity between each second text and the preset topic, obtain a second similarity corresponding to each second text, and generate a second text tag corresponding to each second text based on the second similarity of each second text.

[0018] In one embodiment of this application, based on the foregoing scheme, the processing unit is further configured to: if a first text with a first similarity greater than a first preset similarity threshold is detected, generate a first text tag representing that the detected first text is related to the preset topic; if a first text with a first similarity less than or equal to the first preset similarity threshold is detected, generate a first text tag representing that the detected first text is not related to the preset topic; if a second text with a second similarity greater than a second preset similarity threshold is detected, generate a second text tag representing that the detected second text is related to the preset topic; if a second text with a second similarity less than or equal to the second preset similarity threshold is detected, generate a second text tag representing that the detected second text is not related to the preset topic.

[0019] In one embodiment of this application, based on the foregoing scheme, the extraction unit is further configured to extract from the plurality of first texts a first text corresponding to a first text tag that represents a correlation with the preset topic, and to use the extracted first text as the target first text; and to extract from the plurality of second texts a second text corresponding to a second text tag that represents a correlation with the preset topic, and to use the extracted second text as the target second text.

[0020] In one embodiment of this application, based on the foregoing scheme, the acquisition unit is further configured to acquire the target first text feature corresponding to the target first text and the target second text feature corresponding to the target second text; the processing unit is further configured to concatenate the target first text feature and the target second text feature to obtain concatenated text features; and determine the similarity between the first text to be verified and the second text to be verified based on the concatenated text features.

[0021] In one embodiment of this application, based on the foregoing scheme, the calculation unit is further configured to: if there are multiple target first text features, calculate the average of the multiple target first text features to obtain a first average text feature; if there are multiple target second text features, calculate the average of the multiple target second text features to obtain a second average text feature; the processing unit is further configured to: concatenate the first average text feature and the second average text feature to obtain a concatenated text feature.

[0022] In one embodiment of this application, based on the foregoing scheme, the calculation unit is further configured to calculate the text feature difference between the target first text feature and the target second text feature; the processing unit is further configured to concatenate the target first text feature, the target second text feature and the text feature difference to obtain the concatenated text feature.

[0023] In one embodiment of this application, based on the aforementioned scheme, the similarity between the first text to be verified and the second text to be verified is determined by a text matching model; the filtering model is trained in the following manner: the acquisition unit is further configured to acquire a first training sample containing multiple first samples, each first sample corresponding to a first sample label, the first sample label being used to characterize the relevance between the first sample and the sample topic; acquire a second training sample containing multiple second samples, each second sample corresponding to a second sample label, the second sample label being used to characterize the relevance between the second sample and the sample topic; the calculation unit is further configured to calculate the predicted similarity between the first training sample and the second training sample based on the first sample label and the second sample label; the processing unit is further configured to adjust the model parameters of the model to be trained based on the input similarity between the first training sample and the second training sample, and the loss between the predicted similarities, to obtain the text matching model.

[0024] In one embodiment of this application, based on the foregoing scheme, before obtaining the first text to be verified containing multiple first texts, the obtaining unit is further configured to obtain multiple texts to be verified, and attribute information of each text to be verified, the attribute information including at least one of title information and source information; the processing unit is further configured to obtain the first text to be verified and the second text to be verified associated with the preset topic from the multiple texts to be verified based on the attribute information of each text to be verified.

[0025] A text data processing device includes a processor and a memory, wherein the memory stores computer-readable instructions, and the computer-readable instructions are executed by the processor to implement the text data processing method described above.

[0026] A computer-readable storage medium having stored computer-readable instructions thereon, which, when executed by a computer's processor, cause the computer to perform the text data processing method described above.

[0027] A computer program product includes computer-readable instructions that, when executed by a processor, implement the text data processing method described above.

[0028] In the above technical solution:

[0029] The system can obtain a first text to be verified and a second text to be verified. The first text to be verified contains multiple first texts, each with a corresponding first text tag. Similarly, the second text to be verified contains multiple second texts, each with a corresponding first text tag. Text tags characterize the relevance between the text and a preset topic. Thus, the system can extract the target first text from the multiple first texts based on the first text tags, and extract the target second text from the multiple second texts based on the second text tags, to determine the similarity between the first and second texts to be verified.

[0030] This method filters out first and second texts that are irrelevant to the preset topic, and uses texts that are relevant to the preset topic to determine the similarity between the first and second texts to be verified. This improves the accuracy of text similarity calculation, reduces the impact of irrelevant content on similarity calculation, and also improves the efficiency of similarity calculation. Therefore, the reliability of text data processing is high.

[0031] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0033] Figure 1 This is a schematic diagram of one implementation environment involved in this application;

[0034] Figure 2 This is a flowchart illustrating a text data processing method according to an exemplary embodiment;

[0035] Figure 3 This is a flowchart illustrating a text data processing method according to another exemplary embodiment.

[0036] Figure 4 This is a flowchart illustrating a text data processing method according to another exemplary embodiment;

[0037] Figure 5 This is a flowchart illustrating a text data processing method according to another exemplary embodiment;

[0038] Figure 6a This is a schematic diagram illustrating the text similarity calculation process involved in this application;

[0039] Figure 6b This is a schematic diagram illustrating another text similarity calculation process involved in this application;

[0040] Figure 7 This is a flowchart illustrating a text data processing method according to another exemplary embodiment;

[0041] Figure 8a This is a schematic diagram illustrating another text similarity calculation process involved in this application;

[0042] Figure 8b This is a schematic diagram illustrating another text similarity calculation process involved in this application;

[0043] Figure 9 This is a flowchart illustrating a text data processing method according to another exemplary embodiment;

[0044] Figure 10 This is a diagram illustrating the training data needed to train a text matching model.

[0045] Figure 11 This is a block diagram illustrating a text data processing apparatus according to an exemplary embodiment;

[0046] Figure 12 This is a schematic diagram of the structure of a computer system for a text data processing device according to an exemplary embodiment. Detailed Implementation

[0047] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments identical to those described in this application. Rather, they are merely examples of apparatuses and methods identical to some aspects of this application as detailed in the appended claims.

[0048] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0049] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, while others can be integrated or partially integrated; therefore, the actual execution order may change depending on the specific circumstances.

[0050] It should be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0051] It should be noted that, in the specific embodiments of this application, user-related data is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. Furthermore, the formulas involved in the embodiments of this application can be flexibly adjusted, for example, by adding or reducing corresponding parameters.

[0052] Before introducing the technical solutions of the embodiments of this application, let's first introduce the technical terms involved in the embodiments of this application.

[0053] The Transformer is a deep learning model architecture based on self-attention. It addresses the efficiency challenges faced by traditional Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks when processing long sequences of data. The core of the Transformer is its self-attention mechanism, which allows it to focus on all positions within a sequence simultaneously. This enables the model to capture dependencies between distant words without progressively passing information. A Transformer typically consists of an encoder and a decoder. The encoder transforms the input sequence into a set of feature vectors, while the decoder transforms these feature vectors into the target sequence. Although this structure was initially used for machine translation, it can be applied to other tasks. Unlike RNNs, all computational steps in the Transformer are performed in parallel, significantly improving training speed and efficiency. The Transformer employs multi-head attention, capturing different contextual information through multiple attention heads, thereby enhancing the model's expressive power.

[0054] The powerful pre-trained model of Bidirectional Encoder Representations from Transformers (BERT) is a bidirectional language model built on the Transformer encoder, achieving state-of-the-art (SOTA) results on multiple natural language processing tasks. During training, BERT considers both the left and right contexts of the input text, enabling a deeper understanding of the language's contextual information. In this embodiment, the text to be verified can be input into the BERT model, which will then output the text features corresponding to multiple texts within that text.

[0055] A fully connected layer is a fundamental layer type in neural networks, referring to a structure where all input and output neurons are interconnected between layers. Its principle is to linearly transform the input vector through weight matrices and bias vectors, and then introduce non-linearity through an activation function, enabling the network to learn and express complex features. Fully connected layers play a role in feature combination and pattern recognition in neural networks, typically used at the end of the network for final classification or regression operations. They fully utilize input information, effectively integrate features from previous layers, and are suitable as the final layer for various tasks to output decision results.

[0056] Text features (or text vectors, text feature vectors) refer to the way text data is converted into numerical form so that machine learning models can understand and process it. The principle is to map words, sentences, or paragraphs in the text into vectors of one or more dimensions using various techniques (such as word embeddings, TF-IDF, One-Hot encoding, etc.). These vectors capture the semantic information and grammatical structure of the text and can be used for further text analysis and processing. The role of text features is to transform unstructured text data into structured data, enabling algorithms to perform operations such as classification, clustering, and similarity calculation. It simplifies complex linguistic information into a computer-processable form while preserving the key meanings and relationships in the text. The dimensionality of a text feature can vary depending on the specific application scenario; for example, it can be 768 dimensions.

[0057] Cross-entropy loss is a commonly used loss function, particularly suitable for classification tasks. It evaluates model performance by measuring the difference between the predicted and true distributions. Specifically, it calculates the distance between the model's predicted probability distribution and the true probability distribution. The cross-entropy loss value is smaller when the model's prediction is close to the true distribution; the loss value increases when the prediction deviates significantly from the true distribution. The goal is to minimize this loss by training the model, thereby improving classification accuracy. Cross-entropy loss is widely used in binary and multi-class classification problems, such as image classification and text classification. It helps guide the model to more accurately predict the probability distribution of labels. Cross-entropy loss effectively handles multi-class problems, and its gradient properties help the model converge more stably and efficiently during optimization.

[0058] In related technologies, when computers identify the similarity between different articles, they often perform similarity matching on the entire content of the article to determine the similarity between different articles. However, users or media may add a large amount of content unrelated to the topic to the article, which causes the computer to spend a lot of computing power on the similarity matching of irrelevant content, resulting in low accuracy of text similarity matching.

[0059] Based on this, embodiments of this application propose a text data processing method, a text data processing apparatus, a text data processing device, a computer-readable storage medium, and a computer program product. In these embodiments, a first text to be verified and a second text to be verified can be obtained. The first text to be verified contains multiple first texts, each corresponding to a first text tag. Similarly, the second text to be verified contains multiple second texts, each corresponding to a first text tag. Text tags can characterize the relevance between the text and a preset topic. Thus, a target first text can be extracted from multiple first texts based on the first text tag, and a target second text can be extracted from multiple second texts based on the second text tag, to determine the similarity between the first text to be verified and the second text to be verified. This method can filter out first and second texts unrelated to the preset topic, using texts related to the preset topic to determine the similarity between the first and second texts to be verified, thereby improving the accuracy of text similarity calculation. Furthermore, it reduces the impact of irrelevant content on similarity calculation, and also improves the efficiency and reliability of similarity calculation.

[0060] Please see Figure 1 , Figure 1 This is a schematic diagram of one implementation environment involved in this application.

[0061] Figure 1The implementation environment shown includes a text matching model, which includes a sentence segmentation module 110, a feature converter 121, a feature converter 122, a filter 130, and a similarity calculation module 140.

[0062] The sentence segmentation module 110 can split the text to be verified into multiple texts corresponding to the text to be verified. Specifically, the sentence segmentation module 110 can split each text to be verified at the sentence level. For example, in... Figure 1 In the process, after the first text to be verified is split by the sentence splitting module 110, it can be obtained as first text 1, first text 2, and first text 3, etc. Similarly, after the second text to be verified is split by the sentence splitting module 110, it can be obtained as second text 1, second text 2, and second text 3, etc. Here, the text to be verified can refer to a complete article, and the text obtained by splitting the text to be verified can be sentences from that text.

[0063] Optionally, the sentence splitting module 110 can add specific text delimiters at the beginning and end of each split text, such as adding the identifier CLS at the beginning of the text and the identifier SEP at the end, so that different texts can be delimited.

[0064] Feature converters 121 and 122 can perform feature transformation on the text segmented by sentence segmentation module 110 to obtain the text features corresponding to each text. For example, first text 1 can be transformed into first text feature 1, first text 2 into first text feature 2, and so on. Feature converters 121 and 122 can be converters with completely identical parameters or converters with different parameters, which can be set by those skilled in the art, and are not limited in this application embodiment. Feature converters 121 and 122 can adopt BERT models (e.g., Chinese-roberta-wwm-ext), pre-trained word vectors plus convolutional neural networks (CNN), long short-term memory networks (LSTM), and other neural networks, so as to more accurately convert each text into corresponding text features, and the dimension corresponding to each text feature can be 768 dimensions.

[0065] Filter 130 can match each text feature with a preset topic to determine whether each text feature is related to the preset topic. If a text feature is related to the preset topic, a label can be assigned to the text feature, indicating that the text feature is related to the preset topic. Conversely, if a text feature is not related to the preset topic, a label can be assigned to the text feature, indicating that the text feature is not related to the preset topic. The preset topic can be set by those skilled in the art according to actual needs.

[0066] Filter 130 can be formed using a fully connected layer, and filter 130 can be a classifier. Figure 1 It is known that neither the first text feature 3 nor the second text feature 2 is related to the preset topic. Therefore, the filter 130 will filter out the first text feature 3 and the second text feature 2. However, since the first text feature 1, the first text feature 2, the second text feature 1, and the second text feature 3 are all related to the preset topic, they can all be retained. These text features can be used as target first text feature 1, target first text feature 2, target second text feature 1, and target second text feature 3, respectively.

[0067] The similarity calculation module 140 is used to determine the similarity between a first text to be verified and a second text to be verified based on the similarity between a first target text and a second target text. Specifically, the similarity calculation module 140 calculates the similarity between the first text to be verified and the second text to be verified based on the first target text features and the second text features retained by the filter 130. If there are multiple first target text features, it is also necessary to calculate the first average text feature corresponding to these multiple first target text features; if there are multiple second target text features, it is also necessary to calculate the second average text feature corresponding to these multiple second target text features. Then, the first average text feature and the second average text feature, as well as the text feature difference between the first average text feature and the second average text feature, are concatenated to obtain the concatenated text feature. In this way, the similarity between the first text to be verified and the second text to be verified can be determined based on the concatenated text feature. The similarity calculation module 140 includes a fully connected layer, into which the concatenated text feature can be input to obtain the similarity between the first text to be verified and the second text to be verified.

[0068] Through the collaborative work of the above modules, the text matching model described in this application can efficiently and accurately calculate the similarity of the text to be verified. The sentence segmentation module 110 is responsible for splitting the complete text to be verified into multiple sentence-level texts. Feature converters 121 and 122 convert these texts into corresponding text features. Filter 130 filters these text features based on preset topic relevance. Finally, the similarity calculation module 140 calculates the similarity between the first and second texts to be verified. This architecture ensures the efficiency and accuracy of the calculation process, especially when dealing with texts containing a large amount of irrelevant content, significantly improving the accuracy of text similarity matching.

[0069] Optionally, the text matching model of this application can be deployed on a terminal device or a server.

[0070] It should be noted that the terminal device in the embodiments of this application can be a mobile phone, tablet computer, laptop computer, handheld computer, mobile internet device (MID), vehicle device, aircraft, wearable device (such as smart devices such as smartwatches, smart bracelets, pedometers, etc.), virtual reality device (such as virtual reality (VR) device, augmented reality (AR) device), etc.

[0071] Optionally, the computer in this application embodiment can also be deployed in a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0072] Please see Figure 2 , Figure 2 This is a flowchart illustrating a text data processing method according to an exemplary embodiment. The method can be applied to... Figure 1 The implementation environment is shown, and the method is executed by a computer. Of course, this method can also be applied to other implementation environments, and there are no restrictions on the subject executing this method.

[0073] The following section will use a computer as an example to illustrate this text data processing method in detail. Figure 2 As shown, in an exemplary embodiment, the method includes at least the following steps:

[0074] S210. Obtain a first text to be verified that contains multiple first texts. Each first text has a corresponding first text tag. The first text tag is used to characterize the relevance between the first text and the preset topic.

[0075] The computer can obtain the first text to be verified, which can be an article. The article can be a news report, a reprinted news report, a personal article, or a reprinted article from another person. Furthermore, the article in this application can be obtained from various articles publicly published on the Internet.

[0076] Optionally, the computer can analyze the attribute information of various articles publicly published on the Internet to perform preliminary screening and select articles related to the preset topic. This attribute information may include at least one of title information and source information, and the first text to be verified is the article related to the preset topic. The source information may include the article's link address (e.g., a URL). The computer can match article titles using keywords or related terms from the preset topic. Articles with titles containing many words related to the preset topic are given priority consideration as relevant articles. The computer can also analyze the article's source information; certain domains or subpaths may correspond to specific content categories (e.g., news, technical blogs). If the domain or path of a source is related to a relevant area of ​​the preset topic, the computer can consider the article more likely to be related to the preset topic and can consider it a relevant article.

[0077] It should be noted that, in this application, articles related to the preset theme refer to articles that are directly or indirectly related to a specific topic. These articles may have a certain degree of relevance in terms of theme to preset keywords, locations, concepts, events, people, etc. For example, if the preset theme is "road," then articles related to the preset theme may include articles about road names, events that occurred on that road, and their impact, while content unrelated to roads (such as food or entertainment news) is not considered relevant articles.

[0078] The relevance of articles can be evaluated in various ways, such as analyzing title information, content keywords, and source information, or using higher-level semantic analysis based on natural language processing techniques to determine the degree of match between the articles and the preset topic. In the initial screening stage, the computer can use this attribute information to filter out articles that are more likely to be related to the preset topic for subsequent detailed analysis and processing. It should be noted that the determination of relevant articles, being an initial screening, does not guarantee that all relevant articles will be associated with the preset topic, but it can reduce the number of irrelevant articles, thereby reducing the pressure and computational burden on the computer in subsequent processing.

[0079] The computer can use a text matching model to process the first text to be verified.

[0080] Specifically, the computer can first use the sentence segmentation module in the text matching model to split the first text to be verified into multiple first texts. When the first text to be verified is an article, the multiple first texts can be multiple sentences in the first text to be verified.

[0081] After splitting the first text to be verified into multiple first texts, the computer can add text delimiters to each first text. These text delimiters can include two symbols: [CLS] and [SEP]. In text classification tasks, [CLS] typically represents the beginning of a sentence, while [SEP] typically represents the end of a sentence. Therefore, the function of text delimiters (such as the two symbols [CLS] and [SEP]) is to separate different sentences.

[0082] Next, the computer can convert each of the multiple first texts into multiple first text features. A feature converter can be used to transform these multiple first texts, and this feature converter can be implemented using the BERT model. These multiple first text features can be denoted as U = {U1, U2, ..., U...}. n}, where n is the number of the multiple first texts.

[0083] In this way, the computer can calculate the first text label corresponding to each of the multiple first text features. Specifically, the computer can calculate the similarity between each first text feature and a preset topic, obtain the first similarity corresponding to each first text, and generate the first text label corresponding to each first text based on the first similarity of each first text. Specifically, if a first text with a first similarity greater than a first preset similarity threshold is detected, a first text label representing that the detected first text is related to the preset topic is generated; if a first text with a first similarity less than or equal to the first preset similarity threshold is detected, a first text label representing that the detected first text is not related to the preset topic is generated.

[0084] The first preset similarity threshold and the second preset similarity threshold can be set by those skilled in the art, and they can be the same or different. This application embodiment does not limit this.

[0085] In this way, each first text corresponds to a first text tag, which is used to characterize the relevance between the first text and the preset topic.

[0086] By calculating the similarity between each first text feature and a preset topic, and generating a clear first text label based on the similarity, the system can accurately identify first texts related to the preset topic. This labeling process clearly identifies the relevance between each first text and the preset topic, facilitating accurate filtering of multiple first texts and effectively reducing interference from first texts unrelated to the preset topic.

[0087] S220. Obtain a second text to be verified that contains multiple second texts. Each second text has a corresponding second text label. The second text label is used to characterize the relevance between the second text and the preset topic.

[0088] The second text to be verified is also an article related to the preset topic, and the computer can use a text matching model to process the second text to be verified.

[0089] Specifically, the computer can first use the sentence segmentation module in the text matching model to split the second text to be verified into multiple second texts, and add text delimiters (such as [CLS] and [SEP] symbols) to the beginning and end of each second text. Further, the multiple second texts are converted into multiple second text features, and the second text label corresponding to each of these features is calculated.

[0090] When the computer converts the multiple second texts into multiple second text features, a feature converter can be used to perform the conversion. This feature converter can be implemented using the BERT model. The multiple second text features can be denoted as C = {C1, C2, ..., C}. m}, where m is the number of the multiple second texts.

[0091] It should be noted that the computer can process the first and second texts to be verified in the same way, which will not be elaborated here. Furthermore, the feature converter used by the computer to convert multiple first texts into multiple first text features and the feature converter used to convert multiple second texts into multiple second text features can be the same or different, depending on the specific settings of those skilled in the art, and are not limited here.

[0092] The computer can calculate the similarity between each second text feature and a preset topic, obtain a second similarity for each second text, and generate a second text label for each second text based on the second similarity. Specifically, if a second text with a second similarity greater than a second preset similarity threshold is detected, a second text label representing a relevance to the preset topic is generated; if a second text with a second similarity less than or equal to the second preset similarity threshold is detected, a second text label representing a non-relevance to the preset topic is generated.

[0093] By calculating the similarity between each second text feature and a preset topic, and generating clear second text labels based on the similarity, it is possible to accurately identify second texts related to the preset topic. This same labeling process ensures that the relevance between each second text and the preset topic is clearly identified, facilitating accurate filtering of multiple second texts and effectively reducing interference from second texts unrelated to the preset topic.

[0094] S230, extract the target first text from multiple first texts based on the first text label, and extract the target second text from multiple second texts based on the second text label.

[0095] Since each first text tag can characterize whether its corresponding first text (or first text feature) is related to a preset topic, a target first text can be extracted from multiple first texts based on the first text tags. This target first text is the first text related to the preset topic. This process can be implemented by a computer using filters in a text matching model.

[0096] In other words, the computer can filter out first texts that are not related to the preset topic and retain the first texts that are related to the preset topic as the target first texts based on the first text tags of each first text.

[0097] Similarly, the computer can also filter out second texts that are not related to the preset topic and retain the second texts that are related to the preset topic as the target second texts based on the second text tags of each second text.

[0098] By filtering irrelevant first or second text based on first and second text tags, target text related to the preset topic can be effectively selected. This ensures that in subsequent text processing and analysis, the computer only needs to process text related to the preset topic, thereby reducing interference from irrelevant text, reducing computational complexity, and improving processing efficiency.

[0099] S240. Based on the similarity between the first target text and the second target text, determine the similarity between the first text to be verified and the second text to be verified.

[0100] Specifically, the computer can obtain the target first text features corresponding to the target first text, and obtain the target second text features corresponding to the target second text. Then, the target first text features and the target second text features are concatenated to obtain concatenated text features. Finally, the similarity between the first text to be verified and the second text to be verified can be determined based on the concatenated text features.

[0101] If there are multiple target first text features, then the average of these multiple target first text features needs to be calculated to obtain a first average text feature. Similarly, if there are multiple target second text features, then the average of these multiple target second text features needs to be calculated to obtain a second average text feature. Next, the computer can concatenate the first average text feature and the second average text feature to obtain a concatenated text feature, and then determine the similarity between the first and second texts to be verified based on the concatenated text feature.

[0102] In one embodiment of this application, the spliced ​​text features include, in addition to the target first text feature (or the first average text feature) and the target second text feature (or the second average text feature), also the text feature difference between the target first text feature and the target second text feature (or the text feature difference between the first average text feature and the second average text feature).

[0103] Computers can use a similarity calculation module to determine the similarity between a first text to be verified and a second text to be verified based on the features of concatenated text.

[0104] This method allows a computer to obtain a first text to be verified and a second text to be verified. The first text contains multiple first texts, each with a corresponding first text tag. Similarly, the second text contains multiple second texts, each with a corresponding first text tag. Text tags characterize the relevance of a text to a preset topic. Thus, a target first text can be extracted from multiple first texts based on the first text tags, and a target second text can be extracted from multiple second texts based on the second text tags, to determine the similarity between the first and second texts to be verified. This method filters out first and second texts unrelated to the preset topic, using texts relevant to the preset topic to determine the similarity, thus improving the accuracy of text similarity calculation. Furthermore, it reduces the impact of irrelevant content on similarity calculation, improving the efficiency and reliability of text similarity calculation.

[0105] In one embodiment of this application, another text data processing method is provided, which can be executed by a computer. For example... Figure 3 As shown, the text data processing method may include steps S310 to S320. That is, steps S310 to S320 are... Figure 2 In addition to the steps shown, there are other steps that need to be performed.

[0106] The following describes S310 to S320:

[0107] S310. Calculate the similarity between each first text and the preset topic, obtain the first similarity corresponding to each first text, and generate the first text label corresponding to each first text based on the first similarity of each first text.

[0108] Specifically, the computer needs to obtain the preset topic features corresponding to the preset topic. These preset topic features are obtained by training a text matching model. The computer can then calculate the similarity between the first text features corresponding to each first text and the preset topic features. The similarity calculation method can employ distance metrics such as cosine similarity or Euclidean distance. This yields a similarity score between each first text and the preset topic, reflecting the distance between the first text and the preset topic. This similarity score can be a rating, distributed within a specific range, such as between -1000 and 1000.

[0109] Next, to better understand the similarity between each first text and the preset topic, the similarity score of each first text can be mapped to a range of 0-1. This can be done using a specific mapping function, such as the softmax activation function. In this way, we can obtain the first similarity score corresponding to each first text.

[0110] For example, the similarity score between the first text 1 and the preset topic is 500. After being mapped to the range of 0-1 by the softmax activation function, the first similarity 1 corresponding to the first text 1 is 0.75.

[0111] Next, the computer compares the first similarity score corresponding to each first text with a first preset similarity threshold. If a first text with a first similarity score greater than the first preset similarity threshold is detected, a first text label representing that the detected first text is related to the preset topic is generated. If a first text with a first similarity score less than or equal to the first preset similarity threshold is detected, a first text label representing that the detected first text is not related to the preset topic is generated. The first preset similarity threshold can be set by someone skilled in the art, for example, it can be 0.5.

[0112] In this way, the computer compares the first similarity corresponding to each first text with the first preset similarity threshold, and can accurately determine whether the first text is related to the preset topic. This allows the generation of first text labels that represent whether they are related, thus improving the efficiency of determining the preset topic relevance of the first text.

[0113] In one embodiment of this application, a first text tag used to characterize a relevance to a preset topic can be represented by a specific numerical value, such as using 1 to indicate that its corresponding first text is related to the preset topic; similarly, another numerical value can be used to characterize a first text tag not related to the preset topic, such as using 0 to indicate that its corresponding first text is not related to the preset topic.

[0114] This method utilizes similarity scores and performs a unified similarity calculation on multiple first texts through a mapping function, enabling different first texts to use the same method to calculate their similarity, thereby improving the accuracy of calculating the similarity between different first texts and a preset topic.

[0115] S320. Calculate the similarity between each second text and the preset topic, obtain the second similarity corresponding to each second text, and generate the second text label corresponding to each second text based on the second similarity of each second text.

[0116] The process by which the computer calculates the similarity between each second text and the preset topic, and generates a second text label for each second text based on the second similarity, is the same as the process for the first text, and will not be elaborated here.

[0117] Specifically, if a second text with a second similarity greater than a second preset similarity threshold is detected, a second text tag is generated to represent that the detected second text is related to a preset topic. If a second text with a second similarity less than or equal to the second preset similarity threshold is detected, a second text tag is generated to represent that the detected second text is not related to the preset topic. The second preset similarity threshold can be set by someone skilled in the art, and its value can be the same as or different from the first preset similarity threshold; for example, it can be 0.5 or 0.55.

[0118] In this way, the computer compares the second similarity corresponding to each second text with the second preset similarity threshold, and can accurately determine whether the second text is related to the preset topic. This allows the generation of second text labels that represent whether they are related, thus improving the efficiency of determining the preset topic relevance of the second text.

[0119] In one embodiment of this application, a second text tag used to characterize a relevance to a preset topic can be represented by a specific numerical value, such as using 1 to indicate that its corresponding second text is related to the preset topic; similarly, two other numerical values ​​can be used to characterize a second text tag used to characterize a lack of relevance to the preset topic, such as using 0 to indicate that its corresponding second text is not related to the preset topic.

[0120] Similarly, this method also utilizes similarity scores and performs a unified similarity calculation on multiple second texts through a mapping function, so that different second texts can use the same method to calculate their similarity, which can improve the accuracy of calculating the similarity between different second texts and the preset topic.

[0121] It should be noted that S310 and S320 can be executed simultaneously, without distinguishing the order of execution.

[0122] Using this method, the computer can calculate the first text tag corresponding to each first text and the second text tag corresponding to each second text, so that the computer can know whether each text is related to the preset topic. This is beneficial for subsequent filtering of texts related to the preset topic, reducing the processing of texts that are not related to the preset topic, and improving the efficiency and accuracy of text similarity calculation.

[0123] In one embodiment of this application, another text data processing method is provided, which can be executed by a computer. For example... Figure 4 As shown, the text data processing method may include S210 to S220, S410 to S420, and S240. That is, S410 to S420 are... Figure 2 The specific implementation method of S230 is shown.

[0124] The following describes S410 to S420:

[0125] S410. Extract the first text corresponding to the first text label used to represent the relevance to the preset topic from multiple first texts, and use the extracted first text as the target first text.

[0126] Since each first text already has a corresponding first text tag, the computer can extract the first text that is related to the preset topic based on the first text tag of each first text, and use it as the target first text.

[0127] In one embodiment of this application, the first text tag can be represented by a specific numerical value, so that whether the first text is the target first text can be determined directly based on the numerical value. For example, if the value of the first text tag corresponding to a certain first text is 1, it means that it is related to the preset topic, so the first text can be used as the target first text. If the value of the first text tag corresponding to a certain first text is 0, it means that it is not related to the preset topic, so the first text will be filtered out.

[0128] By pre-calculating the first text tag corresponding to each first text, the computer can directly identify whether the first text corresponding to each first text tag is related to the preset topic. This allows for the rapid filtering out of first texts whose tag representations are unrelated to the preset topic, thus improving the efficiency and accuracy of text filtering.

[0129] S420. Extract the second text corresponding to the second text label used to represent the relevance to the preset topic from multiple second texts, and use the extracted second text as the target second text.

[0130] Similarly, the computer will perform the same operation on multiple second texts to extract the second text corresponding to the second text label used to represent the relevance to the preset topic, and use the extracted second text as the target second text, which will not be elaborated here.

[0131] Similarly, by pre-calculating the second text tag corresponding to each second text, the computer can directly identify whether the corresponding second text is related to the preset topic based on each second text tag. This allows for the rapid filtering out of second texts whose tag representations are unrelated to the preset topic, thus improving the efficiency and accuracy of text filtering.

[0132] It should be noted that S410 and S420 can be executed simultaneously, without distinguishing the order of execution.

[0133] This method extracts the target first text based on the first text label of each first text and the target second text based on the second text label of each second text, thereby improving the accuracy of target text extraction.

[0134] In one embodiment of this application, another text data processing method is provided, which can be executed by a computer. For example... Figure 5 As shown, the text data processing method may include steps S210 to S230 and S510 to S530. That is, steps S510 to S530 are... Figure 2 The specific implementation method of S240 is shown.

[0135] The following describes S510 to S530:

[0136] S510. Obtain the target first text features corresponding to the target first text, and obtain the target second text features corresponding to the target second text.

[0137] The computer will convert each first text into a corresponding first text feature. Therefore, it is possible to obtain the target first text feature corresponding to the target first text and the target second text feature corresponding to the target second text.

[0138] Converting text into features can help computers understand the text content corresponding to the first target text and the second target text, thereby improving the accuracy of text processing.

[0139] S520. Concatenate the first text feature of the target and the second text feature of the target to obtain the concatenated text feature.

[0140] To better compare the relationship between the first and second target texts, the computer concatenates their feature vectors. Concatenation involves combining the features of the first and second target texts sequentially to form a new concatenated text feature. For example... Figure 6a As shown, assuming the first text feature of the target is U1 and the second text feature of the target is C1, then the concatenated text feature is the text feature of U1 and C1 concatenated together.

[0141] In one embodiment of this application, if there are multiple target first text features, the average of the multiple target first text features is calculated to obtain a first average text feature. The averaging method can be to add the values ​​of each target first text feature together and then divide by the number of the multiple target first text features to obtain the first average text feature.

[0142] In one embodiment of this application, if there are multiple target second text features, the average of the multiple target second text features is calculated to obtain a second average text feature. This processing step is the same as the processing step for multiple target first text features, and will not be described in detail here.

[0143] For example, such as Figure 6b As shown, assuming the target's first text features include U1, U2, U3, and U4, then the first average text feature can be denoted as U' = (U1 + U2 + U3 + U4) / 4. Assuming the target's second text features include C1, C2, and C3, then the first average text feature can be denoted as C' = (C1 + C2 + C3) / 3.

[0144] By concatenating the first and second target text features, the feature information of the two texts is integrated to form a more comprehensive concatenated text feature. This preserves the individual text features while integrating them into a unified feature vector, laying the foundation for subsequent similarity calculations. Furthermore, by averaging multiple target text features, it is unnecessary to perform similarity calculations for each target text feature separately, reducing the computational load and improving the efficiency of similarity calculations.

[0145] S530. Determine the similarity between the first text to be verified and the second text to be verified based on the features of the concatenated text.

[0146] Computers can use a similarity calculation module to calculate the similarity of the feature vectors of concatenated text. This similarity represents the similarity between the first and second texts to be verified under a preset topic.

[0147] The higher the similarity score, the more similar the content of the first and second texts to be verified is under the preset topic, thus determining that they contain content of the same topic.

[0148] This method combines the features of the target text with those of the two texts to be verified, allowing subsequent similarity calculations to fully utilize both pieces of information, thus improving the comprehensiveness of the similarity comparison and making the similarity calculation results more accurate.

[0149] In one embodiment of this application, another text data processing method is provided, which can be executed by a computer. For example... Figure 7 As shown, the text data processing method may include steps S210 to S230 and S710 to S720. That is, steps S710 to S720 are... Figure 2 The specific implementation method of S240 is shown.

[0150] The following describes S710 to S720:

[0151] S710. Calculate the text feature difference between the first text feature of the target and the second text feature of the target.

[0152] In other words, the computer can obtain the text feature difference by subtracting the first text feature of the target from the second text feature of the target.

[0153] For example, such as Figure 8a As shown, the first text feature of the target is U1, and the second text feature of the target is C1. The difference between these text features can be denoted as U1-C1.

[0154] The introduction of text feature differences can reveal subtle differences between two texts, allowing the model to refer to more information sources when calculating similarity, which helps to improve the comprehensiveness and accuracy of similarity calculation.

[0155] S720. The target first text feature, the target second text feature, and the text feature difference are concatenated to obtain the concatenated text feature.

[0156] Computers can concatenate the first text feature, the second text feature, and the text feature difference to obtain the concatenated text feature. For example... Figure 8a As shown, the computer concatenates U1, C1, and U1-C1 to obtain the concatenated text features.

[0157] In one embodiment of this application, if any one of the target first text feature and the target second text feature is satisfied, the computer can correspondingly use at least one of the first average text feature and the second average text feature to generate concatenated text features. For example, such as Figure 8bAs shown, assuming the first text feature of the target contains U1, U2, U3, and U4, then the first average text feature can be denoted as U' = (U1 + U2 + U3 + U4) / 4. Assuming the second text feature of the target contains C1, C2, and C3, then the first average text feature can be denoted as C' = (C1 + C2 + C3) / 3. At this point, U', C', and U'-C' can be concatenated to obtain the concatenated text feature.

[0158] Furthermore, the computer can then calculate the similarity between the first and second texts to be verified based on the features of the concatenated text.

[0159] This method, by calculating the text feature difference between the first and second target text features, can reveal subtle differences between the two texts, allowing the model to refer to more information sources when calculating similarity. Therefore, concatenating the first target text feature, the second target text feature, and the text feature difference improves the accuracy of similarity calculation between the first and second texts to be verified.

[0160] In one embodiment of this application, a model training method related to the text data processing method of this application is provided, which can be executed by a computer. Figure 9 As shown, the model training method may include steps S910 to S940. The computer can perform steps S210 to S240 using a text matching model.

[0161] The following describes S910 to S940:

[0162] S910. Obtain a first training sample containing multiple first samples. Each first sample has a corresponding first sample label. The first sample label is used to characterize the correlation between the first sample and the sample topic.

[0163] The first training sample can be an article containing multiple sentences, each of which can be considered a first sample. The first sample label for each first sample is known and can characterize whether the label is relevant to the sample topic. Specific numerical values ​​can be used to determine the relevance of a first sample to the sample topic. For example, if the first sample label is 1, it means the first sample is relevant to the sample topic; if the first sample label is 0, it means the first sample is not relevant to the sample topic.

[0164] The sample topic can be set by someone skilled in the art. If the text matching model is used to perform text processing related to a preset topic, then the sample topic needs to be set to be the same as the preset topic.

[0165] By assigning a first-sample label to each first sample, the relevance of each first sample to the sample topic can be effectively labeled. In this way, the model to be trained can more easily learn the relevance between the sample topic and each first sample during the training process, thereby generating sample topic features of the sample topic more accurately.

[0166] S920. Obtain a second training sample containing multiple second samples. Each second sample has a corresponding second sample label. The second sample label is used to characterize the correlation between the second sample and the sample topic.

[0167] Similarly, the second training sample can also be an article containing multiple second samples, each with a corresponding label. A 1 indicates that a second sample is relevant to the sample topic, while a 0 indicates that it is irrelevant. The first and second training samples can be a training sample pair, meaning they are a single pair. The first and second training samples also have matching labels that indicate whether they are similar under the sample topic. If similar, a matching label of 1 is used; otherwise, a matching label of 0 is used.

[0168] For example, such as Figure 10 The diagram shows a schematic representation of the training data required for training a text matching model according to this application. Figure 10 The dataset contains a first training sample and a second training sample. The first training sample contains three first samples, and the second training sample contains four second samples. Each first sample and each second sample is separated by text delimiters, namely [CLS] and [SEP], respectively. Each first sample has a corresponding first sample label, and each second sample has a corresponding second sample label. For example, the first sample 1 is labeled as "related to the sample topic," the first sample 3 is labeled as "unrelated to the sample topic," and so on. Furthermore, the matching labels between the first and second training samples are characterized as similarity.

[0169] If the first training sample has m first samples, then there are m first sample labels. If the second training sample has n second samples, then there are n second sample labels. In addition, there are matching labels between the first and second training samples. Therefore, in the process of training the model to be trained based on the first and second training samples, m+n+1 labels are needed.

[0170] Those skilled in the art can provide different training samples according to actual needs, and the embodiments in this application are not limited thereto.

[0171] Similarly, by assigning a label to each second sample, the relevance of each second sample to the sample topic can be effectively labeled. This allows the model to learn the relevance between the sample topic and each second sample more easily during training, thus enabling it to generate sample topic features more accurately.

[0172] S930. Based on the first sample label and the second sample label, calculate the predicted similarity between the first training sample and the second training sample.

[0173] The computer predicts the similarity between two training samples based on the sample labels contained in the first and second training samples. Specifically, the computer uses information from each first sample label and each second sample label to analyze and calculate the predicted similarity between the first and second training samples under the same sample topic. This predicted similarity characterizes the degree of similarity between the two training samples in terms of the sample topic.

[0174] Optionally, the predicted similarity can be binarized, which can characterize whether the first training sample and the second training sample are similar under the sample topic. If they are similar, the predicted similarity is 1; if they are not similar, the predicted similarity is 0.

[0175] By analyzing the label similarity between each training sample, the model can learn the association between different samples under the same topic, thereby improving the accuracy of predicted similarity. Furthermore, calculating predicted similarity allows the model to better understand the similarity relationships between training samples, enabling more accurate optimization of the model.

[0176] S940. Based on the similarity between the first and second training samples and the loss between the predicted similarities, adjust the model parameters of the model to be trained to obtain a text matching model.

[0177] The computer optimizes the parameters of the model being trained by comparing the difference (i.e., the loss) between the true similarity between the first and second training samples and the similarity predicted by the model. This loss can be calculated using a loss function (such as the cross-entropy loss function). By minimizing this loss, the parameters of the model are gradually adjusted, enabling it to more accurately predict the similarity between training samples during subsequent training, thereby improving the model's overall matching ability. Ultimately, after multiple iterations, the model is optimized to a certain degree, becoming a text matching model capable of accurately performing text similarity calculations.

[0178] The similarity between the first training sample and the second training sample refers to the similarity between the matching labels of the first training sample and the second training sample.

[0179] The text matching model can include sample features corresponding to the sample topic. If the sample topic is a preset topic, then the text matching model can include preset topic features.

[0180] In one embodiment of this application, in addition to adjusting the model parameters of the model to be trained based on the similarity between the input first training sample and the second training sample, and the loss between the predicted similarities, the model parameters of the model to be trained can also be adjusted based on the loss of each sample (including the first sample and the second sample) to form an overall loss. Specifically, the overall loss can be calculated using the following formula:

[0181]

[0182] Among them, L sentence y represents the loss value corresponding to each sample, where L represents the loss between the similarity of the matched labels and the predicted similarity. s p represents the relevance between each sample and the sample topic (1 if relevant, 0 if irrelevant). s Represents y s The predicted probability. Similarly, y i p represents the predicted similarity. i Represents y i The predicted probability.

[0183] By comparing the loss between the actual similarity and the predicted similarity, the model to be trained will iterate multiple times to gradually optimize the model parameters, so that the model to be trained can gradually reduce the model error and improve the accuracy and stability of the model operation.

[0184] This method allows a computer to continuously modify the internal model parameters by training the model to obtain a text matching model. Using this trained text matching model for text similarity calculation can improve the efficiency and accuracy of text similarity calculation.

[0185] Figure 11 This is a block diagram illustrating a text data processing apparatus according to one embodiment of this application. Figure 11 As shown, this text data processing device can be applied to a computer, and the device includes:

[0186] A text data processing device, comprising:

[0187] The acquisition unit 1110 is used to acquire a first text to be verified containing multiple first texts, each first text having a corresponding first text tag, and the first text tag being used to characterize the relevance between the first text and the preset topic;

[0188] The acquisition unit 1110 is also used to acquire a second text to be verified containing multiple second texts, each second text having a corresponding second text tag, the second text tag being used to characterize the relevance between the second text and the preset topic;

[0189] Extraction unit 1120 is used to extract target first text from multiple first texts based on first text tags, and to extract target second text from multiple second texts based on second text tags;

[0190] The processing unit 1130 is used to determine the similarity between the first text to be verified and the second text to be verified based on the similarity between the first target text and the second target text.

[0191] In one embodiment of this application, based on the foregoing scheme, the device further includes a calculation unit 1140, configured to calculate the similarity between each first text and a preset topic, obtain a first similarity corresponding to each first text, and generate a first text tag corresponding to each first text based on the first similarity of each first text; and to calculate the similarity between each second text and a preset topic, obtain a second similarity corresponding to each second text, and generate a second text tag corresponding to each second text based on the second similarity of each second text.

[0192] In one embodiment of this application, based on the foregoing scheme, the processing unit 1130 is further configured to: if a first text with a first similarity greater than a first preset similarity threshold is detected, generate a first text tag representing that the detected first text is related to a preset topic; if a first text with a first similarity less than or equal to the first preset similarity threshold is detected, generate a first text tag representing that the detected first text is not related to the preset topic; if a second text with a second similarity greater than a second preset similarity threshold is detected, generate a second text tag representing that the detected second text is related to the preset topic; if a second text with a second similarity less than or equal to the second preset similarity threshold is detected, generate a second text tag representing that the detected second text is not related to the preset topic.

[0193] In one embodiment of this application, based on the foregoing scheme, the extraction unit 1120 is further configured to extract the first text corresponding to the first text tag used to characterize the relevance to the preset topic from a plurality of first texts, and use the extracted first text as the target first text; and to extract the second text corresponding to the second text tag used to characterize the relevance to the preset topic from a plurality of second texts, and use the extracted second text as the target second text.

[0194] In one embodiment of this application, based on the aforementioned scheme, the acquisition unit 1110 is further configured to acquire the target first text feature corresponding to the target first text and the target second text feature corresponding to the target second text; the processing unit 1130 is further configured to concatenate the target first text feature and the target second text feature to obtain the concatenated text feature; and determine the similarity between the first text to be verified and the second text to be verified based on the concatenated text feature.

[0195] In one embodiment of this application, based on the aforementioned scheme, the calculation unit 1140 is further configured to, if there are multiple target first text features, calculate the average of the multiple target first text features to obtain a first average text feature; if there are multiple target second text features, calculate the average of the multiple target second text features to obtain a second average text feature; the processing unit 1130 is further configured to concatenate the first average text feature and the second average text feature to obtain a concatenated text feature.

[0196] In one embodiment of this application, based on the aforementioned scheme, the calculation unit 1140 is further configured to calculate the text feature difference between the target first text feature and the target second text feature; the processing unit 1130 is further configured to concatenate the target first text feature, the target second text feature and the text feature difference to obtain concatenated text features.

[0197] In one embodiment of this application, based on the aforementioned scheme, the similarity between the first text to be verified and the second text to be verified is determined by a text matching model; the filtering model is trained in the following manner: the acquisition unit 1110 is further configured to acquire a first training sample containing multiple first samples, each first sample corresponding to a first sample label, the first sample label being used to characterize the relevance between the first sample and the sample topic; acquire a second training sample containing multiple second samples, each second sample corresponding to a second sample label, the second sample label being used to characterize the relevance between the second sample and the sample topic; the calculation unit 1140 is further configured to calculate the predicted similarity between the first training sample and the second training sample based on the first sample label and the second sample label; the processing unit 1130 is further configured to adjust the model parameters of the model to be trained based on the similarity between the input first training sample and the second training sample, and the loss between the predicted similarity, to obtain a text matching model.

[0198] In one embodiment of this application, based on the foregoing scheme, before obtaining the first text to be verified containing multiple first texts, the obtaining unit 1110 is further configured to obtain multiple texts to be verified, and attribute information of each text to be verified, the attribute information including at least one of title information and source information; the processing unit 1130 is further configured to obtain the first text to be verified and the second text to be verified associated with a preset topic from the multiple texts to be verified based on the attribute information of each text to be verified.

[0199] It should be noted that the apparatus provided in the foregoing embodiments and the method provided in the foregoing embodiments belong to the same concept, and the specific way in which each module and unit performs operations has been described in detail in the method embodiments.

[0200] Embodiments of this application also provide a text data processing device, including: one or more processors; and a memory for storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the aforementioned text data processing method.

[0201] Figure 12 This is a schematic diagram of the structure of a computer system suitable for implementing the text data processing device of the embodiments of this application.

[0202] It should be noted that, Figure 12 The computer system 1200 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0203] like Figure 12As shown, the computer system 1200 includes a Central Processing Unit (CPU) 1201, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1202 or programs loaded from storage portion 1208 into Random Access Memory (RAM) 1203. The RAM 1203 also stores various programs and data required for system operation. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An Input / Output (I / O) interface 1205 is also connected to the bus 1204.

[0204] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0205] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs various functions defined in the system of this application.

[0206] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. For example, a computer-readable medium can be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0207] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0208] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0209] Another aspect of this application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the text data processing method described above. This computer-readable medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0210] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable medium. A processor of a computer device reads the computer instructions from the computer-readable medium and executes the computer instructions, causing the computer device to perform the text data processing methods provided in the various embodiments described above.

[0211] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. A method of processing text data, characterized by, The method comprises: obtaining first to-be-verified texts comprising a plurality of first texts, each first text corresponding to a first text label used to represent the relevance between the first text and a preset theme; obtaining second to-be-verified texts comprising a plurality of second texts, each second text corresponding to a second text label used to represent the relevance between the second text and the preset theme; extracting a target first text from the plurality of first texts based on the first text label, and extracting a target second text from the plurality of second texts based on the second text label; determining the similarity between the first to-be-verified texts and the second to-be-verified texts based on the similarity between the target first text and the target second text.

2. The method of claim 1, wherein, The method further comprises: calculating the similarity between each first text and the preset theme to obtain a first similarity corresponding to each first text, and generating a first text label corresponding to each first text based on the first similarity of each first text; and calculating the similarity between each second text and the preset theme to obtain a second similarity corresponding to each second text, and generating a second text label corresponding to each second text based on the second similarity of each second text.

3. The method of claim 2, wherein, The method of generating a first text label corresponding to each first text based on the first similarity of each first text comprises: if a first text with a first similarity greater than a first preset similarity threshold is detected, generating a first text label of the detected first text to represent the existence of relevance to the preset theme; if a first text with a first similarity less than or equal to the first preset similarity threshold is detected, generating a first text label of the detected first text to represent the non-existence of relevance to the preset theme; The method of generating a second text label corresponding to each second text based on the second similarity of each second text comprises: if a second text with a second similarity greater than a second preset similarity threshold is detected, generating a second text label of the detected second text to represent the existence of relevance to the preset theme; if a second text with a second similarity less than or equal to the second preset similarity threshold is detected, generating a second text label of the detected second text to represent the non-existence of relevance to the preset theme.

4. The method of claim 1, wherein, The method of extracting a target first text from the plurality of first texts based on the first text label, and extracting a target second text from the plurality of second texts based on the second text label comprises: extracting a first text corresponding to a first text label representing the existence of relevance to the preset theme from the plurality of first texts, and taking the extracted first text as the target first text; and extracting a second text corresponding to a second text label representing the existence of relevance to the preset theme from the plurality of second texts, and taking the extracted second text as the target second text.

5. The method of claim 1, wherein, The similarity between the first to-be-verified text and the second to-be-verified text is determined based on the similarity between the target first text and the target second text, including: obtaining target first text features corresponding to the target first text, and obtaining target second text features corresponding to the target second text; splicing the target first text features and the target second text features to obtain spliced text features; determine the similarity between the first to-be-verified text and the second to-be-verified text based on the spliced text features.

6. The method of claim 5, wherein, The splicing of the target first text features and the target second text features to obtain spliced text features includes: if the target first text features are multiple, the multiple target first text features are averaged to obtain a first average text feature; if the target second text features are multiple, the multiple target second text features are averaged to obtain a second average text feature; splicing the first average text feature and the second average text feature to obtain the spliced text feature.

7. The method of claim 5, wherein, The splicing of the target first text features and the target second text features to obtain spliced text features includes: calculate the text feature difference between the target first text feature and the target second text feature; splicing the target first text feature, the target second text feature and the text feature difference to obtain the spliced text feature.

8. The method of claim 1, wherein, The similarity between the first to-be-verified text and the second to-be-verified text is determined by a text matching model; the filtering model is obtained by training as follows: obtain a first training sample containing a plurality of first samples, each first sample corresponding to a first sample label, the first sample label being used to represent the correlation between the first sample and the sample theme; obtain a second training sample containing a plurality of second samples, each second sample corresponding to a second sample label, the second sample label being used to represent the correlation between the second sample and the sample theme; based on the first sample label and the second sample label, calculate the predicted similarity between the first training sample and the second training sample; based on the loss between the input similarity between the first training sample and the second training sample and the predicted similarity, adjust the model parameters of the to-be-trained model to obtain the text matching model.

9. The method of claim 1, wherein, Before obtaining the first to-be-verified text containing a plurality of first texts, the method further includes: obtain a plurality of to-be-verified texts, and attribute information of each to-be-verified text, the attribute information including at least one of title information and source information; based on the attribute information of each to-be-verified text, obtain the first to-be-verified text and the second to-be-verified text associated with the preset theme from the plurality of to-be-verified texts.

10. A text data processing apparatus characterized by comprising: includes: an obtaining unit is configured to obtain a first to-be-verified text containing a plurality of first texts, each first text corresponding to a first text label, the first text label being used to represent the correlation between the first text and the preset theme; The acquisition unit is further configured to acquire second to-be-verified texts containing a plurality of second texts, each second text corresponding to a second text label, the second text label being used to represent the relevance between the second text and the preset theme; The extraction unit is configured to extract a target first text from the plurality of first texts based on the first text label, and extract a target second text from the plurality of second texts based on the second text label; The processing unit is configured to determine the similarity between the first to-be-verified text and the second to-be-verified text based on the similarity between the target first text and the target second text.

11. A text data processing apparatus, characterized by comprising: Comprise: A memory storing computer readable instructions; A processor reading the computer readable instructions stored in the memory to execute the method of any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, A computer readable instruction stored thereon, when executed by a processor of a computer, causes the computer to execute the method of any one of claims 1 to 9.

13. A computer program product comprising computer instructions, characterized in that, The computer instructions are executed by the processor to implement the method of any one of claims 1 to 9.

Images