Data semantic recognition method and device, storage medium and electronic device

Abstract

The invention discloses a semantic recognition method and device of data, a storage medium and an electronic device, and relates to the technical field of natural language processing, and the method comprises the following steps: carrying out data disturbance processing on a first character in first data, performing first training on a word fine tuning model in the target domain vector model according to first data after data perturbation processing, wherein the first data is data in the target domain; according to the first category and the second category of the second data, performing second training on the word fine tuning model after the first training, the second category being the category of data obtained by inputting the second data into the word fine tuning model after the first training, and the second data being data in the target domain; and constructing a positive sample pair and a negative sample pair according to the third data and the fourth data, training a sentence fine tuning model in the target domain vector model according to the positive sample pair and the negative sample pair, and performing semantic recognition on the data in the target domain according to the trained target domain vector model.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and more specifically, to a method and apparatus for semantic recognition of data, a storage medium, and an electronic device. Background Technology

[0002] The entire lifecycle of the bond market generates a vast amount of textual data, encompassing various types of information such as market information, policies and regulations, and corporate announcements. In the fields of natural language processing and text mining, vector models, as an efficient tool for analyzing complex data, can understand the diverse data in the bond market. They can transform complex textual information into high-dimensional vector forms that are understandable and computational by machines, providing a solid foundation for intelligent research in the bond market.

[0003] Vector models can include word vector models and sentence vector models. Word vector models, such as word-to-vector (word2vec) and fasttext models, effectively capture the similarity and semantic relationships between words. However, because they are trained on static corpora, word vectors cannot adjust with changing context, making it difficult to solve the problem of polysemy. On the other hand, sentence vector models, such as document-to-vector (doc2vec) or methods based on recurrent neural networks (RNNs) and convolutional neural networks (CNNs), can handle sentence-level information, but have limitations in expressing long-range dependencies and deep semantics. These limitations restrict sentence vector models from understanding specialized domains, such as the technical terms used in the bond market.

[0004] Therefore, there is a problem in related technologies that the meaning of words or sentences in a specific domain cannot be accurately identified using general word vector models or sentence vector models.

[0005] There is no effective solution yet to address the problem that general word vector models or sentence vector models cannot accurately identify the meaning of words or sentences in specific domains in related technologies. Summary of the Invention

[0006] This application provides a semantic recognition method and apparatus, storage medium and electronic device for data, to at least solve the problem in related technologies that the meaning of words or sentences in a specific field cannot be accurately identified by general word vector models or sentence vector models.

[0007] According to one embodiment of this application, a semantic recognition method for data is provided, comprising: performing data perturbation processing on a first character in first data, and performing a first training on a word fine-tuning model in a target domain vector model based on the perturbated first data, wherein the first data is data within the target domain; performing a second training on the word fine-tuning model after the first training based on a first category and a second category of second data, wherein the second category is the category of data obtained by inputting the second data into the word fine-tuning model after the first training, and the second data is data within the target domain; constructing positive sample pairs and negative sample pairs based on third data and fourth data, and training a sentence fine-tuning model in the target domain vector model based on the positive sample pairs and the negative sample pairs, wherein the third data and the fourth data are both data within the target domain; and performing semantic recognition on the data within the target domain based on the trained target domain vector model.

[0008] In an exemplary embodiment, before performing data perturbation processing on the first character in the first data, the method further includes: acquiring a plurality of fifth data within a target domain, and filtering the interference elements contained in each fifth data, wherein the interference elements include at least one of the following: whitespace characters, garbled characters, and sequential words; performing word segmentation processing on each filtered fifth data according to a word segmentation tool to obtain a plurality of character sequences; generating a plurality of training data in the target domain according to the plurality of character sequences corresponding to the plurality of fifth data, and dividing the plurality of training data into the first data, the second data, the third data, and the fourth data.

[0009] In an exemplary embodiment, data perturbation processing is performed on a first character in the first data, and a word fine-tuning model in a target domain vector model is trained for the first time based on the perturbated first data. This includes: performing data perturbation processing on the first character based on a target tag to obtain the perturbated first data; inputting the perturbated first data into the word fine-tuning model so that the word fine-tuning model outputs the recovered first data corresponding to the perturbated first data; calculating a first loss value between the recovered first data and the first data using a first loss function, and training the word fine-tuning model for the first time based on the first loss value.

[0010] In an exemplary embodiment, before performing a second training on the word fine-tuning model after the first training based on the first and second categories of the second data, the method further includes: inputting the second data into the word fine-tuning model after the first training to obtain a first feature vector corresponding to the second data; and inputting the first feature vector into a classifier so that the classifier determines the second category of the second data based on the first feature vector.

[0011] In an exemplary embodiment, a second training of the word fine-tuning model after the first training is performed based on the first category and the second category of the second data includes: comparing the first category and the second category to calculate a classification loss value between the first category and the second category; and backpropagating the classification loss value to the word fine-tuning model after the first training to adjust the first model parameters corresponding to the word fine-tuning model after the first training using the classification loss value.

[0012] In an exemplary embodiment, constructing positive and negative sample pairs based on third and fourth data includes: inputting the third data into a second-trained word fine-tuning model, so that the second-trained word fine-tuning model performs a first forward propagation on the third data through a random inactivation mechanism to obtain a second feature vector, wherein the random inactivation mechanism is used to randomly inactivate one or more neurons in the second-trained word fine-tuning model; inputting the third data again into the second-trained word fine-tuning model, so that the second-trained word fine-tuning model performs a second forward propagation on the third data through the random inactivation mechanism to obtain a third feature vector, and determining the second and third feature vectors as the positive sample pair, wherein the random inactivation mechanism randomly inactivates different neurons in the first and second forward propagation processes; inputting the fourth data into the second-trained word fine-tuning model, so that the second-trained word fine-tuning model performs a forward propagation on the fourth data through the random inactivation mechanism to obtain a fourth feature vector, and determining the fourth feature vector and the second feature vector as the negative sample pair.

[0013] In an exemplary embodiment, training a sentence fine-tuning model in the target domain vector model based on the positive sample pairs and the negative sample pairs includes: calculating a first similarity between two samples in the positive sample pairs and calculating a second similarity between two samples in the negative sample pairs; inputting the first similarity and the second similarity into a second loss function to determine a second loss value corresponding to the sentence fine-tuning model, and adjusting the second model parameters corresponding to the sentence fine-tuning vector using the second loss value, wherein the second loss function is: For the first similarity, For the second similarity, These are two samples from the positive sample pair. Let τ be the temperature constant, N be the number of negative sample pairs, and l be the two samples in the negative sample pair. i This is the second loss value.

[0014] According to another embodiment of this application, a semantic recognition device for data is also provided, comprising: a first training module, configured to perform data perturbation processing on a first character in first data, and perform a first training on a word fine-tuning model in a target domain vector model based on the first data after data perturbation processing, wherein the first data is data within the target domain; a second training module, configured to perform a second training on the word fine-tuning model after the first training based on a first category and a second category of the second data, wherein the second category is the category of data obtained by inputting the second data into the word fine-tuning model after the first training, and the second data is data within the target domain; a third training module, configured to construct positive sample pairs and negative sample pairs based on third data and fourth data, and train a sentence fine-tuning model in the target domain vector model based on the positive sample pairs and the negative sample pairs, wherein the third data and the fourth data are both data within the target domain; and a semantic recognition module, configured to perform semantic recognition on the data within the target domain based on the trained target domain vector model.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is configured to execute the semantic recognition method of the above-mentioned data at runtime.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the semantic recognition method of the data through the computer program.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program, wherein the computer program is executed by a processor to perform the semantic recognition method for the data.

[0018] In this embodiment, after perturbing the first character in the first data, the word fine-tuning model in the target domain vector model is trained for the first time based on the perturbed first data. Then, the word fine-tuning model trained after the first training is trained for the second time based on the first and second categories of the second data. Finally, the sentence fine-tuning model in the target domain vector model is trained based on positive and negative sample pairs. This completes the training of the target domain vector model, and semantic recognition of data within the target domain can then be performed based on the trained target domain vector model. In other words, the training process of the target domain vector model in this embodiment includes: training the word fine-tuning model in the target domain vector model for the first time using the perturbed first data; training the word fine-tuning model trained after the first training based on the first and second categories of the second data; and training the sentence fine-tuning model in the target domain vector model using positive and negative sample pairs. Finally, semantic recognition of data within the target domain is performed using the trained target domain vector model. The embodiments of this application can solve the problem in related technologies that the meaning of words or sentences in a specific domain cannot be accurately identified by general word vector models or sentence vector models. Furthermore, the target domain vector model trained by the embodiments of this application (including the word fine-tuning model after the second training and the sentence fine-tuning model after training) can accurately identify the meaning of words or sentences in the target domain. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a hardware structure block diagram of a computer terminal for a data semantic recognition method according to an embodiment of this application;

[0022] Figure 2 This is a flowchart of a data semantic recognition method according to an embodiment of this application;

[0023] Figure 3This is a logic diagram of a training method for a bond domain vector model based on transfer learning and contrastive learning according to an optional embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the training process in the first stage of word fine-tuning according to an optional embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the training process in the second stage of word fine-tuning according to an optional embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the training process during the sentence fine-tuning stage according to an optional embodiment of this application;

[0027] Figure 7 This is a structural block diagram of a data semantic recognition device according to an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0029] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0030] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] The specific application environment architecture or specific hardware architecture on which the execution of the semantic recognition method based on the combined data depends is described here.

[0032] The methods and embodiments provided in this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a data semantic recognition method according to an embodiment of this application. For example... Figure 1As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor (MPU) or programmable logic device (PLD)) and a memory 104 for storing data are also shown. The computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0033] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the method for determining the interaction state in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0034] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0035] This embodiment provides a semantic recognition method for data, applied to the aforementioned computer terminal. Figure 2 This is a flowchart of a semantic recognition method for data according to an embodiment of this application, which includes the following steps:

[0036] Step S202: Perform data perturbation processing on the first character in the first data, and perform the first training on the word fine-tuning model in the target domain vector model based on the first data after data perturbation processing, wherein the first data is data within the target domain;

[0037] The aforementioned data perturbation processing may include: covering, replacing, inserting, deleting, etc.

[0038] Step S204: Perform a second training on the word fine-tuning model after the first training based on the first category and the second category of the second data, wherein the second category is the category of the data obtained by inputting the second data into the word fine-tuning model after the first training, and the second data is data within the target domain;

[0039] Step S206: Construct positive sample pairs and negative sample pairs based on the third data and the fourth data, and train the sentence fine-tuning model in the target domain vector model based on the positive sample pairs and the negative sample pairs, wherein the third data and the fourth data are both data within the target domain;

[0040] Among them, the first data, the second data, the third data and the fourth data mentioned above are all data used to train the model, and they can be the same data or different data.

[0041] Step S208: Perform semantic recognition on the data in the target domain based on the trained target domain vector model.

[0042] Through the above steps, after perturbing the first character in the first data, the word fine-tuning model in the target domain vector model is trained for the first time based on the perturbed first data. Then, the word fine-tuning model trained after the first training is trained for the second time based on the first and second categories of the second data. Finally, the sentence fine-tuning model in the target domain vector model is trained based on positive and negative sample pairs. This completes the training of the target domain vector model, and the trained target domain vector model can then be used to perform semantic recognition on data within the target domain. In other words, the training process of the target domain vector model in this embodiment includes: training the word fine-tuning model in the target domain vector model for the first time using the perturbed first data; training the word fine-tuning model trained after the first training based on the first and second categories of the second data; and training the sentence fine-tuning model in the target domain vector model using positive and negative sample pairs. The trained target domain vector model is then used to perform semantic recognition on data within the target domain. The embodiments of this application can solve the problem in related technologies that the meaning of words or sentences in a specific domain cannot be accurately identified by general word vector models or sentence vector models. Furthermore, the target domain vector model trained by the embodiments of this application (including the word fine-tuning model after the second training and the sentence fine-tuning model after training) can accurately identify the meaning of words or sentences in the target domain.

[0043] Optionally, before performing data perturbation processing on the first character in the first data in step S202 above, the method further includes: acquiring multiple fifth data within the target domain, and filtering the interference elements contained in each fifth data, wherein the interference elements include at least one of the following: whitespace characters, garbled characters, and sequential words; performing word segmentation processing on each filtered fifth data according to a word segmentation tool to obtain multiple character sequences; generating multiple training data in the target domain according to the multiple character sequences corresponding to the multiple fifth data, and dividing the multiple training data into the first data, the second data, the third data, and the fourth data.

[0044] Understandably, before training the target domain vector model, it is necessary to determine the training data (i.e., the first data, the second data, the third data, and the fourth data). Specifically:

[0045] 1) Obtain multiple fifth data in the target domain: A large amount of raw data can be collected from sources such as websites, databases, and documents related to the target domain (such as the bond domain). This data may include forms such as HyperText Markup Language (html) text of web pages, word documents (a document form), and Portable Document Format (pdf) files.

[0046] For example: Suppose a large number of news articles, legal regulations texts, reports, etc. about bonds are crawled from financial websites. This collection of raw data is the "fifth data".

[0047] 2) Interference element filtering and processing: Remove interference elements in the data, such as blank characters, garbled characters, and sequential words that do not carry semantic information, to improve the quality of text data.

[0048] For example: In a word text, tags such as "First" and "##*" are regarded as interference elements and need to be removed. Sequential words such as "One." and "Two." frequently appear in structured texts but are not helpful for understanding the text semantics and also need to be filtered.

[0049] 3) Word segmentation processing: Use a word segmentation tool to segment each filtered text data into a sequence of words. The word segmentation tool may include a proprietary vocabulary list of the target domain.

[0050] Example: Suppose there is a filtered sentence: "The bank issued a new batch of bonds." After using the word segmentation tool, the character sequence: "bank", "issued", "a", "new", "batch", "bonds" can be obtained.

[0051] 4) Generate training data and divide the training data into first data, second data, third data, and fourth data: Further transform the text data after word segmentation processing into the format required for training the model, and then divide these data into different subsets, such as first data (training data for the first training of word fine-tuning), second data (training data for the second training of word fine-tuning), third data (training data for sentence fine-tuning), and fourth data (training data for sentence fine-tuning), to meet the needs of model training.

[0052] Example: After filtering and word segmentation of the collected raw data, thousands of training sentences can be generated. 50% of the data in these sentences may be used for word fine-tuning (first data and second data), and 50% of the data is used for sentence fine-tuning (third data and fourth data).

[0053] Optionally, step S202 above, which involves perturbing the first character in the first data and performing the first training of the word fine-tuning model in the target domain vector model based on the perturbed first data, includes: perturbing the first character based on the target tag to obtain the perturbed first data; inputting the perturbed first data into the word fine-tuning model so that the word fine-tuning model outputs the recovered first data corresponding to the perturbed first data; calculating the first loss value between the recovered first data and the first data using a first loss function, and performing the first training of the word fine-tuning model based on the first loss value.

[0054] Understandably, after determining the first, second, third, and fourth data points, the target domain vector model can be trained. The above technical solution limits the initial training of the word fine-tuning model within the target domain vector model (which can be based on a masked language model). Specifically:

[0055] 1) Data perturbation processing: Select some words in the first data (i.e. the training data used for word fine-tuning) and mask them with a special label [MASK] (i.e. target label), while keeping other words and sentence structure unchanged, in order to perform data perturbation processing on the first data.

[0056] For example, suppose there is a sentence: "The bank issued a new batch of bonds." If this sentence is selected as the first character for data perturbation, the possible masking result is: "The bank issued a new batch of [MASK]." "Bonds" is randomly selected and replaced with [MASK].

[0057] 2) Input the first data after data perturbation into the word fine-tuning model: Input the first data after data perturbation into the pre-trained word fine-tuning model (which can be a WoBERT model). The word fine-tuning model predicts the words replaced by the [MASK] tag based on contextual information.

[0058] For example, when a perturbed sentence is input into a word fine-tuning model, the model will attempt to predict the word represented by the [MASK] tag based on the contextual information of "bank", "issue", and "new batch".

[0059] 3) Word Recovery and Loss Calculation: The word fine-tuning model outputs a probability distribution representing the possible words at the [MASK] position. The word with the highest probability is selected as the prediction result of the word fine-tuning model, i.e., the recovered first data. Then, the difference between the predicted word output by the word fine-tuning model and the actual word is calculated using a first loss function (e.g., cross-entropy loss function), i.e., the first loss value.

[0060] For example, the word fine-tuning model might predict the probability distribution of multiple words such as "finance," "currency," and "bond" at the [MASK] position. If the model predicts "bond" as having the highest probability, then "bond" becomes the first data point after recovery. By comparing the word fine-tuning model's prediction of "bond" with the actual "bond," if the prediction is correct and the prediction is "finance," then the first loss value is calculated based on the difference between the word predicted by the word fine-tuning model and the actual word.

[0061] 3) First Training: The word fine-tuning model is trained based on the first loss value, with the goal of reducing the first loss value to make the word fine-tuning model more accurate in predicting masked words. The parameters of the word fine-tuning model can be updated using backpropagation algorithms and optimizers.

[0062] For example, suppose the word fine-tuning model predicts a probability of 0.6 for "currency" and 0.4 for "finance," but the actual word is "bond." Because of the discrepancy between the prediction and the actual result, the word fine-tuning model calculates a first loss value. Then, by adjusting the parameters of the word fine-tuning model, it can more accurately predict "bond" the next time it encounters the same or similar context.

[0063] Optionally, before performing a second training on the word fine-tuning model after the first training based on the first and second categories of the second data in step S204 above, the method further includes: inputting the second data into the word fine-tuning model after the first training to obtain a first feature vector corresponding to the second data; and inputting the first feature vector into a classifier so that the classifier determines the second category of the second data based on the first feature vector.

[0064] Understandably, after the first training of the word fine-tuning model, it is necessary to determine the first and second categories of the second data in order to conduct a second training of the word fine-tuning model. The technical solution for determining the first and second categories of the second data is as follows:

[0065] 1) Input the second data into the word fine-tuning model after the first training: After completing the first training based on the masked language model, the output of the word fine-tuning model after the first training, i.e., the feature vectors of the words, is used to perform a higher-level task, namely, classifying the second data. To do this, the second data (which can be sentences or documents with classification labels) is fed into the word fine-tuning model after the first training to obtain the first feature vector of each sentence or document.

[0066] Suppose the second dataset contains analytical articles about the bond market, and each article has a category label, such as "Macroeconomics," "Corporate Bond Analysis," or "Policy Interpretation." The text content of these articles is input into a word fine-tuning model after its first training iteration. This model generates a first feature vector for each article based on the words and sentence structure.

[0067] 2) The obtained first feature vector is input into the classifier: The first feature vector output from the word fine-tuning model after the first training contains rich semantic information, especially the understanding of professional vocabulary in the bond field. The first feature vector is then fed into a classifier, whose role is to predict or determine the second category of the text based on the first feature vector.

[0068] For example, after the word fine-tuning model outputs the first feature vector for each article during the initial training, this first feature vector is fed into a pre-prepared classifier. The classifier could be a simple logistic regression or support vector machine, or it could be a more complex neural network. The classifier learns the association between the first feature vector and the article category, attempting to predict which category the article belongs to based on the vector.

[0069] 3) Determine the second category of the second data: The classifier will predict the second category of the second data (i.e., sentences or documents with category labels) based on the first feature vector input.

[0070] For example, the classifier analyzes the first feature vector of an article and predicts that it belongs to the "corporate debt analysis" category. This shows that the word fine-tuning model after the first training successfully captured the professional vocabulary and concepts related to "corporate debt analysis" when extracting the semantic features of the article, enabling the classifier to make the correct classification decision.

[0071] 4) Further details: The classifier's learning and prediction are actually based on the output of the word fine-tuning model after the first training. This means that the classifier's task is not only to identify the category of the article, but also to indirectly help the word fine-tuning model after the first training to better understand the terminology and concepts in the bond domain. Through a feedback mechanism (i.e., calculating the error between the classifier's prediction result (second category) and the actual label (first category), and then backpropagating this error to the word fine-tuning model after the first training), the word fine-tuning model after the first training can continuously optimize its representation of the target domain-specific vocabulary. This allows the word fine-tuning model to not only better handle word-level tasks, but also understand the semantics of more complex sentences and documents, providing a more powerful word representation capability for the final bond domain vector model.

[0072] Optionally, step S204 above, which involves performing a second training on the word fine-tuning model after the first training based on the first and second categories of the second data, includes: comparing the first category and the second category to calculate the classification loss value between the first category and the second category; and backpropagating the classification loss value to the word fine-tuning model after the first training to adjust the first model parameters corresponding to the word fine-tuning model after the first training using the classification loss value.

[0073] Understandably, after determining the first and second categories, the word fine-tuning model trained in the first training can be trained a second time. Specifically:

[0074] 1) Calculate the classification loss: Compare the classifier's predicted second category with the manually labeled actual first category. If the prediction is accurate, the loss will be smaller; if the prediction is incorrect, the loss will be larger. The classification loss can be calculated using the cross-entropy loss function.

[0075] For example, suppose a classifier predicts a bond market analysis article belongs to the "Macroeconomics" category (Category 2), while the article is actually tagged "Corporate Bond Analysis" (Category 1). Using the cross-entropy loss function, the loss value between the classifier's predicted "Macroeconomics" category and the actual "Corporate Bond Analysis" category can be calculated. If the classifier's prediction is completely wrong, the loss value will be large, and vice versa.

[0076] 2) Backpropagation and Parameter Tuning: The classification loss value is backpropagated to the word fine-tuning model after the first training iteration. This means that the parameters of the word fine-tuning model after the first training iteration (i.e., the first model parameters) will be updated based on this loss value. The goal of parameter tuning is to reduce the loss value so that the model can better reflect the characteristics of the category to which a word belongs when predicting words, thereby improving the accuracy of the classifier.

[0077] For example, if there is a significant difference between the classifier's predicted category "macroeconomics" and the actual category "corporate bond analysis," meaning the classification loss is high, this loss value will be passed back to the word fine-tuning model after the first training iteration via backpropagation. The word fine-tuning model after the first training iteration will adjust the parameters of the first model based on the classification loss value. Specifically, it will adjust the neural network weights associated with the incorrectly predicted words to reduce future classification errors on similar inputs.

[0078] Optionally, step S206 above, which constructs positive and negative sample pairs based on the third and fourth data, includes: inputting the third data into the word fine-tuning model after the second training, so that the word fine-tuning model after the second training performs a first forward propagation on the third data through a random inactivation mechanism to obtain a second feature vector, wherein the random inactivation mechanism is used to randomly inactivate one or more neurons in the word fine-tuning model after the second training; inputting the third data again into the word fine-tuning model after the second training, so that the word fine-tuning model after the second training performs a first forward propagation on the third data through a random inactivation mechanism to obtain a second feature vector. The activation mechanism performs a second forward propagation on the third data to obtain a third feature vector, and determines the second feature vector and the third feature vector as the positive sample pair. The random deactivation mechanism randomly deactivates different neurons during the first and second forward propagations. The fourth data is then input into the second-trained word fine-tuning model, so that the second-trained word fine-tuning model performs forward propagation on the fourth data through the random deactivation mechanism to obtain a fourth feature vector, and determines the fourth feature vector and the second feature vector as the negative sample pair.

[0079] Understandably, before training the sentence fine-tuning model, it is necessary to create positive and negative sample pairs, specifically:

[0080] 1) Construction of positive sample pairs: In the word fine-tuning model after the second training, the third data (a portion of the training data used for sentence fine-tuning) undergoes two forward propagations. During these two propagations, a random deactivation mechanism is used internally, so that different neurons are temporarily shut down during each propagation. The two propagations generate a second feature vector and a third feature vector, respectively. Due to the randomness of the random deactivation mechanism, these two feature vectors, although originating from the same input, represent different perspectives or angles, and therefore can be used as a pair of positive samples for comparative learning.

[0081] For example, suppose the third data point is a sentence about bond ratings: "This company's bond rating is AAA." When this sentence is input into the word fine-tuning model after the second training, the first forward propagation might shut down time-related neurons in the model, while the second forward propagation might shut down neurons related to the company's financial condition. Even so, the two propagations can still capture the main information of the sentence, namely "bond rating" and "AAA," so the generated second and third feature vectors will be very close, forming a positive sample pair.

[0082] 2) Negative Sample Pair Construction: In addition to constructing positive sample pairs, contrastive learning also requires negative sample pairs to distinguish different inputs. The fourth data (part of the training data used for sentence fine-tuning) will also undergo the same forward propagation process to generate a fourth feature vector. However, since the content of the fourth data is different from that of the third data, their feature vectors should also be significantly different. Therefore, the fourth feature vector and the second (or third) feature vector constitute a negative sample pair, which is used to train the model to learn to distinguish different sentences.

[0083] For example, if the fourth data point is a sentence about stock market fluctuations: "The stock market fluctuated wildly today," then this sentence is completely different from a sentence about bond ratings, representing a different topic. When this sentence is input into the word fine-tuning model after the second training, the generated fourth feature vector should differ significantly from the second or third feature vector. Therefore, the fourth feature vector can form a negative sample pair with the second (or third) feature vector, telling the model that the two sentences are semantically unrelated.

[0084] Optionally, step S206 above, training the sentence fine-tuning model in the target domain vector model based on the positive sample pairs and the negative sample pairs, includes: calculating a first similarity between two samples in the positive sample pairs and calculating a second similarity between two samples in the negative sample pairs; inputting the first similarity and the second similarity into a second loss function to determine a second loss value corresponding to the sentence fine-tuning model, and adjusting the second model parameters corresponding to the sentence fine-tuning vector using the second loss value, wherein the second loss function is: For the first similarity, For the second similarity, These are two samples from the positive sample pair. Let τ be the temperature constant, N be the number of negative sample pairs, and l be the two samples in the negative sample pair. i This is the second loss value.

[0085] Understandably, once the positive and negative sample pairs are determined, the sentence model can be trained. Specifically:

[0086] 1) Calculating the first and second similarities: In the sentence fine-tuning stage, positive and negative sample pairs need to be constructed. Positive sample pairs refer to two sentences that are semantically similar or close, while negative sample pairs refer to two sentences that are semantically unrelated or significantly different. Next, the model will calculate the first similarity between the two sentences in the positive sample pair and the second similarity between the two sentences in the negative sample pair.

[0087] For example, suppose there are two positive sample sentences: "The bank issued a new batch of bonds." and "Recently, the China Development Bank publicly issued a new batch of bonds." These two sentences are semantically very similar, so the model will calculate the first similarity between them, which should be a high value. Meanwhile, a sentence completely unrelated to the positive sample, such as "The weather is sunny today, suitable for outdoor activities," is selected and combined with either of the above positive sample sentences to form a negative sample pair. The model will then calculate the second similarity between these two sentences, which should be a low value.

[0088] 2) Input to the second loss function: The calculated first similarity (similarity between positive sample pairs) and second similarity (similarity between negative sample pairs) are input into the second loss function. The goal of the second loss function is to maximize the first similarity between positive sample pairs while minimizing the second similarity between negative sample pairs. This contrastive learning approach enhances the model's ability to distinguish between semantically similar and dissimilar sentences.

[0089] For example, if the first similarity of a positive sample pair is 0.9 (meaning they are very similar), and the second similarity of a negative sample pair is 0.1 (meaning they are not similar), then these two similarities will be used as inputs to calculate the contrastive loss.

[0090] To better understand the process of the semantic recognition method for the above data, the implementation flow of the semantic recognition method for the above data will be described below in conjunction with optional embodiments, but this is not intended to limit the technical solution of the embodiments of this application.

[0091] In an optional embodiment of this application, a training method for a bond domain vector model based on transfer learning and contrastive learning is provided. The vector model may include a word vector model and a sentence vector model. A word vector model is a technique for converting words into machine-processable vector representations. It learns the similarity and semantic relationships between words by training on a large-scale corpus, thereby representing words as vectors in a high-dimensional space. A sentence vector model is a method for mapping text sequences (such as sentences) to a vector space. Its basic principle is to capture the semantic information in the text sequence and represent the semantic features of each sentence as a vector in a high-dimensional space.

[0092] Figure 3 This is a logic diagram of a training method for a bond domain vector model based on transfer learning and contrastive learning according to an optional embodiment of this application, as shown below. Figure 3 As shown:

[0093] The optional embodiments of this application are divided into two parts: data processing and training stages. The data processing mainly includes four steps: sentence segmentation, quality filtering, redundancy removal, and word segmentation, ultimately generating training and testing datasets. The training stage is divided into training the word fine-tuning model and training the sentence fine-tuning model. The training of the word fine-tuning model includes: training WoBERT (a Chinese word vector model) combined with a masked language model and training WoBERT combined with a classification task. The training of the sentence fine-tuning model includes: simple comparative sentence vector training and knowledge distillation, ultimately generating a bond domain vector model (i.e., the trained target domain vector model).

[0094] Data quality has a significant impact on the model. In the data processing stage, optional embodiments of this application analyze and process the raw data, sequentially feeding the massive amounts of collected professional domain data into sentence segmentation, quality filtering, redundancy removal, and word segmentation, performing text-level, sentence-level, and word-level processing on the data respectively, resulting in high-quality vector model training and testing datasets.

[0095] During the model training phase, the WoBERT model and the SimCSE model (Simple Contrastive Sentence Vector Training Model) are integrated, and two stages, word fine-tuning and sentence fine-tuning, are established. This not only enhances the ability to distinguish and recognize domain-specific features but also improves the expressive ability of domain-related sentences. The optional embodiment of this application uses the Bidirectional Encoder Representation from Transformers (BERT) model as its base model. Based on the transformer model, the bidirectional encoder is a pre-trained language representation model. In the word fine-tuning stage, also known as the transfer learning stage, the WoBERT model, a Chinese BERT model based on word granularity, is used to transfer the general-domain large-scale language word model to the professional domain vector model on the Masked Language Model (MLM) task and the Classification Task (CLS) task, respectively, enabling the vector model to express domain-specific vocabulary. In the sentence fine-tuning stage, also known as the contrastive learning stage, the SimCSE model, a sentence embedding contrastive learning model based on unsupervised learning, is used to improve the vector model's expressive ability of sentences in the professional domain. Finally, the vector models from the first two stages are distilled to reduce model size and meet the requirements of real-time online computation. Specifically:

[0096] (1) Data processing stage:

[0097] 1) In optional embodiments of this application, the collected professional domain data is read. This data includes HyperText Markup Language (HTML) text from web pages, as well as Word text (a text format) and Portable Document Format (PDF) text. HTML text is extracted using regular expressions; Word and PDF text are extracted using a Python (a programming language) reading toolkit.

[0098] 2) In the quality filtering stage, the optional embodiments of this application employ a rule-based quality identification method to identify whitespace characters, garbled characters, etc. (i.e., interference elements) in the text. Since retaining these characters would easily lead to resource waste, they are directly removed. Considering that some data in the optional embodiments of this application comes from structured texts such as laws and regulations, the writing style of such texts is usually connected by a set of sequential words, such as: "Firstly; Secondly;...". These sequential words will not produce any ambiguity or misinterpretation of the correct expression of the text. The rule-based quality identification method statistically analyzes several writing styles of structured texts and removes them, further improving the data quality at the sentence level.

[0099] 3) The redundancy removal stage in the optional embodiments of this application involves redundancy identification and cleaning at the sentence level. In the optional embodiments of this application, the text data after quality filtering is split into sentences, using punctuation marks such as periods to divide the text data into multiple sentences. Identical sentences are deduplicated, retaining only one. This is because training with a large amount of repetitive data is of no help to subsequent model training and may even affect the overall data distribution, causing the model to be biased when training on domain-specific data.

[0100] 4) In the optional embodiment of this application, the word segmentation stage mainly processes word-level data. Unlike English, Chinese language consists of multiple words separated by spaces to form multiple sentences. Before processing Chinese language data, word segmentation methods must be used to segment Chinese sentences into multiple words before feeding them into the model for training. Considering that traditional Chinese word segmentation methods would fragment and scramble specialized vocabulary, the optional embodiment of this application generates a rich vocabulary list of specialized terms based on the statistical characteristics of the data. Then, based on Jieba Chinese Word Segmentation (jieba segmentation) and the tokenizer segmentation method built into the WoBERT model, the sentence-level dataset is segmented into multiple word sequences (i.e., character sequence pairs) to obtain word-level training data.

[0101] (2) Training phase.

[0102] In the optional embodiment of this application, the word fine-tuning stage uses the WoBERT model, employing transfer learning with words as the basic unit. This not only reduces semantic uncertainty but also shortens the text sequence length, reducing modeling complexity and improving processing speed. The word fine-tuning stage transfers a pre-trained model from the general domain to the bond domain, using a Masked Language Model (MLM) as the transfer task to improve the model's ability to recognize bond-specific terms. A classification task is also constructed to further enhance the model's ability to recognize bond-specific terms.

[0103] 1) The first stage of word fine-tuning (i.e. the first training of the word fine-tuning model) is the transfer learning task of the masked language model. Figure 4 This is a schematic diagram of the first stage of the training process for word fine-tuning according to an optional embodiment of this application, as shown below. Figure 4 As shown, tokens in the input sequence are randomly masked. The randomization method involves selecting 15% of the words in the input sequence (i.e., the first data), replacing 80% with a [MASK] mask, 10% with a random token, and leaving the remaining 10% as the original words. The model's task is to predict the original token at the [MASK] position. When selecting 80% of the words to be replaced with [MASK], this method focuses on words from the proprietary Chinese vocabulary in the first stage. This manual intervention also makes the model's transfer task more focused on the bond domain.

[0104] 2) The second stage of word fine-tuning (i.e., the second training of the word fine-tuning model) is the fine-tuning training of the downstream classification task. Figure 5 This is a schematic diagram of the second stage of the training process for word fine-tuning according to an optional embodiment of this application, as shown below. Figure 5 As shown, an optional embodiment of this application utilizes existing text tags in the article containing the input data (i.e., Figure 5 The labels were used to construct downstream classification tasks, further fine-tuning the model trained in the first stage. For example... Figure 5 As shown, the complete text sequence is input into the model trained in the first stage, and the first token of the sequence, [CLS], is output as the input of the classification model. The loss of the classification model is then backpropagated to the WoBERT model to further fine-tune the model trained in the first stage.

[0105] 3) The SimCSE model is selected for the sentence fine-tuning stage (i.e., training the sentence fine-tuning model) in the optional embodiments of this application. Figure 6 This is a schematic diagram of the training process during the sentence fine-tuning stage according to an optional embodiment of this application, such as... Figure 6As shown, based on the transferred WoBERT model, a contrastive learning framework is used to fine-tune the sentence vector model through unsupervised training. The purpose of contrastive learning is to increase the distance between negative examples and decrease the distance between positive examples. Utilizing the Dropout feature built into the pre-trained model as an augmentation method, manual annotation is no longer required. The model not only reduces its dependence on the accuracy and quantity of labeled samples but also improves its expressive ability for relevant sentences in the bond domain.

[0106] 4) The first stage of sentence fine-tuning is the construction of positive and negative example sentence pairs. The vectors obtained from two forward propagations are used as positive sample pairs, and the vectors generated by the remaining sentences in the same batch are used as negative sample pairs. Specifically:

[0107] For each sentence (i.e., the third data), two forward propagations are performed to obtain positive sample pairs. During the forward propagation of the word fine-tuning model, a Dropout mask is introduced. This randomness ensures that two distinct sentence vectors are obtained as positive sample pairs.

[0108] For each vector, the vectors generated by other different sentences (i.e., the fourth data) in the same batch are used as negative samples;

[0109] 5) The second stage of sentence fine-tuning is the model training stage. This involves fine-tuning the model using positive and negative sample pairs of input words. The goal of this training is to increase the similarity of sentence vectors for positive samples and decrease the similarity for sentence vectors for negative sample pairs. The objective function uses the loss function of a contrastive learning framework, resulting in lower loss values ​​and smaller penalties for positive sample pairs, and higher loss values ​​and larger penalties for negative sample pairs. The formula is: and Let represent the cosine similarity of two similar vectors (in positive and negative sample pairs), respectively, and let be a temperature constant. Based on this penalized backpropagation, the sentence vector model is fine-tuned.

[0110] 6) The third stage of fine-tuning in the optional embodiments of this application is the model distillation stage. Because the number of vector models trained in the previous stages and their memory consumption are too large, the online actual computation will be affected by the model loading speed. Furthermore, the online actual computation inference process is performed on the Central Processing Unit (CPU), and the computation speed is also slower than the computation speed of the Graphics Processing Unit (GPU) in the training stage. Therefore, it is necessary to distill the model while ensuring the accuracy of the vector model, thereby reducing the pressure on online real-time inference. The optional embodiments of this application compress and migrate the original vector models obtained in the previous stages to a BERT model with only three hidden layers.

[0111] In summary, the optional embodiments of this application used over 140,000 website articles and over 500 PDF attachments in the training and testing phases of each fine-tuned model. After deduplication, filtering of special characters, and format conversion, over 720,000 domain-specific training sentences were obtained. In the word model fine-tuning classification task fine-tuning phase, the classification label was the article label of the article containing the paragraph. In the sentence model fine-tuning phase, 1000 sentences were randomly selected as the validation set and 100 sentences as the test set. Similar texts were generated from the validation and test sets and scored manually, with scores ranging from 0 to 5, where 0 is the least similar and 5 is the most similar. The models were verified and compared from both word semantic similarity and sentence semantic similarity. The vector model with only word fine-tuning (WoBERT), the vector model with word and sentence fine-tuning (WoBERT+SimCSE), and the distillation model (+distillation) of the vector model with word and sentence fine-tuning were compared. Specifically:

[0112] First, we compared the semantic similarity. Based on industry knowledge in the bond sector, we constructed synonym and near-synonym pairs to compare the model's performance. We constructed 26 synonym pairs and 66 near-synonym pairs, such as comparing "financial crisis" with "economic crisis." The model's recall rate is shown in Table 1.

[0113] Table 1

[0114] index WoBERT WoBERT+SimCSE +distillation Synonym: Recall 0.73 0.78 0.5 Synonym: Recall 0.25 0.3540 0.4807

[0115] As shown in Table 1, both WoBERT (i.e., word fine-tuning model) and WoBERT+SimCSE model have good recall effects for synonyms and near-synonyms. After adding SimCSE (i.e., sentence fine-tuning model), the recall effect for synonyms and near-synonyms is further improved.

[0116] Secondly, semantic similarity was compared. In the sentence vector fine-tuning stage, 100 similar sentence pairs were manually scored. The results are shown in Table 2. As can be seen from Table 2, after transfer learning, the WoBERT model's performance was significantly improved, enabling it to better identify and distinguish sentences in specific domains. After supplementing with SimCSE, the model's performance was further improved, with enhanced sentence differentiation and recognition capabilities in the bond domain.

[0117] Table 2

[0118] Model Indicator value WoBERT 0.7476 WoBERT+SimCSE 0.8190 +Distillation 0.8223

[0119] In addition, the WoBERT+SimCSE model was used for testing on out-of-domain data. The out-of-domain data consisted of 32 manually labeled question pairs not used in this training process, each containing 2-4 similar sentences, for a total of 79 sentence pairs. The evaluation metric used was Recall@K (a metric used to evaluate the performance of recommendation systems or ranking models), which measures the model's recall. The Recall@1, Recall@5, and Recall@10 values ​​are shown in Table 3. As can be seen from Table 3, the model also performed well on out-of-domain data.

[0120] Table 3

[0121] index WoBERT+SimCSE +distillation Recall@1 0.4115 0.3802 Recall@5 0.8073 0.8437 Recall@10 0.8646 0.8958

[0122] The optional embodiments of this application can be applied to a variety of application scenarios. For example, in the intelligent customer service scenario, the questions entered by the user on the front end and the knowledge text stored in the system back end are converted into vectors, and the similarity between the two is compared. This enables the machine to automatically understand the user's intent and discover the answer. At the same time, it can be combined with traditional methods such as manual addition to provide more accurate answers or solutions.

[0123] In personalized recommendation scenarios, vector models in the bond domain can be used to filter articles relevant to user interests from large amounts of article data. The model transforms historical user behavior data into vectors and processes them according to a certain strategy to obtain user feature vectors. Similarity comparisons are then performed between these user feature vectors and the article feature vectors derived from the articles, thereby identifying articles that the user is interested in. Furthermore, personalized recommendation scenarios often employ a combination of strategies, integrating text vector similarity strategies with other strategies such as latest article strategies and most popular article strategies to better uncover user interests and achieve personalized recommendations.

[0124] In financial sentiment analysis, sentiment analysis is a crucial component. This invention's vector model transforms comments or news reports into vectors, enabling the determination of the text's sentiment orientation and analysis of investor sentiment. Furthermore, financial sentiment texts often contain a wealth of thematic information, such as company stock prices, company performance, and market trends. Using the model, text can be represented as high-dimensional vectors, and clustering algorithms can be used to classify the text, easily identifying the thematic categories to which the text belongs, helping investors better grasp market dynamics.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0126] Figure 7 This is a structural block diagram of a data semantic recognition device according to an embodiment of this application; as shown below. Figure 5 As shown, it includes:

[0127] The first training module 72 is used to perform data perturbation processing on the first character in the first data, and to perform the first training on the word fine-tuning model in the target domain vector model based on the first data after data perturbation processing, wherein the first data is data within the target domain;

[0128] The second training module 74 is used to perform a second training on the word fine-tuning model after the first training based on the first category and the second category of the second data, wherein the second category is the category of the data obtained by inputting the second data into the word fine-tuning model after the first training, and the second data is data within the target domain;

[0129] The third training module 76 is used to construct positive sample pairs and negative sample pairs based on the third data and the fourth data, and to train the sentence fine-tuning model in the target domain vector model based on the positive sample pairs and the negative sample pairs, wherein the third data and the fourth data are both data within the target domain;

[0130] The semantic recognition module 78 is used to perform semantic recognition on the data in the target domain based on the trained target domain vector model.

[0131] Using the aforementioned apparatus, after perturbing the first character in the first data, the word fine-tuning model in the target domain vector model is trained for the first time based on the perturbed first data. Then, the word fine-tuning model trained after the first training is trained for the second time based on the first and second categories of the second data. Finally, the sentence fine-tuning model in the target domain vector model is trained based on positive and negative sample pairs. This completes the training of the target domain vector model, and semantic recognition of data within the target domain can then be performed based on the trained target domain vector model. In other words, the training process of the target domain vector model in this embodiment includes: training the word fine-tuning model in the target domain vector model for the first time using the perturbed first data; training the word fine-tuning model trained after the first training based on the first and second categories of the second data; and training the sentence fine-tuning model in the target domain vector model using positive and negative sample pairs. Finally, semantic recognition of data within the target domain is performed using the trained target domain vector model. The embodiments of this application can solve the problem in related technologies that the meaning of words or sentences in a specific domain cannot be accurately identified by general word vector models or sentence vector models. Furthermore, the target domain vector model trained by the embodiments of this application (including the word fine-tuning model after the second training and the sentence fine-tuning model after training) can accurately identify the meaning of words or sentences in the target domain.

[0132] In an exemplary embodiment, the first training module 72 is further configured to acquire multiple fifth data within the target domain, and filter the interference elements contained in each fifth data, wherein the interference elements include at least one of the following: whitespace characters, garbled characters, and sequential words; perform word segmentation on each filtered fifth data using a word segmentation tool to obtain multiple character sequences; generate multiple training data in the target domain based on the multiple character sequences corresponding to the multiple fifth data, and divide the multiple training data into the first data, the second data, the third data, and the fourth data.

[0133] In an exemplary embodiment, the first training module 72 is further configured to perform data perturbation processing on the first character based on the target tag to obtain the first data after data perturbation processing; input the first data after data perturbation processing into the word fine-tuning model so that the word fine-tuning model outputs the recovered first data corresponding to the first data after data perturbation processing; calculate the first loss value between the recovered first data and the first data through a first loss function, and perform the first training on the word fine-tuning model based on the first loss value.

[0134] In an exemplary embodiment, the second training module 74 is further configured to input the second data into the word fine-tuning model after the first training to obtain a first feature vector corresponding to the second data; and input the first feature vector into a classifier so that the classifier determines the second category of the second data based on the first feature vector.

[0135] In an exemplary embodiment, the second training module 74 is further configured to compare the first category and the second category to calculate a classification loss value between the first category and the second category; and to backpropagate the classification loss value to the word fine-tuning model after the first training to adjust the first model parameters corresponding to the word fine-tuning model after the first training using the classification loss value.

[0136] In an exemplary embodiment, the third training module 76 is further configured to input the third data into the word fine-tuning model after the second training, so that the word fine-tuning model after the second training performs a first forward propagation on the third data through a random inactivation mechanism to obtain a second feature vector, wherein the random inactivation mechanism is used to randomly inactivate one or more neurons in the word fine-tuning model after the second training; input the third data again into the word fine-tuning model after the second training, so that the word fine-tuning model after the second training performs a second forward propagation on the third data through the random inactivation mechanism to obtain a third feature vector, and determine the second feature vector and the third feature vector as the positive sample pair, wherein the neurons randomly inactivated by the random inactivation mechanism are different in the first forward propagation and the second forward propagation; input the fourth data into the word fine-tuning model after the second training, so that the word fine-tuning model after the second training performs a forward propagation on the fourth data through the random inactivation mechanism to obtain a fourth feature vector, and determine the fourth feature vector and the second feature vector as the negative sample pair.

[0137] In an exemplary embodiment, the third training module 76 is further configured to calculate a first similarity between two samples in the positive sample pair and a second similarity between two samples in the negative sample pair; input the first similarity and the second similarity into a second loss function to determine a second loss value corresponding to the sentence fine-tuning model, and adjust the second model parameters corresponding to the sentence fine-tuning vector using the second loss value, wherein the second loss function is: For the first similarity, For the second similarity, These are two samples from the positive sample pair. Let τ be the temperature constant, N be the number of negative sample pairs, and l be the two samples in the negative sample pair. i This is the second loss value.

[0138] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.

[0139] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0140] S1, perform data perturbation processing on the first character in the first data, and perform the first training on the word fine-tuning model in the target domain vector model based on the first data after data perturbation processing, wherein the first data is data within the target domain;

[0141] S2, perform a second training on the word fine-tuning model after the first training based on the first category and the second category of the second data, wherein the second category is the category of the data obtained by inputting the second data into the word fine-tuning model after the first training, and the second data is data within the target domain;

[0142] S3, construct positive sample pairs and negative sample pairs based on the third data and the fourth data, and train the sentence fine-tuning model in the target domain vector model based on the positive sample pairs and the negative sample pairs, wherein the third data and the fourth data are both data within the target domain;

[0143] S4, perform semantic recognition on the data in the target domain based on the trained target domain vector model.

[0144] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0145] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0146] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0147] S1, perform data perturbation processing on the first character in the first data, and perform the first training on the word fine-tuning model in the target domain vector model based on the first data after data perturbation processing, wherein the first data is data within the target domain;

[0148] S2, perform a second training on the word fine-tuning model after the first training based on the first category and the second category of the second data, wherein the second category is the category of the data obtained by inputting the second data into the word fine-tuning model after the first training, and the second data is data within the target domain;

[0149] S3, construct positive sample pairs and negative sample pairs based on the third data and the fourth data, and train the sentence fine-tuning model in the target domain vector model based on the positive sample pairs and the negative sample pairs, wherein the third data and the fourth data are both data within the target domain;

[0150] S4, perform semantic recognition on the data in the target domain based on the trained target domain vector model.

[0151] Embodiments of this application also provide a computer program product, including a computer program that is executed by a processor through the steps of any of the above method embodiments.

[0152] Optionally, in this embodiment, the above-mentioned computer program product can be executed by a processor using the following steps:

[0153] S1, perform data perturbation processing on the first character in the first data, and perform the first training on the word fine-tuning model in the target domain vector model based on the first data after data perturbation processing, wherein the first data is data within the target domain;

[0154] S2, perform a second training on the word fine-tuning model after the first training based on the first category and the second category of the second data, wherein the second category is the category of the data obtained by inputting the second data into the word fine-tuning model after the first training, and the second data is data within the target domain;

[0155] S3, construct positive sample pairs and negative sample pairs based on the third data and the fourth data, and train the sentence fine-tuning model in the target domain vector model based on the positive sample pairs and the negative sample pairs, wherein the third data and the fourth data are both data within the target domain;

[0156] S4, perform semantic recognition on the data in the target domain based on the trained target domain vector model.

[0157] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0158] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0159] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0160] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for semantic recognition of data, characterized in that, include: The first character in the first data is perturbed, and the word fine-tuning model in the target domain vector model is trained for the first time based on the perturbed first data, wherein the first data is data within the target domain; The word fine-tuning model after the first training is trained a second time based on the first and second categories of the second data, wherein the second category is the category of the data obtained by inputting the second data into the word fine-tuning model after the first training, and the second data is data within the target domain; positive sample pairs and negative sample pairs are constructed based on the third and fourth data, and the sentence fine-tuning model in the target domain vector model is trained based on the positive sample pairs and the negative sample pairs, wherein the third and fourth data are both data within the target domain; Semantic recognition is performed on the data within the target domain based on the trained target domain vector model.

2. The semantic recognition method for data according to claim 1, characterized in that, Before performing data perturbation on the first character in the first data, the method further includes: Multiple fifth data points within the target domain are acquired, and the interference elements contained in each fifth data point are filtered out. The interference elements include at least one of the following: whitespace characters, garbled characters, and sequential words. Each filtered fifth data point is segmented using a word segmentation tool to obtain multiple character sequences; Multiple training data for the target domain are generated based on multiple character sequences corresponding to the multiple fifth data, and the multiple training data are divided into the first data, the second data, the third data and the fourth data.

3. The semantic recognition method for data according to claim 1, characterized in that, The first character in the first data is perturbed, and the word fine-tuning model in the target domain vector model is trained for the first time based on the perturbed first data, including: The first character is subjected to data perturbation processing based on the target marker to obtain the first data after data perturbation processing; The first data after the data perturbation processing is input into the word fine-tuning model so that the word fine-tuning model outputs the recovered first data corresponding to the first data after the data perturbation processing; The first loss value between the recovered first data and the first data is calculated using the first loss function, and the word fine-tuning model is trained for the first time based on the first loss value.

4. The semantic recognition method for data according to claim 1, characterized in that, Before performing a second training on the word fine-tuning model after the first training based on the first and second categories of the second data, the method further includes: The second data is input into the word fine-tuning model after the first training to obtain the first feature vector corresponding to the second data; The first feature vector is input into the classifier so that the classifier determines the second category of the second data based on the first feature vector.

5. The semantic recognition method for data according to claim 1, characterized in that, Based on the first and second categories of the second data, the word fine-tuning model after the first training is trained a second time, including: The first category and the second category are compared to calculate the classification loss value between the first category and the second category; The classification loss value is backpropagated to the word fine-tuning model after the first training, so as to adjust the first model parameters corresponding to the word fine-tuning model after the first training using the classification loss value.

6. The semantic recognition method for data according to claim 1, characterized in that, Construct positive and negative sample pairs based on the third and fourth data, including: The third data is input into the word fine-tuning model after the second training, so that the word fine-tuning model after the second training performs the first forward propagation of the third data through a random deactivation mechanism to obtain the second feature vector. The random deactivation mechanism is used to randomly deactivate one or more neurons in the word fine-tuning model after the second training. The third data is input again into the word fine-tuning model after the second training, so that the word fine-tuning model after the second training performs a second forward propagation on the third data through the random deactivation mechanism to obtain a third feature vector, and the second feature vector and the third feature vector are determined as the positive sample pair. The random deactivation mechanism performs random deactivation on different neurons in the first forward propagation and the second forward propagation. The fourth data is input into the word fine-tuning model after the second training, so that the word fine-tuning model after the second training performs forward propagation on the fourth data through the random deactivation mechanism to obtain the fourth feature vector, and the fourth feature vector and the second feature vector are determined as the negative sample pair.

7. The semantic recognition method for data according to claim 1, characterized in that, Training the sentence fine-tuning model in the target domain vector model based on the positive sample pairs and the negative sample pairs includes: Calculate the first similarity between the two samples in the positive sample pair, and calculate the second similarity between the two samples in the negative sample pair; The first similarity and the second similarity are input into the second loss function to determine the second loss value corresponding to the sentence fine-tuning model, and the second model parameters corresponding to the sentence fine-tuning vector are adjusted using the second loss value. The second loss function is: For the first similarity, For the second similarity, These are two samples from the positive sample pair. Let τ be the temperature constant, N be the number of negative sample pairs, and l be the two samples in the negative sample pair. i This is the second loss value.

8. A semantic recognition device for data, characterized in that, include: The first training module is used to perform data perturbation processing on the first character in the first data, and to perform the first training on the word fine-tuning model in the target domain vector model based on the first data after data perturbation processing, wherein the first data is data within the target domain; The second training module is used to perform a second training on the word fine-tuning model after the first training based on the first category and the second category of the second data, wherein the second category is the category of the data obtained by inputting the second data into the word fine-tuning model after the first training, and the second data is data within the target domain; The third training module is used to construct positive sample pairs and negative sample pairs based on the third data and the fourth data, and to train the sentence fine-tuning model in the target domain vector model based on the positive sample pairs and the negative sample pairs, wherein the third data and the fourth data are both data within the target domain; the semantic recognition module is used to perform semantic recognition on the data within the target domain based on the trained target domain vector model.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 7 through the computer program.

Images