Methods, devices, and media for training semantic recognition models based on contrastive learning

Abstract

The present application relates to the field of deep learning technology, and proposes a semantic recognition model training method, device and medium based on contrastive learning, the method comprising: performing synonym replacement on each of the first target words in the original sentence text to obtain a first processed sentence text; and performing antonym replacement on each of the second target words in the original sentence text to obtain a second processed sentence text; training the initial semantic recognition model through the original sentence text, the first processed sentence text and the second processed sentence text to obtain a target semantic recognition model. The purpose of the embodiment of the present invention is to provide a contrastive learning method that can identify small changes in semantics, and to modify the problem that positive and negative samples are easy to distinguish before, so that the model can learn richer knowledge and improve the model training effect.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a semantic recognition model training method, system, electronic device and computer-readable storage medium based on contrastive learning. Background Art

[0002] Contrastive Learning is a commonly used self-supervised learning method. The core idea is to reduce the distance to positive samples and expand the distance to negative samples. By training the model to find out which data points are similar or different, the general characteristics of unlabeled data sets can be learned.

[0003] However, when contrastive learning is applied to semantic recognition models to train semantic recognition models to distinguish text sentences with similar or different semantics, many difficulties are encountered. The main difficulties are: first, negative sample pairs of text contrastive learning are relatively easy to identify. The semantic differences of different text representations are large, and negative sample pairs can be distinguished only by the length of the text; second, false positives may occur in the process of constructing positive sample pairs through data enhancement, because changing one or more words in the text may change the semantics. Based on the above, the effect of training sentence-level semantic recognition models through contrastive learning is not good. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a semantic recognition model training method, system, electronic device and computer-readable storage medium based on contrastive learning, aiming to improve the effect of training sentence-level semantic recognition models through contrastive learning.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a semantic recognition model training method based on contrastive learning, the method comprising:

[0006] Get the original sentence text to be processed;

[0007] Determine N candidate words in the original sentence text according to the part of speech of each word in the original sentence text;

[0008] Selecting M words from the N candidate words as first target words and selecting T words as second target words, wherein N, M and T satisfy the following numerical relationship: 1<T<M<N;

[0009] In the original sentence text, each of the first target words is replaced with a synonym to obtain a first processed sentence text;

[0010] In the original sentence text, each of the second target words is replaced with an antonym to obtain a second processed sentence text;

[0011] The initial semantic recognition model is trained using the original sentence text, the first processed sentence text and the second processed sentence text to obtain a target semantic recognition model.

[0012] According to the semantic recognition model training method based on contrastive learning provided in some embodiments of the present application, the initial semantic recognition model is trained by the original sentence text, the first processed sentence text and the second processed sentence text to obtain a target semantic recognition model, including:

[0013] Inputting the original sentence text, the first processed sentence text and the second processed sentence text into an initial semantic recognition model respectively, obtaining an original semantic vector corresponding to the original sentence text, a first processed semantic vector corresponding to the first processed sentence text and a second processed semantic vector corresponding to the second processed sentence text;

[0014] Calculate a first loss value based on the original semantic vector and the first processed semantic vector;

[0015] Calculate a second loss value based on the original semantic vector and the second processed semantic vector;

[0016] Determining whether a preset training end condition is met based on the first loss value and the second loss value;

[0017] When the training end condition is not met, adjusting the model parameters of the semantic recognition model, and continuing to train the adjusted semantic recognition model through the original sentence text, the first processed sentence text and the second processed sentence text;

[0018] When the training end condition is met, the current semantic recognition model is used as the target semantic recognition model.

[0019] According to the semantic recognition model training method based on contrastive learning provided in some embodiments of the present application, the semantic recognition model includes a BERT encoding network and a semantic prediction network;

[0020] The step of inputting the original sentence text, the first processed sentence text, and the second processed sentence text into an initial semantic recognition model respectively to obtain an original semantic vector corresponding to the original sentence text, a first processed semantic vector corresponding to the first processed sentence text, and a second processed semantic vector corresponding to the second processed sentence text comprises:

[0021] Inputting the original sentence text, the first processed sentence text and the second processed sentence text into the BERT encoding network respectively, obtaining an original sentence encoding vector corresponding to the original sentence text, a first processed sentence encoding vector corresponding to the first processed sentence text and a second processed sentence encoding vector corresponding to the second processed sentence text; wherein the original sentence encoding vector, the first processed sentence encoding vector and the second processed sentence encoding vector respectively include a plurality of tokens;

[0022] The original sentence encoding vector, the first processed sentence encoding vector and the second processed sentence encoding vector are input into the semantic prediction network to obtain an original semantic vector corresponding to the original sentence text, a first processed semantic vector corresponding to the first processed sentence text and a second processed semantic vector corresponding to the second processed sentence text.

[0023] According to the semantic recognition model training method based on contrastive learning provided in some embodiments of the present application, the semantic recognition model also includes a token discrimination network;

[0024] The method further comprises:

[0025] Inputting the first processed sentence encoding vector and the second processed sentence encoding vector into a token discriminant network respectively, obtaining a first token label sequence corresponding to the first processed sentence encoding vector and a second token label sequence corresponding to the second processed sentence encoding vector, wherein the first token label sequence is used to indicate the token position in the first processed sentence encoding vector that is different from the original sentence encoding vector, and the second token label sequence is used to indicate the token position in the second processed sentence encoding vector that is different from the original sentence encoding vector;

[0026] When the original sentence encoding vector, the first processed sentence encoding vector and the second processed sentence encoding vector are input into the semantic prediction network, the first token label sequence and the second token label sequence are also input into the semantic prediction network, so that the semantic prediction network outputs the original sentence encoding vector, the first processed semantic vector and the second processed semantic vector based on the first token label sequence and the second token label sequence.

[0027] According to the semantic recognition model training method based on contrastive learning provided by some embodiments of the present application, determining whether a preset training end condition is met based on the first loss value and the second loss value includes:

[0028] comparing the first loss value to a minimized convergence value, and comparing the second loss value to a maximized convergence value;

[0029] When the first loss value is less than or equal to the minimum convergence value and the second loss value is greater than or equal to the maximum convergence value, it is determined that a preset training end condition is met.

[0030] According to the semantic recognition model training method based on contrastive learning provided by some embodiments of the present application, the determining of N candidate words in the original sentence text according to the part of speech of each word in the original sentence text includes:

[0031] Performing part-of-speech tagging on each word in the original sentence text by using a pre-trained part-of-speech tagging model to obtain the part-of-speech of each word in the original sentence text;

[0032] In the original sentence text, determining words with a preset part of speech as candidate words;

[0033] The preset part of speech includes at least one of an adjective and a verb.

[0034] According to the semantic recognition model training method based on contrastive learning provided by some embodiments of the present application, the step of selecting M words from the N candidate words as the first target words and selecting T words as the second target words includes:

[0035] Randomly selecting M words from N candidate words as first target words according to a preset first ratio; and

[0036] Randomly selecting T words from the N candidate words as second target words according to a preset second ratio;

[0037] The second ratio is smaller than the first ratio.

[0038] To achieve the above purpose, a second aspect of an embodiment of the present application proposes a semantic recognition model training system based on contrastive learning, the system comprising:

[0039] An acquisition module is used to acquire the original sentence text to be processed;

[0040] A determination module, configured to determine N candidate words in the original sentence text according to the part of speech of each word in the original sentence text;

[0041] A selection module is used to select M words from the N candidate words as first target words and select T words as second target words, wherein N, M and T satisfy the following numerical relationship: 1<T<M<N;

[0042] A first replacement processing module, used for performing synonym replacement on each of the first target words in the original sentence text to obtain a first processed sentence text;

[0043] A second replacement processing module is used to replace each of the second target words with an antonym in the original sentence text to obtain a second processed sentence text;

[0044] A training module is used to train an initial semantic recognition model using the original sentence text, the first processed sentence text and the second processed sentence text to obtain a target semantic recognition model.

[0045] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application provides an electronic device, including: at least one processor;

[0046] at least one processor; and,

[0047] a memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores a computer program, and when the computer program is executed by the at least one processor, the at least one processor executes the semantic recognition model training method based on contrastive learning as described in any embodiment of the first aspect above.

[0049] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the semantic recognition model training method based on contrastive learning as described in any embodiment of the first aspect above.

[0050] The embodiment of the present application proposes a semantic recognition model training method, system, electronic device and computer-readable storage medium based on contrastive learning. According to the part of speech of each word in the original sentence text, N candidate words are determined in the original sentence text, and then M words are selected from the N candidate words as the first target words and T words are selected as the second target words; each of the first target words in the original sentence text is replaced by a synonym to obtain a first processed sentence text; and each of the second target words in the original sentence text is replaced by an antonym to obtain a second processed sentence text. Here, the original sentence text and the first processed sentence text can form a positive text pair, and the original sentence text and the second processed sentence text can form a negative text pair. The purpose of the embodiment of the present invention is to provide a contrastive learning method that can identify slight changes in semantics. It is modified for the problem that positive and negative samples are better distinguished before. In this embodiment, the number of words replaced by synonyms is greater than the number of words replaced by antonyms. The purpose is to make negative samples smaller and more difficult to identify than positive samples, but the semantics of the sentence have changed in the opposite way. The semantic recognition model obtained by training the above positive and negative sample pairs can identify the semantic impact of slight changes in sentences. At the same time, it is more difficult to distinguish the positive and negative sample pairs produced by slight changes, so that the model can learn richer knowledge. The improved contrast method is much more effective than the traditional contrast learning method. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flowchart of a semantic recognition model training method based on contrastive learning provided in an embodiment of the present application;

[0052] Figure 2 yes Figure 1 Schematic diagram of sub-steps of step S120 in FIG.

[0053] Figure 3 is a schematic diagram of a classification network training process provided by an embodiment of the present application;

[0054] Figure 4 yes Figure 1 Schematic diagram of sub-steps of step S160 in FIG.

[0055] Figure 5A , Figure 5B yes Figure 4 Schematic diagram of sub-steps of step S161 in FIG.

[0056] Figure 6 is a flowchart of a specific example of the present application;

[0057] Figure 7It is a structural diagram of a semantic recognition model training system based on contrastive learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] First, some nouns involved in this application are analyzed:

[0061] Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0062] BERT (Bidirectional Encoder Representation from Transformers) is a pre-trained language representation model. It emphasizes that it no longer uses the traditional unidirectional language model or the shallow concatenation of two unidirectional language models for pre-training as in the past, but uses a new Masked Language Model (MLM) to generate deep bidirectional language representation.

[0063] Contrastive Learning: It is a commonly used self-supervised learning method that focuses on learning the common features between similar instances and distinguishing the differences between non-similar instances. Compared with generative learning, contrastive learning does not need to pay attention to the tedious details of the instance, but only needs to learn to distinguish the data in the feature space at the abstract semantic level. Therefore, the model and its optimization become simpler and have stronger generalization ability. The goal of contrastive learning is to learn an encoder that encodes similar data in a similar way and makes the encoding results of different types of data as different as possible. In other words, its core idea is to reduce the distance with positive samples and expand the distance with negative samples, so that the distance between positive samples and anchor points is much smaller than the distance between negative samples and anchor points (or the similarity between positive samples and anchor points is much greater than the similarity between negative samples and anchor points), so as to achieve the real distance of their original spatial distribution.

[0064] Embedding: Embedding is a vector representation, which means using a low-dimensional vector to represent an object. The object can be a word, a product, a movie, etc. The nature of this Embedding vector is that it can make objects corresponding to vectors with similar distances have similar meanings. For example, the distance between Embedding (Avengers) and Embedding (Iron Man) will be very close, but the distance between Embedding (Avengers) and Embedding (Gone with the Wind) will be farther. Embedding is essentially a mapping from semantic space to vector space, while trying to maintain the relationship of the original samples in the semantic space in the vector space. For example, the positions of two words with similar semantics in the vector space are also relatively close. Embedding can encode objects with low-dimensional vectors and retain their meaning. It is often used in machine learning. In the process of building machine learning models, objects are encoded into a low-dimensional dense vector and then passed to DNN to improve efficiency.

[0065] In the related art, when contrastive learning is applied to semantic recognition models to train semantic recognition models to distinguish text sentences with similar or different semantics, many difficulties are encountered. The main difficulties are: first, negative sample pairs of text contrastive learning are relatively easy to identify. The semantic differences represented by different texts are large in themselves, and negative sample pairs can be distinguished only by the length of the text; second, false positives may occur in the process of constructing positive sample pairs through data enhancement, because changing one or more words in the text may change the semantics. Based on the above, the effect of training sentence-level semantic recognition models through contrastive learning is not good.

[0066] The main purpose of the embodiments of the present application is to propose a semantic recognition model training method, device, electronic device and computer-readable storage medium based on contrastive learning, aiming to improve the effect of training sentence-level semantic recognition models through contrastive learning.

[0067] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0068] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0069] The semantic recognition model training method based on contrastive learning provided in the embodiment of the present application relates to the field of artificial intelligence and deep learning technology. The semantic recognition model training method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms; the software can be an application that implements the semantic recognition model training method, etc., but is not limited to the above forms.

[0070] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] See also Figure 1 , Figure 1 FIG. 2 is a flow chart showing a semantic recognition model training method based on contrastive learning proposed in an embodiment of the present application. Figure 1 As shown, the method provided in the embodiment of the present application includes but is not limited to the following steps:

[0072] Step S110, obtaining the original sentence text to be processed.

[0073] It can be understood that the semantic recognition model of the embodiment of the present application is used to perform semantic recognition on sentence-level text. In specific applications, the sentence text can be input into the semantic recognition model so that the semantic recognition model outputs a feature vector representing the semantics of the sentence text.

[0074] In specific implementation, the original sentence text may come from web page text data on the Internet, or from forums, forums, shopping websites, portal websites, etc. The web page text data is not limited to the content data published on the website, but may also include user comment data on the published content, etc. Of course, it should be understood that the web page text data is not limited to the data mentioned above; the batch text corpus of the embodiment of the present application is not limited to web page text data from the Internet, and may also come from, for example, electronic word dictionaries, paper journal databases, etc.

[0075] Step S120: determining N candidate words in the original sentence text according to the part of speech of each word in the original sentence text.

[0076] It is understandable that the embodiment of the present application selects a portion of words from the original sentence text for use as synonym replacement / antonym replacement in the subsequent steps, and the goal is to replace this portion of words so that the sentence semantics remains basically unchanged / the sentence semantics are opposite. The embodiment of the present application selects replaceable candidate words from the original sentence text according to part of speech.

[0077] See also Figure 2 Step S120 can be implemented by the following steps:

[0078] Step S121, performing part-of-speech tagging on each word in the original sentence text by using a pre-trained part-of-speech tagging model to obtain the part-of-speech of each word in the original sentence text;

[0079] Step S122: In the original sentence text, determine words with a preset part of speech as candidate words, wherein the preset part of speech includes at least one of an adjective and a verb.

[0080] Exemplarily, the part-of-speech tagging model may specifically include a word segmentation network and a classification network, wherein the word segmentation network is used to perform word segmentation processing on the original sentence text to obtain multiple words; the classification network is used to perform part-of-speech classification processing on each word to obtain the part-of-speech corresponding to each word.

[0081] In specific implementation, the original sentence text can be input into the word segmentation network so that the word segmentation network outputs a word segmentation sequence, and the word segmentation sequence contains multiple words, which can be words, phrases or short phrases, but are not limited thereto.

[0082] The classification network can classify the parts of speech of each word in the word segmentation sequence based on the word segmentation sequence output by the word segmentation network, and output a part-of-speech tagging sequence, which includes multiple part-of-speech tags, and the multiple part-of-speech tags here are in a one-to-one correspondence with the multiple words in the word segmentation sequence.

[0083] For example, the original sentence text is "Batman is an fictional super hero written by". The original sentence text is input into the word segmentation network to obtain the word segmentation sequence "Batman-is-an fictional-super-hero-written-by"; then the word segmentation sequence is input into the part-of-speech tagging sequence to obtain the part-of-speech tagging sequence "noun-verb-adjective-adjective-adjective-noun-verb-adverb". Based on the part-of-speech tagging sequence, the part of speech of each word in the original sentence text can be known.

[0084] For example, the word segmentation network in the part-of-speech tagging model can adopt an existing or future system that can implement the word segmentation function. Figure 3 , the classification network can be trained through the following steps:

[0085] Step S201, obtaining a corpus sample set, the corpus sample set including a plurality of sentence corpus samples;

[0086] Step S202, using a word segmentation system to segment the sentence corpus sample, and arranging the obtained multiple word segments into a word segmentation sequence according to word order and then inputting it into the initial classification network;

[0087] Step S203, adjusting the parameters of the classification network based on the result output by the classification network until the classification network converges.

[0088] It is understandable that some words have different parts of speech in different sentences, for example, the part of speech of "book" in the sentence "I have a book" is a noun, and the part of speech in "I want to book a room" is a verb. Therefore, the embodiment of the present application converts the entire sentence into a word segmentation sequence, and then trains the classification network using the word segmentation sequence, so that the classification network can learn the part of speech of each word based on the entire sentence, thereby making the part of speech tagging result output by the trained classification network more accurate.

[0089] It is understandable that the embodiment of the present application selects a part of vocabulary from the original sentence text, and is used as a synonym replacement / antonym replacement in the following steps, and the goal is to replace this part of vocabulary so that the sentence semantics remains basically unchanged / the sentence semantics is opposite. Therefore, the vocabulary that has a greater impact on the sentence semantics can be selected as a replaceable candidate vocabulary according to the part of speech. Specifically, the vocabulary that has a greater impact on the sentence semantics can generally be an adjective or a verb, so the adjective and / or verb in the original sentence text can be used as a candidate vocabulary.

[0090] Step S130, selecting M words from the N candidate words as first target words and selecting T words as second target words, wherein N, M and T satisfy the following numerical relationship: 1<T<M<N.

[0091] It is understandable that in the embodiment of the present application, the first target vocabulary is used for synonym replacement to obtain positive samples; the second target vocabulary is used for antonym replacement to obtain negative samples. In the embodiment of the present application, the expectation for the positive sample is that the positive sample has a large change compared to the original sentence text, but the semantics of the two should be as close as possible; the expectation for the negative sample is that the negative sample has only a slight difference from the original sentence text, but the semantics of the two should be as opposite as possible.

[0092] To achieve the above purpose, step S130 can be implemented by the following steps:

[0093] M words are randomly selected from N candidate words as first target words according to a preset first ratio, and T words are randomly selected from N candidate words as second target words according to a preset second ratio, wherein the second ratio is smaller than the first ratio.

[0094] Here, based on the number N of candidate words, the value of M is determined according to a preset first ratio, and the value of T is determined according to a preset second ratio. Here, the second ratio is smaller than the first ratio, so the value of T is also smaller than the value of M.

[0095] Step S140: In the original sentence text, each of the first target words is replaced with a synonym to obtain a first processed sentence text.

[0096] For example, after the original sentence text “Batman is an fictional super hero written by” is processed by synonym replacement, the first processed sentence text “Batman is an imaginary super hero created by” can be obtained.

[0097] Step S150: In the original sentence text, antonyms are replaced for each of the second target words to obtain a second processed sentence text.

[0098] For example, after the original sentence text "Batman is an fictional super hero written by" is processed by antonym replacement, the second processed sentence text "Batman is an real-life super hero written by" can be obtained.

[0099] Step S160: training an initial semantic recognition model using the original sentence text, the first processed sentence text, and the second processed sentence text to obtain a target semantic recognition model.

[0100] It is understandable that the original sentence text and the first processed sentence text can form a positive text pair, and the original sentence text and the second processed sentence text can form a negative text pair. The purpose of the embodiment of the present invention is to provide a contrastive learning method that can identify slight changes in semantics. It has been modified to address the problem that positive and negative samples are easy to distinguish. In this embodiment, the number of words replaced by synonyms is greater than the number of words replaced by antonyms. The purpose is to make negative samples smaller and more difficult to identify than positive samples, but the semantics of the sentences have changed in the opposite way. The semantic recognition model obtained by training the above positive and negative sample pairs can identify the semantic impact of slight changes in sentences. At the same time, it is more difficult to distinguish between positive and negative sample pairs caused by slight changes, so that the model can learn richer knowledge. The improved contrast method is much more effective than the traditional contrastive learning method.

[0101] See also Figure 4 Step S160 can be implemented by the following steps:

[0102] Step S161, input the original sentence text, the first processed sentence text and the second processed sentence text into the initial semantic recognition model respectively, to obtain the original semantic vector corresponding to the original sentence text, the first processed semantic vector corresponding to the first processed sentence text and the second processed semantic vector corresponding to the second processed sentence text.

[0103] Step S162, calculating a first loss value based on the original semantic vector and the first processed semantic vector;

[0104] Step S163, calculating a second loss value based on the original semantic vector and the second processed semantic vector;

[0105] Step S164, determining whether a preset training end condition is met based on the first loss value and the second loss value;

[0106] Step S165, when the training end condition is not met, adjusting the model parameters of the semantic recognition model, and continuing to train the adjusted semantic recognition model through the original sentence text, the first processed sentence text and the second processed sentence text;

[0107] Step S166, when the training end condition is met, the current semantic recognition model is used as the target semantic recognition model.

[0108] It is understandable that the training goal of the embodiment of the present application is to make the vector distance of the positive sample pair as small as possible and the vector distance of the negative sample pair as large as possible. Based on this, the specific implementation process of step S164 is as follows:

[0109] comparing the first loss value to a minimized convergence value, and comparing the second loss value to a maximized convergence value;

[0110] When the first loss value is less than or equal to the minimum convergence value and the second loss value is greater than or equal to the maximum convergence value, it is determined that a preset training end condition is met.

[0111] As an example, the calculation process of the first loss value is as follows: the Euclidean distance between the original semantic vector and the first processed semantic vector is calculated according to the Euclidean distance calculation formula, and the Euclidean distance between the original semantic vector and the first processed semantic vector is used as the first loss value.

[0112] As an example, the calculation process of the second loss value is as follows: the Euclidean distance between the original semantic vector and the second processed semantic vector is calculated according to the Euclidean distance calculation formula, and the Euclidean distance between the original semantic vector and the second processed semantic vector is used as the second loss value.

[0113] The training goal of the embodiment of the present application is to minimize the first loss value and satisfy the minimum convergence value; to maximize the second loss value and satisfy the maximum convergence value; when the first loss value is less than or equal to the minimum convergence value and the second loss value is greater than or equal to the maximum convergence value, the model training is terminated to obtain the target semantic recognition model.

[0114] As an example, the semantic recognition model in step S161 may include a BERT encoding network and a semantic prediction network. Figure 5A Step S161 can be specifically implemented by the following steps:

[0115] Step S1611, inputting the original sentence text, the first processed sentence text and the second processed sentence text into the BERT encoding network respectively, obtaining an original sentence encoding vector corresponding to the original sentence text, a first processed sentence encoding vector corresponding to the first processed sentence text and a second processed sentence encoding vector corresponding to the second processed sentence text; wherein the original sentence encoding vector, the first processed sentence encoding vector and the second processed sentence encoding vector can be understood as embeddings respectively including a plurality of tokens;

[0116] Step S1612, input the original sentence encoding vector, the first processed sentence encoding vector and the second processed sentence encoding vector into the semantic prediction network to obtain an original semantic vector corresponding to the original sentence text, a first processed semantic vector corresponding to the first processed sentence text and a second processed semantic vector corresponding to the second processed sentence text.

[0117] It can be understood that the embodiment of the present application outputs the original sentence encoding vector, the first processed sentence encoding vector and the second processed sentence encoding vector through the BERT encoding network, and then learns the difference between the original sentence encoding vector, the first processed sentence encoding vector and the second processed sentence encoding vector through the semantic prediction network, and outputs the corresponding original semantic vector, the first processed semantic vector and the second processed semantic vector.

[0118] Optionally, the semantic recognition model also includes a token discrimination network. Figure 5B When the semantic recognition model also includes a token discrimination network, after step S1611, the following steps are further included:

[0119] Step S1613, inputting the first processed sentence encoding vector and the second processed sentence encoding vector into a token discriminant network respectively, obtaining a first token label sequence corresponding to the first processed sentence encoding vector and a second token label sequence corresponding to the second processed sentence encoding vector, wherein the first token label sequence is used to indicate the token position in the first processed sentence encoding vector that is different from the original sentence encoding vector, and the second token label sequence is used to indicate the token position in the second processed sentence encoding vector that is different from the original sentence encoding vector;

[0120] Step S1614, when the original sentence encoding vector, the first processed sentence encoding vector and the second processed sentence encoding vector are input into the semantic prediction network, the first token label sequence and the second token label sequence are also input into the semantic prediction network, so that the semantic prediction network outputs the original sentence encoding vector, the first processed semantic vector and the second processed semantic vector based on the first token label sequence and the second token label sequence.

[0121] In an embodiment of the present application, the original sentence encoding vector and the first processed sentence encoding vector are used as a positive sample pair, the original sentence encoding vector and the second processed sentence encoding vector are used as a negative sample pair, the first token label sequence is used as positive sample auxiliary knowledge, and the second token label sequence is used as negative sample auxiliary knowledge. With the assistance of the first token label sequence and the second token label sequence, the semantic prediction network outputs the original sentence encoding vector, the first processed semantic vector, and the second processed semantic vector.

[0122] It can be understood that in order to improve the effect of model contrast learning, the embodiment of the present invention also inputs the token label sequence indicating the token difference between two samples in the positive / negative sample pair as auxiliary knowledge into the BERT encoding network, so that the BERT encoding network can quickly identify the difference between the sample pairs and improve the training efficiency.

[0123] See also Figure 6 , the semantic recognition model training method based on contrastive learning provided by the present invention is explained below through specific examples.

[0124] Step S301, obtaining the original sentence text to be processed "Batman is a fictional super hero written by";

[0125] Step S302, performing part-of-speech tagging on each word in the original sentence text by using a pre-trained part-of-speech tagging model, and obtaining the part-of-speech of each word in the original sentence text as "noun-verb-adjective-adjective-adjective-noun-verb-adverb";

[0126] Step S303, in the original sentence text, determining words whose part of speech is adjective or verb as candidate words;

[0127] Step S304, randomly selecting a first target word from the obtained candidate words according to a preset first ratio, and randomly selecting a second target word from the obtained candidate words according to a preset second ratio, wherein the second ratio is smaller than the first ratio;

[0128] Step S305, in the original sentence text, synonym replacement is performed on each of the first target words to obtain a first processed sentence text. In this example, the original sentence text "Batman is an fictional superhero written by" is processed by synonym replacement to obtain the first processed sentence text "Batman is an imaginary super hero created by".

[0129] Step S306, in the original sentence text, antonym replacement is performed on each of the second target words to obtain a second processed sentence text. In this example, after the original sentence text "Batman is an fictional super hero written by" is processed by antonym replacement, the second processed sentence text "Batman is an real-life super hero written by" is obtained.

[0130] Step S307, inputting the original sentence text, the first processed sentence text and the second processed sentence text into the BERT encoding network respectively, obtaining an original sentence encoding vector corresponding to the original sentence text, a first processed sentence encoding vector corresponding to the first processed sentence text and a second processed sentence encoding vector corresponding to the second processed sentence text; wherein the original sentence encoding vector, the first processed sentence encoding vector and the second processed sentence encoding vector respectively include a plurality of tokens;

[0131] Step S308, inputting the first processed sentence encoding vector and the second processed sentence encoding vector into a token discriminant network respectively, obtaining a first token label sequence corresponding to the first processed sentence encoding vector and a second token label sequence corresponding to the second processed sentence encoding vector, wherein the first token label sequence is used to indicate the token position in the first processed sentence encoding vector that is different from the original sentence encoding vector, and the second token label sequence is used to indicate the token position in the second processed sentence encoding vector that is different from the original sentence encoding vector;

[0132] Step S309, input the above-mentioned original sentence encoding vector, the first processed sentence encoding vector, the second processed sentence encoding vector, the positive sample auxiliary knowledge and the negative sample auxiliary knowledge into the semantic prediction network, so that the semantic prediction network outputs the corresponding original semantic vector, the first processed semantic vector and the second processed semantic vector with the assistance of the positive sample auxiliary knowledge and the negative sample auxiliary knowledge.

[0133] Step S310, calculate the first loss value based on the original semantic vector and the first processed semantic vector, calculate the second loss value based on the original semantic vector and the second processed semantic vector, and adjust the model parameters based on the first loss value and the second loss value until the preset model training end condition is met, wherein the model training end condition is: the first loss value is less than or equal to the minimum convergence value, and the second loss value is greater than or equal to the maximum convergence value, and the training goal is to minimize the distance between the original semantic vector and the first processed semantic vector, and to maximize the distance between the original semantic vector and the second processed semantic vector.

[0134] See also Figure 7 , the embodiment of the present application proposes a semantic recognition model training system based on contrastive learning, the system comprising:

[0135] An acquisition module is used to acquire the original sentence text to be processed;

[0136] A determination module, configured to determine N candidate words in the original sentence text according to the part of speech of each word in the original sentence text;

[0137] A selection module is used to select M words from the N candidate words as first target words and select T words as second target words, wherein N, M and T satisfy the following numerical relationship: 1<T<M<N;

[0138] A first replacement processing module, used for performing synonym replacement on each of the first target words in the original sentence text to obtain a first processed sentence text;

[0139] A second replacement processing module is used to replace each of the second target words with an antonym in the original sentence text to obtain a second processed sentence text;

[0140] A training module is used to train an initial semantic recognition model using the original sentence text, the first processed sentence text and the second processed sentence text to obtain a target semantic recognition model.

[0141] The present application also provides an electronic device, including:

[0142] at least one processor; and,

[0143] a memory communicatively connected to the at least one processor; wherein,

[0144] The memory stores a computer program, and when the computer program is executed by the at least one processor, the at least one processor executes the semantic recognition model training method based on contrastive learning as described in any of the above embodiments.

[0145] An embodiment of the present application further proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the semantic recognition model training method based on contrastive learning as described in any of the above embodiments.

[0146] The above embodiments may be used in combination, and modules with the same name in different embodiments may be the same or different.

[0147] The above describes specific embodiments of the present application, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0148] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system, device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0149] The system, device, computer-readable storage medium and method provided in the embodiments of the present application correspond to each other. Therefore, the system, device and computer-readable storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding system, device and computer-readable storage medium will not be repeated here.

[0150] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0151] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0152] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0153] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0154] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0155] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0156] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0158] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0159] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0160] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0161] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0162] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0163] Embodiments of the present application may be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Embodiments of the present application may also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0164] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0165] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A semantic recognition model training method based on contrastive learning, characterized in that: The semantic recognition model includes a BERT encoding network, a semantic prediction network and a token discrimination network, and the method includes: Get the original sentence text to be processed; Determine N candidate words in the original sentence text according to the part of speech of each word in the original sentence text; Randomly select M words from N candidate words as first target words according to a preset first ratio; and randomly select T words from N candidate words as second target words according to a preset second ratio, wherein the second ratio is less than the first ratio, and N, M and T satisfy the following numerical relationship: 1<T<M<N; In the original sentence text, each of the first target words is replaced with a synonym to obtain a first processed sentence text; In the original sentence text, each of the second target words is replaced with an antonym to obtain a second processed sentence text; Inputting the original sentence text, the first processed sentence text and the second processed sentence text into the BERT encoding network respectively, obtaining an original sentence encoding vector corresponding to the original sentence text, a first processed sentence encoding vector corresponding to the first processed sentence text and a second processed sentence encoding vector corresponding to the second processed sentence text; wherein the original sentence encoding vector, the first processed sentence encoding vector and the second processed sentence encoding vector respectively include a plurality of tokens; Inputting the first processed sentence encoding vector and the second processed sentence encoding vector into a token discriminant network respectively, obtaining a first token label sequence corresponding to the first processed sentence encoding vector and a second token label sequence corresponding to the second processed sentence encoding vector, wherein the first token label sequence is used to indicate the token position in the first processed sentence encoding vector that is different from the original sentence encoding vector, and the second token label sequence is used to indicate the token position in the second processed sentence encoding vector that is different from the original sentence encoding vector; Inputting the original sentence encoding vector, the first processed sentence encoding vector, the second processed sentence encoding vector, the first token label sequence, and the second token label sequence into the semantic prediction network to obtain an original semantic vector corresponding to the original sentence text, a first processed semantic vector corresponding to the first processed sentence text, and a second processed semantic vector corresponding to the second processed sentence text; Calculate a first loss value based on the original semantic vector and the first processed semantic vector; Calculate a second loss value based on the original semantic vector and the second processed semantic vector; Determining whether a preset training end condition is met based on the first loss value and the second loss value; When the training end condition is not met, adjusting the model parameters of the semantic recognition model, and continuing to train the adjusted semantic recognition model through the original sentence text, the first processed sentence text and the second processed sentence text; When the training end condition is met, the current semantic recognition model is used as the target semantic recognition model.

2. The method according to claim 1, characterized in that The determining whether a preset training end condition is met based on the first loss value and the second loss value includes: comparing the first loss value to a minimized convergence value, and comparing the second loss value to a maximized convergence value; When the first loss value is less than or equal to the minimum convergence value and the second loss value is greater than or equal to the maximum convergence value, it is determined that a preset training end condition is met.

3. The method according to claim 1, characterized in that The step of determining N candidate words in the original sentence text according to the part of speech of each word in the original sentence text comprises: Performing part-of-speech tagging on each word in the original sentence text by using a pre-trained part-of-speech tagging model to obtain the part-of-speech of each word in the original sentence text; In the original sentence text, determining words with a preset part of speech as candidate words; The preset part of speech includes at least one of an adjective and a verb.

4. A semantic recognition model training system based on contrastive learning, characterized in that: The semantic recognition model includes a BERT encoding network, a semantic prediction network and a token discrimination network, and the system includes: An acquisition module is used to acquire the original sentence text to be processed; A determination module, configured to determine N candidate words in the original sentence text according to the part of speech of each word in the original sentence text; A selection module is used to randomly select M words from N candidate words as first target words according to a preset first ratio; and randomly select T words from N candidate words as second target words according to a preset second ratio, wherein the second ratio is less than the first ratio, and N, M and T satisfy the following numerical relationship: 1<T<M<N; A first replacement processing module, used for performing synonym replacement on each of the first target words in the original sentence text to obtain a first processed sentence text; A second replacement processing module is used to replace each of the second target words with an antonym in the original sentence text to obtain a second processed sentence text; A training module, used to input the original sentence text, the first processed sentence text and the second processed sentence text into the BERT encoding network respectively, to obtain an original sentence encoding vector corresponding to the original sentence text, a first processed sentence encoding vector corresponding to the first processed sentence text and a second processed sentence encoding vector corresponding to the second processed sentence text; wherein the original sentence encoding vector, the first processed sentence encoding vector and the second processed sentence encoding vector respectively include multiple tokens; input the first processed sentence encoding vector and the second processed sentence encoding vector into the token discriminant network respectively, to obtain a first token label sequence corresponding to the first processed sentence encoding vector and a second token label sequence corresponding to the second processed sentence encoding vector, wherein the first token label sequence is used to indicate the token position in the first processed sentence encoding vector that is different from the original sentence encoding vector, and the second token label sequence is used to indicate the token position in the second processed sentence encoding vector that is different from the original sentence encoding vector. The vectors are used to identify the positions of tokens where there are differences in the vectors; the original sentence encoding vector, the first processed sentence encoding vector, the second processed sentence encoding vector, the first token label sequence and the second token label sequence are input into the semantic prediction network to obtain the original semantic vector corresponding to the original sentence text, the first processed semantic vector corresponding to the first processed sentence text and the second processed semantic vector corresponding to the second processed sentence text; a first loss value is calculated based on the original semantic vector and the first processed semantic vector; a second loss value is calculated based on the original semantic vector and the second processed semantic vector; based on the first loss value and the second loss value, it is determined whether the preset training end condition is met; when the training end condition is not met, the model parameters of the semantic recognition model are adjusted, and the adjusted semantic recognition model is continued to be trained through the original sentence text, the first processed sentence text and the second processed sentence text; when the training end condition is met, the current semantic recognition model is used as the target semantic recognition model.

5. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program, and when the computer program is executed by the at least one processor, the at least one processor executes the semantic recognition model training method based on contrastive learning as described in any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the semantic recognition model training method based on contrastive learning described in any one of claims 1 to 3 is implemented.

Images