Method and device for pre-training of language model

Abstract

A pre-training method for a language model is provided that can pre-train a language model used in an open domain environment. The pre-training method can generate pseudo-queries by extracting key sentences from one of multiple documents and pre-train a language model using them.

Description

Technology Field

[0001] The present invention relates to a method and apparatus for pre-training a language model capable of pre-training a language model used in an open domain environment. Background Technology

[0002] Document re-search is a task in which documents are ranked according to their relevance scores within document candidates extracted by a relatively lightweight initial ranking model. Pre-language learning models such as BERT are primarily used for this document re-search.

[0003] However, existing pre-trained language models have a disadvantage in that while they possess high general language comprehension capabilities, they lack the ability to function as search models that must match input queries with corresponding documents in an open domain environment.

[0004] To solve these problems, methods are being studied to improve the initial ranking model used to extract approximate document candidates or to overfit the training data by making the training steps or processes of the pre-trained language model more complex.

[0005] However, these methods had limitations in training pre-trained language models to be suitable for query-based search tasks in open domain environments. Prior art literature

[0006] Korean Registered Patent No. 10-2476104 (December 9, 2022) The problem to be solved

[0007] The present invention aims to provide a method and apparatus for pre-training a language model that can pre-train a language model used in an open domain environment. means of solving the problem

[0008] A method for pre-training a language model according to one embodiment of the present invention comprises: a step of generating a pseudo-query by extracting a core sentence from one of a plurality of documents; and a step of pre-training the language model using the pseudo-query.

[0009] The step of generating the pseudo-query comprises: a step of classifying the core sentence by morpheme to extract a plurality of morphemes; a step of removing one or more stop morphemes from the plurality of morphemes; and a step of generating the pseudo-query by combining one or more morphemes from the plurality of morphemes from which the stop morphemes have been removed.

[0010] Here, the step of generating the pseudo-query further includes, if the number of the one or more morphemes is greater than or equal to a preset threshold value, the step of generating the pseudo-query by randomly extracting a number of morphemes equal to the threshold value from among the one or more morphemes.

[0011] Additionally, the step of generating the pseudo-query further includes, if the number of the one or more morphemes is less than the reference value, the step of generating the pseudo-query using all of the one or more morphemes.

[0012] Here, the above one or more morphemes are arranged in the order of placement within the above core sentence to generate the above pseudo-query.

[0013] In addition, the above one or more morphemes include at least one of the noun and verb included in the above core sentence.

[0014] The step of pre-training the language model includes: providing the pseudo-query and the plurality of documents as input to the language model; and training the language model to predict and output an association score based on the association between the pseudo-query and the plurality of documents.

[0015] Additionally, the step of pre-training the language model comprises: a step of training the language model to classify a correct document and an incorrect document among the plurality of documents based on the pseudo-query, and to predict and output a first association score for the correct document and a second association score for the incorrect document, respectively; and a step of adjusting the parameters of the language model so that the difference between the first association score and the second association score increases based on the correct association score input as label data.

[0016] A pre-training device for a language model according to an embodiment of the present invention includes: a memory in which a pre-training program is stored; and a processor that executes the pre-training program to extract a key sentence from one of a plurality of documents stored in a database to generate a pseudo-query, and pre-trains a language model using the pseudo-query.

[0017] The processor classifies the core sentence by morpheme to extract multiple morphemes, removes one or more stop morphemes from the extracted multiple morphemes, and combines one or more morphemes from which the stop morphemes have been removed to generate the pseudo-query.

[0018] Here, if the number of one or more morphemes is greater than or equal to a preset threshold value, the processor generates the pseudo-query by randomly extracting a number of morphemes equal to the threshold value from among the one or more morphemes, and if the number of one or more morphemes is less than the threshold value, the processor generates the pseudo-query using all of the one or more morphemes.

[0019] At this time, the above one or more morphemes include at least one of the noun and verb included in the above core sentence.

[0020] Additionally, the processor provides the pseudo-query and the plurality of documents as input to the language model, and trains the language model to classify the correct document and the incorrect document among the plurality of documents based on the pseudo-query, and to predict and output a first association score for the correct document and a second association score for the incorrect document, respectively.

[0021] At this time, the processor provides the correct answer for the correlation as label data to the language model, thereby adjusting the parameters of the language model so that the difference between the first correlation score and the second correlation score increases. Effects of the invention

[0022] The pre-training method according to the present invention can increase the universality of a language model and improve document search performance for input queries by pre-training a language model used in an open domain environment using pseudo-queries generated from a plurality of documents.

[0023] In addition, the pre-training method according to the present invention generates pseudo-queries from each of a plurality of documents and pre-trains the language model, thereby eliminating the need for a configuration that labels the relevance of documents to input queries required during conventional language model training, which can increase the efficiency of pre-training the language model. Brief explanation of the drawing

[0024] FIG. 1 is a drawing showing a pre-learning device for a language model according to an embodiment of the present invention. Figure 2 is a diagram conceptually illustrating the functions of the pre-learning program of Figure 1. FIG. 3 is a flowchart of a pre-learning method for a language model according to an embodiment of the present invention. Figure 4 is a flowchart of a method for generating the pseudo-query of Figure 3. FIG. 5 is a diagram showing an example of a pre-learning method for a language model of the present invention. FIG. 6 is a diagram showing an example of a method for generating a pseudo-query of the present invention. Specific details for implementing the invention

[0025] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0026] In describing the embodiments of the present invention, specific descriptions of known functions or configurations will be omitted if it is determined that such detailed descriptions could unnecessarily obscure the essence of the invention. Furthermore, the terms described below are defined in consideration of their functions in the embodiments of the present invention, and these definitions may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification.

[0027] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

[0028] FIG. 1 is a drawing showing a pre-learning device for a language model according to an embodiment of the present invention.

[0029] The language model (200) illustrated in FIG. 1 may be a neural network model capable of searching for and extracting highly relevant information, such as documents, in response to a query input from an external source, such as a user, in an open domain environment.

[0030] Here, an open domain can refer to an environment composed of information organized by various topics, such as documents on diverse subjects. For instance, an open domain may be a domain designed to search for information on a given topic from a wide range of sources. Examples of such open domains include the search systems of Google or Naver.

[0031] Accordingly, the pre-learning device (100) of the present embodiment can increase the universality and accuracy of information retrieval by the language model (200) by enabling pre-learning of the language model (200) used in an open domain environment.

[0032] This pre-learning device (100) may include a processor (110), memory (120), and a database (140).

[0033] The processor (110) can receive one document from among a plurality of documents stored in the database (140), generate a pseudo query related to the document, and pre-train a language model (200) using the generated pseudo query.

[0034] The memory (120) can store information necessary for the execution of the pre-training program (130). The pre-training program (130) may be software that includes commands for generating a pseudo-query for pre-training the language model (200) from a document provided in the database (140) and pre-training the language model (200) through this.

[0035] Accordingly, the processor (110) executes a pre-training program (130) stored in memory (120), uses it to generate a pseudo-query from a document provided in the database (140), and can pre-train the language model (200) through this.

[0036] A plurality of documents for pre-training of the language model (200) may be stored in the database (140). These plurality of documents may be information containing knowledge about various topics, and one or more documents may be stored correspondingly for the same topic. The database (140) may provide one of the plurality of documents stored for the same topic upon request from the processor (110).

[0037] Figure 2 is a diagram conceptually illustrating the functions of the pre-learning program of Figure 1.

[0038] Referring to FIG. 2, the pre-learning program (130) of the present embodiment may include a sentence extraction unit (131), a morpheme classification unit (133), a pseudo-question generation unit (135), a pre-learning unit (137), and a learning verification unit (139).

[0039] The sentence extraction unit (131), morpheme classification unit (133), pseudo-question generation unit (135), pre-learning unit (137), and learning verification unit (139) illustrated in FIG. 2 are conceptually divided to easily explain the functions of the pre-learning program (130), but are not limited thereto.

[0040] For example, according to an embodiment of the present invention, the sentence extraction unit (131), morpheme classification unit (133), pseudo-question generation unit (135), pre-learning unit (137), and learning verification unit (139) of the pre-learning program (130) may have their functions merged or separated, and may be implemented as a series of commands included in a single program.

[0041] The sentence extraction unit (131) can extract one of the multiple documents stored in the database (140) and extract one key sentence among the multiple sentences included in the extracted document.

[0042] For example, the sentence extraction unit (131) can extract key sentences from the extracted document using an algorithm such as Text Rank.

[0043] The morpheme classification unit (133) can classify the core sentence extracted from the sentence extraction unit (131) by morpheme and extract multiple morphemes.

[0044] For example, the morpheme classification unit (133) can classify each of the multiple keywords by morpheme by tagging the core sentence as a part of speech. In addition, the morpheme classification unit (133) can extract multiple morphemes corresponding to the core sentence according to the classification result.

[0045] The pseudo-query generation unit (135) can generate a single pseudo-query corresponding to a core sentence based on a plurality of morphemes extracted from the morpheme classification unit (133). Here, the pseudo-query can be generated as a combination of specific morphemes among the plurality of morphemes.

[0046] For example, the generating part (135) of the pseudo-question can remove one or more stop morphemes among the plurality of morphemes. Here, the stop morphemes may be morphemes such as particles, conjunctions, adjectives, and adverbs.

[0047] Additionally, the pseudo-query generation unit (135) can generate a pseudo-query for a core sentence by combining the remaining morphemes, excluding stop word morphemes, among the plurality of morphemes. Here, the remaining morphemes may include morphemes for nouns and verbs among the plurality of morphemes.

[0048] At this time, the pseudo-query generation unit (135) can generate a pseudo-query by combining the remaining morphemes according to the number of remaining morphemes.

[0049] For example, if the number of remaining morphemes is greater than or equal to a preset threshold value, the pseudo-query generation unit (135) can extract a number of morphemes equal to the threshold value from among the remaining morphemes and generate a pseudo-query by combining the extracted morphemes. At this time, the pseudo-query generation unit (135) can generate a pseudo-query through random extraction of the remaining morphemes and random arrangement of the extracted morphemes.

[0050] Additionally, if the number of remaining morphemes is less than the reference value, the pseudo-query generation unit (135) can generate a pseudo-query by combining all remaining morphemes. At this time, the pseudo-query generation unit (135) can generate a pseudo-query by arranging the remaining morphemes in their original positions, that is, in the order in which they are placed within the core sentence.

[0051] The pre-training unit (137) can pre-train the language model (200) using the pseudo-query generated in the pseudo-query generation unit (135).

[0052] As previously explained, the language model (200) may be a neural network model that searches for and extracts one or more pieces of information that are highly relevant to the input query from among multiple pieces of information, such as multiple documents regarding various tasks.

[0053] Accordingly, the pre-learning unit (137) of the present embodiment can pre-learn the language model (200) by providing a plurality of documents containing documents related to said pseudo-question along with said pseudo-question as input to the language model (200).

[0054] The learning verification unit (139) can verify the level of learning of the pre-trained language model (200). For example, the learning verification unit (139) can provide multiple documents as input to the language model (200) along with actual queries, such as queries entered by a user, and verify the level of pre-learning of the language model (200) based on the output of the language model (200) accordingly.

[0055] In addition, the learning verification unit (139) can improve the accuracy of information retrieval during actual use of the language model (200) by fine-tuning the internal parameters of the language model (200) based on the verification results.

[0056] FIG. 3 is a flowchart of a pre-learning method for a language model according to an embodiment of the present invention.

[0057] Referring to FIG. 3, the processor (110) of the pre-learning device (100) executes a pre-learning program (130) stored in memory (120) and can use it to generate a pseudo-query for pre-learning of the language model (200).

[0058] For example, the sentence extraction unit (131) may receive a document on a specific topic from the database (140). Then, the sentence extraction unit (131) may extract a key sentence from the provided document (S10).

[0059] Next, the morpheme classification unit (133) extracts multiple morphemes from the extracted core sentence, and the pseudo-query generation unit (135) can combine the multiple morphemes to generate a pseudo-query corresponding to the extracted core sentence (S20).

[0060] FIG. 4 is a flowchart of a method for generating a pseudo-query of FIG. 3, and FIG. 6 is a diagram showing an example of a method for generating a pseudo-query of the present invention.

[0061] Referring to FIGS. 4 and FIGS. 6, first, the morpheme classification unit (133) can classify multiple keywords of the previously extracted core sentence by part-of-speech tagging and morpheme. Next, the morpheme classification unit (133) can extract multiple morphemes for the core sentence (S110).

[0062] Here, the extracted multiple morphemes may be morphemes such as nouns, verbs, particles, adverbs, and adjectives. At this time, as shown in FIG. 6, the morpheme classification unit (133) may define noun morphemes as 'NN', verb morphemes as 'VB', particle morphemes as 'IN', and adverb morphemes as 'JJ'.

[0063] Next, the pseudo-text generation unit (135) can extract and remove one or more stop morphemes from among the extracted multiple morphemes (S120).

[0064] As illustrated in FIG. 6, the pseudo-qualification generating unit (135) can remove stop morphemes such as particles, conjunctions, adverbs, and adjectives, excluding nouns and verbs from among the plurality of morphemes.

[0065] Next, the pseudo-question generation unit (135) can generate a pseudo-question corresponding to a core sentence by combining the remaining morphemes from which the stop word morphemes have been removed (S130).

[0066] Here, the pseudo-query generation unit (135) can generate pseudo-queries according to the number of remaining morphemes, that is, among the multiple morphemes, stop morphemes are removed.

[0067] For example, if the number of remaining morphemes is less than a preset threshold value, the pseudo-query generation unit (135) can generate a pseudo-query by combining all remaining morphemes as shown in FIG. 6.

[0068] At this time, the pseudo-query generation unit (135) can generate a pseudo-query by arranging the remaining morphemes according to their original positions, that is, their placement positions within the core sentence.

[0069] Additionally, if the number of remaining morphemes is greater than or equal to the reference value, the pseudo-query generation unit (135) can extract a number of morphemes equal to the reference value from the remaining morphemes and arrange the extracted morphemes to generate a pseudo-query.

[0070] For example, if the number of remaining morphemes is (N+1) and the reference value is N, the pseudo-query generation unit (135) can randomly extract N morphemes from (N+1) morphemes and randomly arrange the extracted N morphemes to generate a pseudo-query.

[0071] Referring again to FIG. 3, the pre-training unit (137) can pre-train the language model (200) based on the pseudo-query generated by the pseudo-query generation unit (135) (S30).

[0072] FIG. 5 is a diagram showing an example of a pre-learning method for a language model of the present invention.

[0073] Referring to FIG. 5, the pre-training unit (137) can pre-train the language model (200) by providing a plurality of documents, including a document related to the pseudo-query, along with a previously generated pseudo-query, as input to the language model (200). At this time, the language model (200) can be trained to predict and output the correlation score of each of the plurality of documents according to the pseudo-query. Here, the language model (200) can be trained to output the corresponding document along with the predicted correlation score.

[0074] Additionally, the pre-training unit (137) can provide the correct association score as label data to the language model (200) as input. Accordingly, the language model (200) can repeatedly perform learning to output the association scores of multiple documents for a pseudo-query while adjusting internal parameters so that the loss function determined by the error between the predicted association score and the correct association score is minimized.

[0075] Meanwhile, the language model (200) of the present embodiment can adjust parameters to increase the difference between the correlation scores for each of the multiple documents in order to increase the accuracy of the output based on prior learning, that is, the predicted correlation scores for multiple documents.

[0076] For example, the language model (200) can classify multiple documents into correct documents that are highly related to the pseudo-query and incorrect documents that are less related to the pseudo-query, and can predict and output a first relevance score for the correct documents and a second relevance score for the incorrect documents, respectively.

[0077] In addition, the pre-training performance of the language model (200) can be improved by adjusting internal parameters so that the difference between the first correlation score and the second correlation score increases based on the correct correlation input as label data.

[0078] In addition, according to an embodiment of the present invention, the language model (200) can perform pre-learning by simultaneously using at least two algorithms.

[0079] For example, the language model (200) can perform pre-learning from pseudo-queries and multiple documents provided from the pre-learning device (100) by simultaneously using a keyword coherence prediction (KCP) algorithm and a masked language modeling (MLM) algorithm.

[0080] Here, the keyword consistency prediction algorithm may be an algorithm that predicts the degree of association between a query and a document using tokens of the input data. Additionally, the mask language modeling algorithm may be an algorithm that predicts the original word using masked tokens of the input data.

[0081] Accordingly, the language model (200) can repeatedly perform pre-learning from pseudo-queries and multiple documents through multiple algorithms while adjusting internal parameters so that the sum of the loss functions generated by each algorithm is minimized.

[0082] Referring again to FIG. 3, the learning verification unit (139) can verify the pre-trained language model (200) (S40).

[0083] The learning verification unit (139) provides multiple documents as input to the language model (200) along with actual queries, such as queries entered by a user, and can verify the degree of prior learning of the language model (200) based on the output of the language model (200) accordingly, such as predicted correlation scores for each of the multiple documents and the corresponding documents.

[0084] In addition, the learning verification unit (139) can improve the accuracy of information retrieval during actual use of the language model (200) by fine-tuning the internal parameters of the language model (200) based on the verification results.

[0085] As described above, the pre-training method of the language model of the present embodiment can increase the universality of the language model and improve document search performance for input queries by pre-training the language model used in an open domain environment using pseudo-queries generated from multiple documents.

[0086] In addition, the pre-training method of the present embodiment generates pseudo-queries from each of a plurality of documents and pre-trains the language model, thereby eliminating the need for the configuration of labeling the relevance of documents to input queries required during conventional language model training, which can increase the efficiency of pre-training the language model.

[0087] The combinations of each block of the block diagram and each step of the flowchart of the present invention described above may be executed by computer program instructions. Since these computer program instructions may be loaded into an encoding processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, the instructions executed through the encoding processor of the computer or other programmable data processing equipment create means for performing the functions described in each block of the block diagram or each step of the flowchart. Since these computer program instructions may also be stored in computer-available or computer-readable memory that can be directed toward the computer or other programmable data processing equipment to implement the function in a specific way, the instructions stored in the computer-available or computer-readable memory may also produce a manufactured item containing instruction means for performing the function described in each block of the block diagram or each step of the flowchart. Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that execute a computer or other programmable data processing equipment by performing a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer may also provide steps for executing the functions described in each block of the block diagram and each step of the flowchart.

[0088] Additionally, each block or each step may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). Also, it should be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps described in succession may actually be performed substantially simultaneously, or the blocks or steps may sometimes be performed in reverse order according to the corresponding function.

[0089] The above description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications and variations within the scope of the essential quality of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by such embodiments. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols

[0090] 100: Pre-training device 110: Processor 120: Memory 130: Pre-learning program 131: Sentence extraction unit 133: Morphological classification unit 135: Generation section of pseudo-question 137: Pre-learning section 139: Training Validation Section 140: Database 200: Language Model

Claims

Claim 1 A pre-training method for a language model performed in a pre-training device for a language model, comprising: a step of generating a pseudo-query by extracting a core sentence from one of a plurality of documents; and a step of pre-training the language model using the pseudo-query; wherein the step of generating the pseudo-query comprises: a step of classifying the core sentence by morpheme to extract a plurality of morphemes; a step of removing one or more stop morphemes from the plurality of morphemes; a step of generating the pseudo-query by combining one or more morphemes from the plurality of morphemes from which the stop morphemes have been removed; a step of generating the pseudo-query by randomly extracting a number of morphemes equal to the threshold value from the one or more morphemes if the number of the one or more morphemes is greater than or equal to a preset threshold value; and a step of generating the pseudo-query by using all of the one or more morphemes if the number of the one or more morphemes is less than the threshold value. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 A method of pre-training a language model generated by the pseudo-query in which, in claim 1, one or more morphemes are arranged in the order of placement within the core sentence. Claim 6 In claim 1, the above one or more morphemes are a pre-learning method of a language model including at least one of a noun and a verb included in the above core sentence. Claim 7 A method for pre-training a language model according to claim 1, wherein the step of pre-training the language model comprises: a step of providing the language model with the pseudo-query and the plurality of documents as inputs; and a step of training the language model to predict and output an association score based on the association between the pseudo-query and the plurality of documents. Claim 8 A method for pre-training a language model according to claim 1, wherein the step of pre-training the language model comprises: a step of training the language model to classify a correct document and an incorrect document among a plurality of documents based on the pseudo-query, and to predict and output a first relevance score for the correct document and a second relevance score for the incorrect document, respectively; and a step of adjusting the parameters of the language model so that the difference between the first relevance score and the second relevance score increases based on the correct relevance input as label data. Claim 9 A pre-training device for a language model comprising: a memory storing a pre-training program; and a processor that executes the pre-training program to extract a core sentence from one of a plurality of documents stored in a database to generate a pseudo-query, and pre-trains a language model using the pseudo-query; wherein the processor classifies the core sentence by morpheme to extract a plurality of morphemes, removes one or more stop morphemes from the extracted plurality of morphemes, and combines the one or more morphemes from which the stop morphemes have been removed to generate the pseudo-query, and if the number of the one or more morphemes is greater than or equal to a preset threshold value, randomly extracts a number of morphemes equal to the threshold value from the one or more morphemes to generate the pseudo-query, and if the number of the one or more morphemes is less than the threshold value, generates the pseudo-query using all of the one or more morphemes. Claim 10 delete Claim 11 delete Claim 12 In claim 9, the above one or more morphemes are a pre-learning device of a language model comprising at least one of a noun and a verb included in the above core sentence. Claim 13 A pre-training device for a language model according to claim 9, wherein the processor provides the pseudo-query and the plurality of documents as input to the language model, and trains the language model to classify a correct document and an incorrect document among the plurality of documents based on the pseudo-query, and to predict and output a first association score for the correct document and a second association score for the incorrect document, respectively, by providing the correct association score as label data to the language model so that the parameters of the language model are adjusted so that the difference between the first association score and the second association score becomes larger. Claim 14 A computer-readable recording medium storing a computer program, wherein the computer program comprises: a step of generating a pseudo-query by extracting a core sentence from one of a plurality of documents; and a step of pre-training a language model using the pseudo-query; wherein the step of generating the pseudo-query comprises: a step of classifying the core sentence by morpheme to extract a plurality of morphemes; a step of removing one or more stop morphemes from the plurality of morphemes; a step of generating the pseudo-query by combining one or more morphemes from the plurality of morphemes from which the stop morphemes have been removed; a step of generating the pseudo-query by randomly extracting a number of morphemes equal to the reference value from the one or more morphemes if the number of the one or more morphemes is greater than or equal to a preset reference value; and a step of generating the pseudo-query by using all of the one or more morphemes if the number of the one or more morphemes is less than the reference value. A computer-readable recording medium comprising instructions for a processor to perform a method of pre-training a language model. Claim 15 A computer program stored on a computer-readable recording medium, wherein the computer program comprises: a step of generating a pseudo-query by extracting a core sentence from one of a plurality of documents; and a step of pre-training a language model using the pseudo-query; wherein the step of generating the pseudo-query includes: a step of classifying the core sentence by morpheme to extract a plurality of morphemes; a step of removing one or more stop morphemes from the plurality of morphemes; a step of generating the pseudo-query by combining one or more morphemes from the plurality of morphemes from which the stop morphemes have been removed; a step of generating the pseudo-query by randomly extracting a number of morphemes equal to the reference value from the one or more morphemes if the number of the one or more morphemes is greater than or equal to a preset reference value; and a step of generating the pseudo-query by using all of the one or more morphemes if the number of the one or more morphemes is less than the reference value.

Images