Terminal and assessment method

Abstract

A terminal (10) comprises: a determination unit (11) that, if label assessments are to be made for each sentence in a text comprising a plurality of sentences, determines a first threshold value for assessing that no label is to be assigned and a second threshold value for assessing that a label is to be assigned; a first assessment unit (12) that, on the basis of a candidate label score representing the degree to which each of a plurality of candidate labels can be derived from a single sentence subjected to labeling in the text, the first threshold value, and the second threshold value, makes a label assessment regarding the sentence subjected to labeling; and a second assessment unit (13) that, if assessment by the first assessment unit (12) is not possible, makes another label assessment of the sentence subjected to labeling, additionally using information on a sentence other than the sentence subjected to labeling in the text.

Description

Device and determination method

[0001] The present disclosure relates to a terminal and a determination method. Note that the term "text" in this specification refers to a text in general that includes one or more sentences, and the content of the text is not limited to a specific genre or category.

[0002] By using natural language inference (hereinafter referred to as "NLI") to determine the associated label (see, for example, Patent Document 1 below) for each sentence contained in a text, a score indicating the extent to which the content of the label is incorporated into the text can be automatically determined. Note that in this disclosure, "label" means a word or phrase that succinctly expresses the content of a sentence, and is synonymous with points, topics, etc. Furthermore, the above-mentioned NLI is one of the natural language processing tasks, and is a task in which learning is performed using two texts consisting of a premise and a hypothesis, and predicts whether the input premise and hypothesis are correct (the hypothesis can be derived from the premise), contradictory (the hypothesis cannot be derived from the premise), or neutral (it cannot be said either way).

[0003] Japanese Patent Application Laid-Open No. 2008-276571

[0004] However, there are relationships between multiple sentences contained in actual text, and if an attempt is made to judge an English sentence containing pronouns such as "it" or "that" on its own, the pronoun's meaning may become unclear, leading to a risk of misjudgment and a decrease in the reliability of the score that results.

[0005] Furthermore, there is a good chance that the text contains sentences that are not related to the labels. Therefore, if the existence of such unrelated sentences is not taken into consideration, there is a possibility of misjudgment, which may reduce the reliability of the score that is the judgment result.

[0006] As a result, even when a text contains sentences with no relevant labels, it is highly anticipated that a technology will be able to accurately determine the associated label for each sentence in the text, taking into account the relevance between the sentences in the text, while preventing erroneous determinations and a decrease in the reliability of the score due to the unrelated sentences. Such a technology is highly anticipated for use in grading English tests, for example, in which the test asks the user to "create an English text that incorporates specified labels (points)."

[0007] The present disclosure aims to accurately determine the associated label for each sentence of a text, taking into account the relevance between sentences contained in the text, while preventing erroneous determinations and a decrease in the reliability of the score due to unrelated sentences.

[0008] A terminal according to the present disclosure includes: a determination unit that, when labeling each sentence in a text consisting of a plurality of sentences based on Natural Language Inference (NLI), determines from test data a first threshold value for determining not to assign a label and a second threshold value for determining to assign the label; a first determination unit that performs label determination for the target sentence based on the score for each candidate label obtained based on the NLI, the score representing the degree to which each of a plurality of candidate labels is derived from a single target sentence in the target text, the first threshold value, and the second threshold value; and a second determination unit that, when the first determination unit is unable to perform a determination for the target sentence, performs label determination for the target sentence again by further utilizing information of a sentence in the text that is different from the target sentence.

[0009] Note that "label determination" refers to the process of determining the label most strongly associated with a sentence in light of its content and associating that label with the sentence, and there may be situations where no label is associated with a sentence (a situation where determination is impossible). Also, "target sentence" refers to one sentence that is currently being subjected to label determination among multiple sentences that make up the target text.

[0010] According to the present disclosure, it is possible to accurately determine the associated label for each sentence of a text by taking into account the relevance between sentences contained in the text, while preventing erroneous determinations and a decrease in the reliability of the score due to unrelated sentences.

[0011] It is a functional block configuration diagram showing the configuration of a terminal. It is a diagram for explaining an example of an NLI technique. It is a diagram for explaining processing by a first determination unit. It is a diagram for explaining processing by a second determination unit. It is a flow diagram showing processing executed by a terminal. It is a flow diagram showing a modified example of the processing of Figure 5. It is a diagram showing an example of the hardware configuration of a terminal.

[0012] Hereinafter, an embodiment of a terminal and a determination method according to the present disclosure will be described with reference to the drawings.

[0013] Fig. 1 shows a configuration diagram of a terminal 10 according to the present disclosure. As shown in Fig. 1, in order to realize the functions according to the present disclosure, the terminal 10 includes a determination unit 11, a first determination unit 12, a second determination unit 13, and a scoring unit 14. The functions of each unit will be outlined below.

[0014] The determination unit 11 is a functional unit that determines, from test data, a first threshold value for determining not to assign a label and a second threshold value for determining to assign a label when labeling each sentence in a text consisting of multiple sentences based on the NLI. Note that a method for determining the threshold values ​​based on the NLI will be described later with reference to FIG. 2.

[0015] The first determination unit 12 is a functional unit that performs label determination for a target sentence based on a score for each candidate label, the score being obtained based on the NLI and representing the degree to which each of the candidate labels is derived from a single target sentence in the target text, a first threshold, and a second threshold. For example, as illustrated in FIG. 3 , if the first threshold is 0.3 and the second threshold is 0.7, the first determination unit 12 compares the score of the candidate label with the highest score (highest score) among the candidate labels for the target sentence with the first threshold of 0.3. If the highest score does not exceed the first threshold of 0.3 (i.e., if the scores of all the candidate labels do not exceed the first threshold of 0.3), the first determination unit 12 determines not to assign a label to the target sentence. On the other hand, the first determination unit 12 compares the score of the candidate label with the highest score among the candidate labels with the second threshold of 0.7. If the highest score exceeds the second threshold of 0.7, the first determination unit 12 determines to assign the candidate label with the highest score to the target sentence. However, if the highest score is equal to or greater than the first threshold of 0.3 and equal to or less than the second threshold of 0.7, the target sentence is deemed "undecidable."

[0016] The second determination unit 13 is a functional unit that, when the first determination unit 12 is unable to determine the target sentence, re-labels the target sentence by further utilizing information on sentences in the text that are different from the target sentence. The "sentence different from the target sentence" may be, for example, the sentence immediately preceding the target sentence in the text, the sentence immediately following the target sentence, or the first sentence in a paragraph that includes the target sentence. As an example, the second determination unit 13 may re-label only candidate labels among the multiple candidate labels whose scores exceed a first threshold, rather than all of the multiple candidate labels. Narrowing down the determination targets in this manner can reduce the processing load and improve processing efficiency.

[0017] Specifically, the second determination unit 13 compares the highest score among the scores for each of the multiple candidate labels, which indicates the degree to which each of the candidate labels is derived from the "combined portion of the target sentence and different sentences," with a second threshold of 0.7, and if the highest score exceeds the second threshold of 0.7 as shown in FIG. 4, it determines that the target sentence should be labeled with the candidate label with the highest score. On the other hand, in cases other than the above (i.e., if the highest score does not exceed the second threshold of 0.7), it determines that the target sentence should not be labeled. Note that there are other variations in the re-labeling by the second determination unit 13, which will be described later as modified examples.

[0018] The scoring unit 14 is a functional unit that scores each sentence in the text based on the label determination result and outputs the scoring result. The scoring method will be described in detail later.

[0019] The terminal 10 may be implemented using various information processing devices such as a smartphone, a mobile phone, a smart watch, a wearable device, a laptop computer, a desktop computer, or a server.

[0020] The process executed by the terminal 10 (the process related to the determination method of the present disclosure) will be described below with reference to the flow diagram of FIG.

[0021] First, the determination unit 11 of the terminal 10 determines a first threshold for determining not to assign a label and a second threshold for determining to assign a label based on the NLI from test data acquired from an external source. Here, for example, the first threshold is preset to 0.3 and the second threshold is preset to 0.7. An example of initially determining thresholds based on the NLI is as follows: (1) As shown in FIG. 2, a machine learning model based on the NLI is used to obtain scores using the labels of each piece of test data as hypotheses. For example, if the test data includes the four English sentences on the left side of the table in FIG. 2, the labels of each sentence—“travel,” “book,” “cat,” and “game”—are assumed to be hypotheses, and the scores listed in the table in FIG. 2 are obtained for each sentence. (2) Next, the F1-score for each score is calculated, and the score with the highest F1-score (0.94 in FIG. 2) is determined as the threshold. Note that the F1-score is an existing technology and is one of the evaluation indices for binary classification tasks (problems). It focuses on the trade-off relationship between precision and recall and is the harmonic mean of these two. In other words, the F1-score can be calculated using the following formula: F1-score = 2 x (Precision x Recall) / (Precision + Recall) Precision = TP / (TP + FP) Recall = TP / (TP + FN) TP: True positive FN: False negative FP: False positive TN: True negative The reference website is as follows: https: / / atmarkit.itmedia.co.jp / ait / articles / 2210 / 24 / news034.html The determination unit 11 may adjust the first and second thresholds based on one or a combination of the degree of relevance between multiple sentences included in the text, the length of the sentences, and the number of candidate labels to be determined. Such adjustments can improve the accuracy of label determination. The first and second thresholds determined or adjusted as described above are passed from the determination unit 11 to the first determination unit 12.

[0022] 5 , in the next step S2, the first determination unit 12 calculates the score of each of multiple candidate labels for one target sentence in the target text using an existing method based on the NLI. Then, the first determination unit 12 compares the score of the candidate label with the highest score among the multiple candidate labels (the highest score) with a first threshold of 0.3, and if the highest score does not exceed the first threshold of 0.3 (i.e., if the scores of all candidate labels do not exceed the first threshold of 0.3: YES in step S3), it determines that no label should be assigned to the target sentence (step S4).

[0023] On the other hand, if the highest score exceeds the first threshold value of 0.3 (NO in step S3), the first judgment unit 12 compares the highest score with the second threshold value of 0.7, and if the highest score exceeds the second threshold value of 0.7 (YES in step S5), it determines that a label should be assigned to the target sentence and associates the candidate label with the highest score with the target sentence (step S6).

[0024] Furthermore, if the highest score is greater than or equal to the first threshold of 0.3 and less than or equal to the second threshold of 0.7, the target sentence is deemed "undeterminable," and the target text, multiple candidate labels, the first threshold, and the second threshold are passed to the second determination unit 13.

[0025] Next, the second determination unit 13 further utilizes information on sentences in the target text that are different from the target sentence to perform a re-label determination on the target sentence as follows (steps S7 to S10). Here, the "sentence different from the target sentence" can be, for example, the sentence immediately before the target sentence in the text, the sentence immediately after the target sentence, or the first sentence in the paragraph containing the target sentence. Furthermore, the re-label determination is not performed on all of the multiple candidate labels, but only on candidate labels among the multiple candidate labels whose scores exceed the first threshold value of 0.3. Narrowing down the determination targets in this way reduces the processing load and improves processing efficiency.

[0026] Specifically, the second determination unit 13 calculates the scores of each of multiple candidate labels (only those candidate labels whose scores exceed the first threshold of 0.3) for the target sentence combined with a different sentence (e.g., the sentence immediately preceding the target sentence) based on the NLI (step S7).The second determination unit 13 then compares the score of the candidate label with the highest score among the multiple candidate labels (the highest score) with the second threshold of 0.7, and if the highest score does not exceed the second threshold of 0.7 (i.e., if the scores of all candidate labels do not exceed the second threshold of 0.7: NO in step S8), it determines not to assign a label to the target sentence (step S10).

[0027] On the other hand, if the highest score exceeds the second threshold value of 0.7 (YES in step S8), the second judgment unit 13 judges that a label should be assigned to the target sentence, and associates the candidate label with the highest score with the target sentence (step S9).

[0028] The processing by the first judgment unit 12 (steps S2 to S6) and the processing by the second judgment unit 13 (steps S7 to S10) described above are performed on each of the multiple sentences included in the target text, and when they have been performed for all sentences (YES in step S11), the scoring unit 14 scores each sentence based on the label judgment result of the target text, for example, as follows, and outputs the scoring result (step S12).

[0029] For example, in an English test that asks users to "create an English text using two or more of points 1 to 4," if the submitted English text includes the following sentences 1 to 9 and the following label judgment results are assumed: Sentence 1...Label: point 1 Sentence 2...Label: None Sentence 3...Label: point 1 Sentence 4...Label: point 2 Sentence 5...Label: point 2 + Δ Sentence 6...Label: point 2 Sentence 7...Label: None Sentence 8...Label: point 1 + Δ Sentence 9...Label: point 1 Sentences with "+Δ" added to the label judgment results above mean that they were deemed undeterminable in the processing by the first judgment unit 12, and a judgment result was obtained through re-judgment processing by the second judgment unit 13. Here, "point 1 + Δ" and "point 1" are determined to use the same point 1. However, the scoring may be differentiated between "point1+Δ" and "point1". For example, a sentence with "+Δ" added (i.e., a sentence that is deemed undeterminable in processing by the first determination unit 12) may be weighted lower than a sentence without "+Δ" added, and scored accordingly.

[0030] In the above assumed example, it is determined that point 1 and point 2 are used in the entire text out of points 1 to 4. Here, if the scoring rule is that the maximum score is 100, and using only one of points 1 to 4 is 50 points, and using two or more is 100 points, in the above assumed example, it is determined that point 1 and point 2 are used, so the scoring unit 14 outputs a scoring result of "100 points."

[0031] According to the embodiment described above, even if the processing by the first determination unit 12 results in "undeterminable," the second determination unit 13 performs a re-determination process on the combination of the target sentence and a different sentence (target sentence + different sentence). This prevents erroneous determinations and a decrease in the reliability of the score due to unrelated sentences, and takes into account the relevance between the sentences included in the text, making it possible to accurately determine the associated label for each sentence of the text.

[0032] (Modifications) There are variations to the re-label determination by the second determination unit 13 other than those described above, which will be described below as modifications.

[0033] In a modified example, the second determination unit 13 determines whether to label the target sentence based on whether the candidate label with the highest score in the processing by the first determination unit 12 (i.e., the candidate label associated with the target sentence in the processing by the first determination unit 12) is the same as the candidate label with the highest score associated with a different sentence.

[0034] Fig. 6 shows a processing flow by the terminal 10 in the case of the modified example. In Fig. 6, the processing of steps S7A and S8A in the re-label determination by the second determination unit 13 differs from the above-described embodiment.

[0035] That is, after the first determination unit 12 determines that the sentence is undeterminable, the second determination unit 13 calculates the scores of each of the multiple candidate labels (only the candidate labels whose scores exceed the first threshold value of 0.3) for the sentences in the target text that are different from the target sentence (step S7A).The second determination unit 13 then determines whether the candidate label with the highest score in the processing by the first determination unit 12 (i.e., the candidate label associated with the target sentence in the processing by the first determination unit 12) is the same as the candidate label with the highest score associated with the different sentence (the highest score in step S7A) (step S8A).

[0036] If it is determined that they are the same (YES in step S8A), the determination process by the first determination unit 12 is considered to be highly reliable, so the second determination unit 13 determines that a label should be assigned to the target sentence, and associates the candidate label that received the highest score in the determination process by the first determination unit 12 with the target sentence (step S9).

[0037] On the other hand, if it is determined in step S8A that the two candidate labels are not the same (NO in step S8A), the determination process by the first determination unit 12 is considered to be unreliable, and therefore the second determination unit 13 determines not to label the target sentence (step S10).

[0038] In the above-described modified example, it is possible to appropriately determine whether the reliability of the determination process by the first determination unit 12 is high depending on whether the candidate labels with the highest scores obtained for the target sentence and a different sentence match. Therefore, in the modified example, as in the above-described embodiment, it is possible to accurately determine the associated label for each sentence of the text by taking into account the relevance between the sentences included in the text, while preventing erroneous determinations and a decrease in the reliability of the score due to unrelated sentences.

[0039] The gist of the present disclosure lies in the following [1] to [9]: [1] A terminal including: a determination unit that, when labeling each sentence in a text consisting of a plurality of sentences based on Natural Language Inference (NLI), determines from test data a first threshold value for determining not to assign a label and a second threshold value for determining to assign the label; a first determination unit that performs label determination for the target sentence based on the score for each candidate label obtained based on the NLI, the score indicating the degree to which each of a plurality of candidate labels is derived from one target sentence in the target text, the first threshold value, and the second threshold value; and a second determination unit that, when the first determination unit is unable to perform a label determination for the target sentence, performs label determination for the target sentence again by further utilizing information of a sentence in the text that is different from the target sentence. [2] The terminal described in [1], wherein the first determination unit determines that the target sentence should not be labeled based on the magnitude relationship between the score of the candidate label with the highest score among the plurality of candidate labels and the first threshold, and when the first determination unit determines that the target sentence should be labeled based on the magnitude relationship between the score and the second threshold. [3] The terminal described in [1] or [2], wherein the second determination unit re-labels the target sentence based on the score for each of the plurality of candidate labels representing the degree to which each of the candidate labels is derived from the target sentence and the different sentence, and the second threshold. [4] The terminal described in [1] or [2], wherein the second determination unit re-labels the target sentence based on whether the candidate label with the highest score associated with the target sentence and the candidate label with the highest score associated with the different sentence are the same. [5] The terminal according to any one of [1] to [4], wherein the sentence in the text that is different from the target sentence is any one of: a sentence immediately before the target sentence, a sentence immediately after the target sentence, or a first sentence in a paragraph that includes the target sentence. [6] The terminal according to any one of [1] to [5], wherein the target that is subjected to re-label determination by the second determination unit is only a candidate label of the plurality of candidate labels whose score exceeds the first threshold.[7] The terminal according to any one of [1] to [6], wherein data for distinguishing a label obtained by the second label determination by the second determination unit from a label obtained by the initial label determination by the first determination unit is added to the label obtained by the second label determination. [8] The terminal according to any one of [1] to [7], wherein the first threshold and the second threshold are adjusted based on at least one of a degree of relevance between the plurality of sentences included in the text, a length of the sentences, and a number of the candidate labels to be determined. [9] A determination method comprising: a step of determining, from test data, a first threshold for determining not to assign a label and a second threshold for determining to assign the label when a terminal determines a label for each sentence in a text consisting of a plurality of sentences based on Natural Language Inference (NLI); a step of determining a label for the target sentence by the terminal based on the score for each candidate label obtained based on the NLI, the score representing the degree to which each of a plurality of candidate labels is derived from one target sentence in the text, the first threshold, and the second threshold; and a step of determining again a label for the target sentence by the terminal when the terminal is unable to determine a label for the target sentence by further utilizing information of a sentence in the text that is different from the target sentence.

[0040] [Explanation of Terms, Explanation of Hardware Configuration (FIG. 7), etc.] The block diagrams used in the description of the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (e.g., wired, wireless, etc.) and these multiple devices. The functional block may be realized by combining the single device or the multiple devices with software.

[0041] Functions include, but are not limited to, judgment, determination, assessment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.

[0042] For example, a terminal according to an embodiment of the present disclosure may function as a computer that executes the processes of the present disclosure. Fig. 7 is a diagram illustrating an example of a hardware configuration of a terminal 10 according to an embodiment of the present disclosure. The terminal 10 described above may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

[0043] In the following description, the term "device" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the terminal 10 may be configured to include one or more of the devices shown in the drawings, or may be configured to exclude some of the devices.

[0044] Each function of the terminal 10 is realized by loading specified software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communication via the communication device 1004, and control at least one of reading and writing data in the memory 1002 and storage 1003.

[0045] The processor 1001 controls the entire computer by running, for example, an operating system, and may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc.

[0046] The processor 1001 also reads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes in accordance with these. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. While the various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may be transmitted from a network via a telecommunications line.

[0047] The memory 1002 is a computer-readable recording medium and may be configured by, for example, at least one of a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing a wireless communication method according to an embodiment of the present disclosure.

[0048] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other appropriate medium including at least one of memory 1002 and storage 1003.

[0049] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 1004 may be configured to include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc. to realize at least one of frequency division duplex (FDD) and time division duplex (TDD).

[0050] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0051] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0052] The terminal 10 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0053] The notification of information is not limited to the aspects / embodiments described in the present disclosure and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI) and Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB) and System Information Block (SIB))), other signals, or a combination thereof. Furthermore, the RRC signaling may be referred to as an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.

[0054] Each aspect / embodiment described in the present disclosure may be implemented using any of the following standards: LTE (Long Term Evolution), LTE-Advanced (LTE-A), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (xG (x is, for example, an integer or a decimal number)), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.17 (WiMAX (registered trademark)), IEEE 802.19 (WiMAX (registered trademark)), IEEE 802.20 (WiMAX (registered trademark)), IEEE 802.21 (Wi-Fi (registered trademark)), IEEE 802.22 (WiMAX (registered trademark)), IEEE 802.23 (WiMAX (registered trademark)), IEEE 802.24 (WiMAX (registered trademark)), IEEE 802.25 (WiMAX (registered trademark)), IEEE 802.26 (WiMAX (registered trademark)), IEEE 802.27 (WiMAX (registered trademark)), IEEE 802.28 (WiMAX (registered trademark)), IEEE 802.29 (WiMAX (registered trademark)), IEEE 802.30 (WiMAX (registered trademark)), IEEE 802.31 (Wi-Fi (registered trademark)), IEEE 802.32 (WiMAX (registered trademark)), IEEE 802.33 (WiMAX (registered trademark)), IEEE 802.34 ( The present invention may be applied to at least one of systems using 802.20, UWB (Ultra-Wide Band), Bluetooth (registered trademark), or other suitable systems, and next-generation systems that are extended, modified, created, or defined based on these systems. The present invention may also be applied to a combination of multiple systems (e.g., a combination of LTE and / or LTE-A with 5G).

[0055] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0056] Input and output information may be stored in a specific location (for example, memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.

[0057] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0058] The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0059] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0060] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0061] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0062] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0063] Note that terms described in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of a channel and a symbol may be a signal (signaling). Furthermore, a signal may be a message. Furthermore, a component carrier (CC) may be called a carrier frequency, a cell, a frequency carrier, etc.

[0064] As used in this disclosure, the terms "system" and "network" are used interchangeably.

[0065] Furthermore, the information, parameters, etc. described in the present disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information. For example, a radio resource may be indicated by an index.

[0066] The names used for the above-described parameters are not intended to be limiting in any way. Furthermore, the mathematical expressions using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not intended to be limiting in any way.

[0067] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0068] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0069] As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0070] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

[0071] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0072] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0073] 10...terminal, 11...determination unit, 12...first judgment unit, 13...second judgment unit, 14...scoring unit, 1001...processor, 1002...memory, 1003...storage, 1004...communication device, 1005...input device, 1006...output device, 1007...bus.

Claims

1. A terminal comprising: a determination unit that, when labeling each sentence in a text consisting of multiple sentences based on Natural Language Inference (NLI), determines from test data a first threshold for determining not to assign a label and a second threshold for determining to assign the label; a first determination unit that determines a label for the target sentence based on the score for each candidate label obtained based on the NLI, the score representing the degree to which each of multiple candidate labels is derived from a single target sentence in the target text, the first threshold, and the second threshold; and a second determination unit that, when the first determination unit is unable to make a determination about the target sentence, re-labels the target sentence by further utilizing information about sentences in the text that are different from the target sentence.

2. The terminal described in claim 1, wherein the first judgment unit determines that the target sentence is undecidable when it cannot determine that a label should not be assigned to the target sentence based on the relationship between the score of the candidate label with the highest score among the plurality of candidate labels and the first threshold value, and cannot determine that a label should be assigned based on the relationship between the score and the second threshold value.

3. The terminal described in claim 1, wherein the second judgment unit re-judges the label of the target sentence based on the score for each of the plurality of candidate labels representing the degree to which each of the candidate labels is derived from the target sentence and the different sentence, and the second threshold value.

4. The terminal according to claim 1, wherein the second determination unit performs a label determination for the target sentence again based on whether the highest-scoring candidate label associated with the target sentence is the same as the highest-scoring candidate label associated with the different sentence.

5. The terminal according to claim 1, wherein the sentence in the text that is different from the target sentence is either the sentence immediately before the target sentence, the sentence immediately after the target sentence, or the first sentence in a paragraph that includes the target sentence.

6. The terminal according to claim 1, wherein the second determination unit performs relabel determination on only candidate labels among the plurality of candidate labels whose scores exceed the first threshold.

7. The terminal according to claim 1, wherein data for distinguishing the label obtained by the second label determination by the second determination unit from the label obtained by the first label determination by the first determination unit is added to the label obtained by the second label determination by the second determination unit.

8. The terminal described in claim 1, wherein the first threshold and the second threshold are adjusted based on at least one of the degree of relevance between the multiple sentences contained in the text, the length of the sentences, and the number of candidate labels to be determined.

9. A determination method comprising: a step in which, when a terminal determines a label for each sentence in a text consisting of multiple sentences based on Natural Language Inference (NLI), it determines from test data a first threshold for determining not to assign a label and a second threshold for determining to assign the label; a step in which the terminal determines a label for the target sentence based on the score for each candidate label obtained based on the NLI, the score representing the degree to which each of multiple candidate labels is derived from a single target sentence in the text, the first threshold, and the second threshold; and a step in which, when the terminal is unable to make a determination for the target sentence, it re-labels the target sentence by further utilizing information on sentences in the text that are different from the target sentence.

Images