Device and method

Abstract

A device (10) comprises: an input part (11) that inputs, to an AI model (trained model (12)) capable of performing a process of a sentence rearrangement technique (sentence ordering) in which a plurality of sentences are rearranged in a correct order, a post-sorting text that is a text obtained by sorting a target text in a sentence unit; and a determination part (13) that determines the naturalness (coherence) of the logical expansion of the target text in accordance with the degree of restoration, to the target text, of a rearranged text that is a text rearranged and outputted from the AI model.

Description

Apparatus and method

[0001] The present disclosure relates to an apparatus and method for scoring text containing multiple sentences, for example, in grading an English test, in terms of the naturalness of logical development (referred to as "Coherence"), which correlates with the clarity of the correspondence between the sentences.

[0002] The coherence of a text containing multiple sentences consists of two elements: the naturalness of logical development (hereinafter referred to as "Coherence"), which correlates with the clarity of the correspondence between sentences, and Consistency, which refers to the lack of inconsistency in the overall argument. For example, when grading an English test, the target text may be scored from the perspective of Coherence (see Non-Patent Document 1). Conventionally, when scoring from the perspective of Coherence, scoring has been done manually or automatically using techniques such as Natural Language Inference (NLI) and topic models.

[0003] "Sentence Ordering and Coherence Modeling using Recurrent Neural Networks," Lajanugen Logeswaran et al., December 2017

[0004] Among the above methods, manual text scoring requires not only the literal meaning of each sentence but also an understanding of its content, making it a very time-consuming task for the scorer, and scoring large amounts of text is costly and impractical. Therefore, automatic scoring is desirable, but the above-mentioned methods using natural language inference (NLI) have difficulty in comprehensively evaluating and scoring the entire text, and the above-mentioned methods using topic models have difficulty in evaluating and scoring the logical development between sentences, posing issues with accuracy.

[0005] Therefore, an object of the present disclosure is to score text from the perspective of coherence at low cost and with high accuracy.

[0006] Based on the finding that when a sentence ordering technique is performed, the higher the naturalness of the logical development (coherence), which correlates with the clarity of the correspondence between sentences in the target text, the more appropriate the sentence ordering, the applicant came up with the idea of ​​using sentence ordering to score text in terms of coherence.

[0007] The device according to the present disclosure includes an input unit that inputs rearranged text, which is text obtained after rearranging a target text on a sentence-by-sentence basis, to an AI model that is capable of performing a sentence ordering process that rearranges multiple sentences into the correct order, and a determination unit that determines the naturalness (coherence) of the logical development of the target text in accordance with the degree to which the rearranged text, which is text that has been output from the AI ​​model, is restored to the target text.

[0008] According to the present disclosure, it is possible to score text in terms of coherence at low cost and with high accuracy.

[0009] FIG. 1 is a functional block configuration diagram of the device. FIG. 2 is a diagram for explaining processing in a learning stage. FIG. 3 is a flow diagram showing processing in a scoring execution stage. FIG. 4 is a diagram showing an example of original text. FIG. 5 is a diagram showing an example of rearranged text. FIG. 6 is a diagram showing an example of rearranged text output from a trained model. FIG. 7 is a diagram showing an example of the hardware configuration of the device.

[0010] An embodiment of an apparatus and method according to the present disclosure will be described below with reference to the drawings. An embodiment will be described below assuming that English test scoring is used as the scoring method. In the following embodiment, an example will be described in which a trained model that has been machine-learned for a sentence ordering process is used as an AI model capable of performing a sentence ordering process for rearranging multiple sentences into the correct order.

[0011] 1, an apparatus 10 according to this embodiment includes, as functional blocks, an input unit 11, a trained model 12, a determination unit 13, and an output unit 14. The functions of each unit will be outlined below.

[0012] The input unit 11 is a functional unit that inputs text (hereinafter referred to as "rearranged text") obtained by rearranging the target text sentence by sentence based on random numbers into a trained model 12 of a sentence ordering method that rearranges multiple sentences in the correct order.

[0013] As shown in FIG. 2 , the trained model 12 is a model acquired by machine learning using text obtained by rearranging a training source text containing multiple sentences on a sentence-by-sentence basis (hereinafter referred to as "rearranged text") as input data and the source text as correct answer data. The trained model 12 is a sentence ordering trained model trained by machine learning to output text rearranged in the correct order. Furthermore, in the inference phase, when the trained model 12 receives the rearranged text obtained by rearranging the target text on a sentence-by-sentence basis, the trained model 12 outputs text rearranged in the correct order (hereinafter referred to as "rearranged text"). Note that the device 10 includes a memory for storing (storing) the trained model 12 as part of its configuration. However, for simplicity, the trained model 12 will be described as being one of the functional blocks of the device 10. Instead of using the sentence ordering trained model 12 trained by machine learning as described above, a large language model (LLM), which is an example of an AI model and is primarily used for text generation, may be used.

[0014] In normal Sentence Ordering, machine learning is performed using source text for training that matches the target text in the inference phase. In contrast, in this Sentence Ordering, although there is no particular specification for the target text in the inference phase, text with good logical development and high reliability in terms of coherence (naturalness of logical development) (e.g., news articles, edited essays, academic papers, etc.) is used as the source text for training.

[0015] The determination unit 13 is a functional unit that determines the coherence (naturalness of logical development) of the target text according to the degree of restoration of the rearranged text output from the trained model 12 to the target text. The determination unit 13 determines the coherence based on an evaluation index. The evaluation index may be at least one of the following: (1) an evaluation index for evaluating whether the rearranged text completely matches the target text; (2) an evaluation index based on the longest number of parts whose order matches between the rearranged text and the target text (longest common subsequence (LCS)); (3) an evaluation index based on the rate at which positions in the rearranged text were accurately predicted; (4) an evaluation index representing the rate at which the order of the rearranged text matches or does not match between the order of the target text and the rearranged text; and (5) an evaluation index based on the longest number of parts whose order matches between the rearranged text and the target text (length of the longest ascending contiguous subsequence).

[0016] The output unit 14 is a functional unit that outputs a score obtained based on a predetermined criterion from the Coherence of the target text determined by the determination unit 13. Note that "output" can take various forms, such as display output, print output, or data transmission to an external device 10.

[0017] (Processing Executed in Device 10) The processing executed in the device 10 in the inference phase (processing based on the method of the present disclosure) will be described below with reference to the flow diagram of FIG.

[0018] First, the input unit 11 receives a target text containing a plurality of sentences (step S1). For example, the 11 sentences shown in Fig. 4 are received. For the sake of explanation, each sentence is numbered 1 to 11 to indicate its order.

[0019] Next, the input unit 11 rearranges the received target text sentence by sentence based on random numbers (step S2). The random numbers used here are not limited to specific random numbers. For example, random numbers based on a pseudo-random number generation algorithm with no duplicates can be used. As shown in Figure 5, the 11 sentences shown in Figure 4 are rearranged sentence by sentence according to the sequence [3, 7, 8, 5, 4, 2, 1, 9, 6, 11, 10] obtained by rearranging a total of 11 numbers from 1 to 11 without duplicates based on the random numbers.

[0020] Next, the input unit 11 inputs the rearranged text obtained in step S2 to the trained model 12 (step S3). At this time, the target text before rearrangement is passed to the determination unit 13.

[0021] The trained model 12 outputs the rearranged text rearranged in the correct order (for example, the rearranged text rearranged based on the Sentence Ordering shown in FIG. 6 ) to the determination unit 13 .

[0022] Next, the determination unit 13 determines coherence according to the degree of restoration of the rearranged text to the target text as follows (step S4). For example, using the rearranged text of Figure 6 and the target text of Figure 4, and the evaluation index (2) LCS, one of the evaluation indexes described above, the rearranged text is [2,1,3,4,5,6,7,9,10,11,8], and the original target text is [1,2,3,4,5,6,7,8,9,10,11]. Therefore, the part with the same order from left to right is [1,3,4,5,6,7,9,10,11], and the longest part with the same order is 9. Therefore, the evaluation index is (longest number / total length) = (9 / 11) = 0.8182. Therefore, the determination unit 13 determines coherence to be 0.8182 and passes the determined coherence to the output unit 14.

[0023] Finally, the output unit 14 outputs a score based on the obtained coherence (step S5). Here, the coherence value itself may be output as a score, or the coherence value may be converted into a numerical value out of 100 points or into a multi-level (e.g., 5-level) evaluation rank according to a predetermined standard, and the converted value may be output as a score.

[0024] In the embodiment described above, based on the finding that the higher the coherence, which correlates with the clarity of the correspondence between sentences in the target text, the more appropriately the sentences can be rearranged, the text is scored from the perspective of coherence using sentence ordering. That is, by inputting rearranged text, in which the target text is rearranged sentence by sentence, into the trained model 12 of sentence ordering, the rearranged text is output from the trained model 12, and the coherence of the target text is determined according to the degree to which the obtained rearranged text is restored to the target text. The coherence of the target text can then be used to score the target text. This avoids the need for conventional manual scoring and automates scoring, thereby reducing the cost required for scoring. Furthermore, by adopting the above-described new scoring method instead of the conventional method using natural language inference (NLI), which had significant issues with scoring accuracy, scoring can be performed with higher accuracy than conventional methods. As described above, a method superior to conventional methods can be obtained in terms of both "scoring cost" and "scoring accuracy."

[0025] In the above embodiment, an example has been described in which a trained model 12 that has been machine-learned for sentence ordering processing is used as an AI model capable of performing the processing. The trained model 12 is obtained by machine learning using text obtained by rearranging an original text sentence by sentence as input data and the original text as correct answer data, that is, machine learning using input data and correct answer data that can be obtained very easily. Therefore, machine learning for obtaining the trained model 12 can be performed very easily.

[0026] The output unit 14 obtains and outputs a score based on predetermined criteria from the Coherence of the target text determined by the determination unit 13. Therefore, in addition to a mode in which the Coherence of the target text itself is output as a score, various modes can be very flexibly adopted, such as converting the Coherence into a numerical value out of 100 points or into an evaluation rank in multiple stages (for example, 5 stages) according to predetermined criteria and outputting the converted value as a score.

[0027] The rearrangement for obtaining the rearranged text (the text after rearranging the target text sentence by sentence) to be input to the trained model 12 can be performed based on random numbers. The random numbers used at this time are not limited to special ones, and for example, random numbers based on a general-purpose non-duplicate pseudo-random number generation algorithm can be used.

[0028] The determiner 13 determines the coherence based on an evaluation index. The evaluation index used at this time is not limited to a specific one, and may be, for example, at least one of the following: (1) an evaluation index for evaluating whether the rearranged text completely matches the target text, (2) an evaluation index based on the longest number of portions whose order matches between the rearranged text and the target text (the aforementioned LCS), (3) an evaluation index based on the rate at which positions in the rearranged text can be accurately predicted, (4) an evaluation index representing the rate at which the order of the rearranged text matches or does not match with that of the target text, and (5) an evaluation index based on the longest number of portions whose order matches between the rearranged text and the target text (the length of the longest ascending contiguous subsequence).

[0029] The gist of the present disclosure lies in the following [1] to [7]. [1] A device comprising: an input unit that inputs reordered text, which is text obtained by reordering a target text on a sentence-by-sentence basis, to an AI model capable of performing a sentence ordering process that reorders multiple sentences into a correct order; and a determination unit that determines the naturalness (coherence) of the logical development of the target text based on the degree to which the reordered text, which is text obtained by reordering a target text on a sentence-by-sentence basis and output from the AI ​​model, is restored to the target text. [2] The device described in [1], wherein the AI ​​model is obtained by machine learning using text obtained by reordering an original text on a sentence-by-sentence basis as input data and the original text as correct answer data. [3] The device described in [1] or [2], further comprising: an output unit that outputs a score obtained based on a predetermined criterion from the coherence of the target text determined by the determination unit. [4] The device described in any one of [1] to [3], wherein the reordering of text on a sentence-by-sentence basis is performed based on random numbers. [5] The device according to any one of [1] to [4], wherein the determiner determines the coherence based on an evaluation index. [6] The device according to [5], wherein the evaluation index is at least one of: an evaluation index for evaluating whether the reordered text completely matches the target text, an evaluation index based on the longest number of portions whose order matches between the reordered text and the target text (the length of a longest common subsequence), an evaluation index based on the rate at which positions in the reordered text were accurately predicted, an evaluation index representing the rate at which the order of the reordered text matches or does not match with that of the target text, and an evaluation index based on the longest number of portions whose order matches between the reordered text and the target text (the length of a longest ascending contiguous subsequence).[7] A method comprising the steps of: inputting rearranged text, which is text obtained after rearranging a target text on a sentence-by-sentence basis, to an AI model capable of performing a sentence ordering process in which a device rearranges multiple sentences into the correct order; and determining the coherence of the logical development of the target text in accordance with the degree to which the rearranged text output from the AI ​​model is restored to the target text.

[0030] [Explanation of Terms, Explanation of Hardware Configuration (FIG. 7), etc.] The block diagrams used in the description of the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (e.g., wired, wireless, etc.) and these multiple devices. The functional block may be realized by combining the single device or the multiple devices with software.

[0031] Functions include, but are not limited to, judgment, determination, assessment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.

[0032] For example, an apparatus according to an embodiment of the present disclosure may function as a computer that executes the processes of the present disclosure. Fig. 7 is a diagram illustrating an example of a hardware configuration of an apparatus 10 according to an embodiment of the present disclosure. The above-described apparatus 10 may be physically configured as a computer including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

[0033] In the following description, the term "apparatus" may be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the apparatus 10 may be configured to include one or more of the apparatuses shown in the drawings, or may be configured to exclude some of the apparatuses.

[0034] Each function of the device 10 is realized by loading specified software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communication via the communication device 1004, and control at least one of reading and writing data in the memory 1002 and storage 1003.

[0035] The processor 1001 controls the entire computer by running, for example, an operating system, and may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc.

[0036] The processor 1001 also reads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes in accordance with these. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. While the various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may be transmitted from a network via a telecommunications line.

[0037] The memory 1002 is a computer-readable recording medium and may be configured by, for example, at least one of a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing a wireless communication method according to an embodiment of the present disclosure.

[0038] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other appropriate medium including at least one of memory 1002 and storage 1003.

[0039] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 1004 may be configured to include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc. to realize at least one of frequency division duplex (FDD) and time division duplex (TDD).

[0040] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0041] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0042] The device 10 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0043] The notification of information is not limited to the aspects / embodiments described in the present disclosure and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI) and Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB) and System Information Block (SIB))), other signals, or a combination thereof. Furthermore, the RRC signaling may be referred to as an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.

[0044] Each aspect / embodiment described in the present disclosure may be implemented using any of the following standards: LTE (Long Term Evolution), LTE-Advanced (LTE-A), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (xG (x is, for example, an integer or a decimal number)), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.17 (WiMAX (registered trademark)), IEEE 802.19 (WiMAX (registered trademark)), IEEE 802.20 (WiMAX (registered trademark)), IEEE 802.21 (Wi-Fi (registered trademark)), IEEE 802.22 (WiMAX (registered trademark)), IEEE 802.23 (WiMAX (registered trademark)), IEEE 802.24 (WiMAX (registered trademark)), IEEE 802.25 (WiMAX (registered trademark)), IEEE 802.26 (WiMAX (registered trademark)), IEEE 802.27 (WiMAX (registered trademark)), IEEE 802.28 (WiMAX (registered trademark)), IEEE 802.29 (WiMAX (registered trademark)), IEEE 802.30 (WiMAX (registered trademark)), IEEE 802.31 (Wi-Fi (registered trademark)), IEEE 802.32 (WiMAX (registered trademark)), IEEE 802.33 (WiMAX (registered trademark)), IEEE 802.34 ( The present invention may be applied to at least one of systems using 802.20, UWB (Ultra-Wide Band), Bluetooth (registered trademark), or other suitable systems, and next-generation systems that are extended, modified, created, or defined based on these systems. The present invention may also be applied to a combination of multiple systems (e.g., a combination of LTE and / or LTE-A with 5G).

[0045] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0046] Input and output information may be stored in a specific location (for example, memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0047] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0048] The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0049] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0050] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0051] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0052] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0053] Note that terms described in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of a channel and a symbol may be a signal (signaling). Furthermore, a signal may be a message. Furthermore, a component carrier (CC) may be called a carrier frequency, a cell, a frequency carrier, etc.

[0054] As used in this disclosure, the terms "system" and "network" are used interchangeably.

[0055] Furthermore, the information, parameters, etc. described in the present disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information. For example, a radio resource may be indicated by an index.

[0056] The names used for the above-described parameters are not intended to be limiting in any way. Furthermore, the mathematical expressions using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not intended to be limiting in any way.

[0057] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0058] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0059] As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0060] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

[0061] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0062] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0063] 10...device, 11...input unit, 12...trained model (AI model), 13...decision unit, 14...output unit, 1001...processor, 1002...memory, 1003...storage, 1004...communication device, 1005...input device, 1006...output device, 1007...bus.

Claims

1. An apparatus comprising: an input unit that inputs rearranged text, which is text obtained after rearranging a target text sentence by sentence, into an AI model capable of performing a sentence ordering process that rearranges multiple sentences into the correct order; and a determination unit that determines the naturalness (coherence) of the logical development of the target text based on the degree to which the rearranged text, which is text output from the AI ​​model, is restored to the target text.

2. The device according to claim 1, wherein the AI ​​model is obtained by machine learning using text obtained by rearranging an original text sentence by sentence as input data and the original text as correct answer data.

3. The device according to claim 1, further comprising: an output unit that outputs a score obtained based on a predetermined criterion from the coherence of the target text determined by the determination unit.

4. The device according to claim 1, wherein the rearrangement of text by sentence is performed based on random numbers.

5. The device according to claim 1, wherein the determination unit determines the coherence based on an evaluation index.

6. The device of claim 5, wherein the evaluation index is at least one of: an evaluation index for evaluating whether the reordered text completely matches the target text; an evaluation index based on the longest number of parts whose order matches between the reordered text and the target text (the length of the longest common subsequence); an evaluation index based on the rate at which positions in the reordered text were accurately predicted; an evaluation index representing the rate at which the order of the reordered text matches or does not match with that of the target text; and an evaluation index based on the longest number of parts whose order matches consecutively between the reordered text and the target text (the length of the longest ascending contiguous subsequence).

7. A method comprising the steps of: an apparatus inputting rearranged text, which is text obtained after rearranging a target text sentence by sentence, into an AI model capable of performing a sentence ordering process that rearranges multiple sentences into the correct order; and the apparatus determining the coherence of the logical development of the target text based on the degree to which the rearranged text output from the AI ​​model is restored to the target text.