Non-transitory computer-readable storage medium, information processing method

Abstract

To easily determine information suitable for use as a feature amount of data related to a state transition.SOLUTION: The information processing apparatus 10 generates a question sentence 3 that instructs, in a natural language, extraction of a first state transition that satisfies a predetermined condition related to meanings of states before and after the transition from a data set 1 in which a series of transitions of states of a plurality of events is indicated. Next, the information processor 10 inputs the question sentence 3 to the dialogue system 2 that performs dialogue in natural languages using the language model 2a. Then, the information processing apparatus 10 determines that information based on whether or not the first state transition indicated in the answer sentence 4 output by the dialog system 2 is included in the series of transitions of the states of each of the plurality of events is used as the feature amount of each of the plurality of events.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] Automated Machine Learning (AutoML) is a technology for efficiently implementing machine learning, which automatically generates a machine learning model using a dataset as input. To obtain a high-performance model using AutoML, it is important to generate features that appropriately represent the characteristics of the dataset.

[0003] As a technology for generating features, for example, a data analysis device has been proposed that can effectively narrow down the features to be generated when obtaining features from a large amount of data, enabling the rapid generation of effective features. As a technology related to machine learning, a machine learning method has also been proposed that allows users to easily determine what future scenarios AI (Artificial Intelligence) is assuming when outputting data. Furthermore, a feature learning method has also been proposed that adds features to a feature learning model when the reconstruction error of a data sample meets a threshold. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-24544 [Patent Document 2] Japanese Patent Application Publication No. 2023-164155 [Patent Document 3] US Patent Application Publication No. 2020 / 311544 Summary of the Invention [Problem to be solved by the invention]

[0005] Machine learning may handle data related to events that have multiple states and transition between states. An example of an event that transitions between states is a process that indicates the progress of a business operation. Such state transitions have a wide range of possible patterns, making it difficult to determine which state transitions should be used as features when implementing machine learning.

[0006] In one aspect, the present invention aims to easily determine information suitable for use as a feature of data related to state transitions. [Means for solving the problem]

[0007] In one proposal, an information processing program is provided that causes a computer to perform the following processing. A computer generates a question in natural language that instructs extraction of a first state transition that satisfies a predetermined condition regarding the meaning of states before and after the transition from a dataset indicating a series of transitions of states of a plurality of events. The computer inputs the question to a dialogue system that conducts a dialogue in natural language using a language model that outputs an answer in natural language text to a question in natural language text. The computer then determines to use information based on whether the first state transition indicated in the answer text output by the dialogue system is included in the series of transitions of states of each of the plurality of events as a feature quantity for each of the plurality of events. [Effects of the Invention]

[0008] According to one aspect, it is possible to easily determine information that is suitable for use as a feature quantity of data related to state transitions. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 illustrates an example of an information processing method according to the first embodiment. [Figure 2] FIG. 10 illustrates an example of a system configuration according to a second embodiment. [Figure 3] FIG. 1 is a diagram illustrating an example of hardware of a machine learning system. [Figure 4] FIG. 10 is a diagram illustrating an example of a data set showing an execution history of a state transition process. [Figure 5] FIG. 10 is a diagram illustrating a first example of a method for automatically creating feature amounts for each process. [Figure 6] FIG. 10 is a diagram illustrating an example of a problem that occurs when information indicating whether the frequency of a path is exceptional is used as a feature. [Figure 7] FIG. 10 is a diagram illustrating a second example of a method for automatically creating feature amounts for each process. [Figure 8] 10A and 10B are diagrams illustrating an example of a problem that occurs when information on whether a path passes through a pattern is used as a feature. [Figure 9] FIG. 1 is a block diagram illustrating an example of a function for performing machine learning on a state transition process. [Figure 10] FIG. 10 is a diagram illustrating an example of a method for generating feature information used in machine learning targeting a state transition process. [Figure 11] FIG. 10 is a diagram illustrating an example of categorization of meaningless patterns. [Figure 12] 10 is a flowchart illustrating an example of a procedure for machine learning processing. [Figure 13] 10 is a flowchart illustrating an example of a procedure for generating a feature amount determination criterion. [Figure 14] 10 is a flowchart illustrating an example of a procedure for a process statistics calculation process. [Figure 15] FIG. 10 is a diagram illustrating an example of a question sentence in response to a feature determination request. [Figure 16] FIG. 10 is a diagram showing an example of a reply sentence. [Figure 17] FIG. 10 is a diagram illustrating an example of a category classification instruction. [Figure 18] FIG. 10 is a diagram illustrating an example of a classification result. [Figure 19] FIG. 10 is a diagram illustrating an example of feature amount determination criterion information. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present embodiment will be described below with reference to the drawings. Note that each embodiment can be implemented in combination with a plurality of other embodiments within a range that does not contradict each other. [First embodiment] The first embodiment is an information processing method that can easily determine appropriate information to be used as a feature indicating the characteristics of each event when a dataset indicating the transition of the state of an event is used for machine learning.

[0011] Fig. 1 is a diagram showing an example of an information processing method according to a first embodiment. Fig. 1 shows an information processing device 10 for implementing the information processing method according to the first embodiment. The information processing device 10 can implement the information processing method according to the first embodiment by, for example, executing an information processing program.

[0012] The information processing device 10 includes a storage unit 11 and a processing unit 12. The storage unit 11 is, for example, a memory or a storage device included in the information processing device 10. The processing unit 12 is, for example, a processor or an arithmetic circuit included in the information processing device 10.

[0013] The storage unit 11 stores, for example, a dataset 1 used for machine learning. The dataset 1 shows a series of state transitions for a plurality of events. The events are, for example, processes that represent the progress of work in a business. When the events are processes, the dataset 1 contains, for example, a process ID that identifies the executed process, and sets therein the date and time when the process entered a new state and text information indicating the state.

[0014] The processing unit 12 obtains feature quantities for each of a plurality of events from the dataset 1 and performs machine learning. In this process, the processing unit 12 uses the dialogue system 2 to determine information appropriate for use as feature quantities. The dialogue system 2 receives a question in a natural language sentence as input and uses a language model 2a that outputs a response to the question in a natural language sentence to carry out a dialogue in natural language. The language model 2a is, for example, a machine learning model of a neural network generated by deep learning.

[0015] The processing unit 12 generates a question sentence 3 in a natural language for issuing a predetermined instruction to the dialogue system 2. The instruction to the dialogue system 2 is an instruction to extract, from the dataset 1, a first state transition that satisfies predetermined conditions regarding the meaning of the states before and after the transition. For example, the question sentence 3 is an instruction to extract a state transition that cannot be said to be a normal series of transitions of the states of an event when the meaning of the states is taken into consideration, such as "Show me a state transition that does not make sense." Then, the processing unit 12 inputs the generated question sentence 3 to the dialogue system 2.

[0016] The processing unit 12 executes the dialogue system 2 to which the question sentence 3 is input. As a result, an answer sentence 4 is obtained when the question sentence 3 is input to the language model 2a. When the dataset 1 indicates a series of state transitions of a business process, the answer sentence 4 indicates, for example, a state transition of "start → end" and a state transition of "start → reception → end" as the first state transition. These state transitions are extracted because they end without passing through a state such as "in progress" that represents work related to the business, and therefore cannot be said to be normal state transitions.

[0017] The processing unit 12 determines to set information based on whether or not the first state transition indicated in the response sentence 4 output by the dialogue system 2 is included in the series of state transitions of each of the plurality of events as a feature for each of the plurality of events. For example, the processing unit 12 sets, for each first state transition, information on whether or not the first state transition is included in the series of state transitions of the event as a feature. For example, each of information such as "does it go through start → end?" and "does it go through start → reception → end?" is determined as a feature.

[0018] The processing unit 12 generates feature information 5 indicating the feature of each of the multiple events shown in the dataset 1, based on information based on whether the first state transition shown in the response sentence 4 output by the dialogue system 2 is included in the series of state transitions of each of the multiple events. For example, if the event (process) passes through the first state transition "start → end", the processing unit 12 sets the feature of the event "passes through start → end" to "True". If the event (process) does not pass through the first state transition "start → end", the processing unit 12 sets the feature of the event "passes through start → end" to "False".

[0019] The processing unit 12 performs machine learning using the feature amount information 5. For example, the processing unit 12 receives the feature amount information 5 and generates a model that outputs an abnormal state transition. In this way, by inputting a question sentence 3 regarding the state transition of the data set 1 into a dialogue system 2 using a language model 2a and obtaining an answer sentence 4, it is possible to easily determine appropriate information as a feature.

[0020] The processing unit 12 may include information indicating possible states of an event, such as a business process, in the question 3. For example, the processing unit 12 receives input of a list of possible states of an event from the user, and includes the list of states input by the user in the question 3. The processing unit 12 also receives designation of a column in the dataset 1 in which possible states are set from the user. The processing unit 12 then generates a list of states set in the specified column, and includes the generated list in the question 3 as a list of possible states. By including a list of possible states of an event, such as a business process, in the question 3, the accuracy of the answer provided by the dialogue system 2 is improved.

[0021] The processing unit 12 can also reduce the number of features used in machine learning by categorizing the first state transitions indicated in the answer sentence 4. For example, the processing unit 12 inputs to the dialogue system 2 a question sentence 3 that instructs the dialogue system 2 to output an answer sentence 4 including an explanatory statement explaining why the first state transition has been determined to satisfy a predetermined condition. When multiple first state transitions are indicated in the answer sentence 4, the processing unit 12 categorizes the multiple first state transitions based on the explanatory statements. Then, the processing unit 12 assigns a unified feature to two or more second state transitions classified into the same category among the first state transitions.

[0022] That is, first state transitions extracted for similar reasons are classified into the same category. For example, "Start → End" and "Start → Reception → End" are unified into the category "Early Close," which indicates a state transition that ends early.

[0023] When categorization is performed, the processing unit 12 generates feature information 6 in which feature amounts of events (processes) in the dataset 1 are set for each category. In the feature information 6, for events (processes) that pass through at least one of two or more second state transitions classified into a category, the feature amount is "True." Furthermore, for events (processes) that do not pass through all of two or more second state transitions classified into a category, the feature amount is "False."

[0024] By performing categorization based on the reason for extracting the first state transition, the number of feature amounts can be appropriately reduced. The processing unit 12 can also perform category classification using the dialogue system 2. For example, the processing unit 12 inputs an instruction indicating category classification of a plurality of first state transitions to the dialogue system 2, and obtains the classification result output by the dialogue system 2. When the reliability of the answer of the language model 2a is high, it is possible to perform highly accurate category classification by performing category classification using the dialogue system 2.

[0025] The processing unit 12 may include in the question 3 statistical values ​​relating to a series of transitions of states of a plurality of events shown in the dataset 1. For example, the processing unit 12 determines, as statistical values, the series of state transitions (from first to last) that occur most frequently, the state that occurs most frequently, or one or a series of state transitions that occur most frequently, based on the dataset 1. Then, the processing unit 12 transmits the question 3 including the determined statistical values ​​to the dialogue system 2.

[0026] By including statistical values ​​in question 3 in this way, the accuracy of determining whether or not each state transition should be the first state transition is improved when the dialogue system 2 is executed. For example, it can be determined that a state transition included in a series of state transitions (from start to finish) that occurs most frequently does not correspond to a meaningless state transition. As a result, the occurrence of a mistake such as including a meaningful state transition as the first state transition in answer 4 to question 3, "Show me a meaningless state transition," is prevented.

[0027] The processing unit 12 generates the sentence for the question 3 according to, for example, the purpose of machine learning. For example, if the purpose is to generate a machine learning model that detects abnormal state transitions, the processing unit 12 sets the sentence instructing the output of state transitions that do not make sense as a normal series of transitions between the states of multiple events as the question 3. Also, if the purpose is to generate a machine learning model that detects state transitions that may cause trouble, the processing unit 12 sets the sentence instructing the extraction of state transitions that should be carefully monitored in natural language as the question 3.

[0028] By generating question sentence 3 in accordance with the purpose of machine learning in this way, it becomes possible to generate feature information that is suitable for the purpose of machine learning, thereby improving the accuracy of the model generated by machine learning. Second Embodiment The second embodiment is a computer system that automatically generates features for generating a machine learning model from a dataset that includes an execution log of a state transition process that indicates the process of performing a business operation.

[0029] 2 is a diagram illustrating an example of a system configuration according to the second embodiment. A machine learning system 100, a business system 200, a management terminal 300, and a plurality of business terminals 30a, 30b, etc. are connected via a network 20.

[0030] The machine learning system 100 is a computer that provides services such as model generation through machine learning and inference using the generated model. The functions of the machine learning system 100 are provided using, for example, a client computing system. The business system 200 is a computer that manages business operations in an organization such as a company. The management terminal 300 is a computer used by a user who manages the implementation of business operations. The business terminals 30a, 30b, ... are computers used by users who perform tasks, inquiries, etc. related to business operations.

[0031] For example, logs of operations performed using business terminals 30a, 30b, ... are stored in business system 200. The stored logs can be used as input data sets for learning or inference in machine learning. For example, to prevent business troubles, machine learning system 100 creates a model for detecting state transitions that require attention based on the business logs, and detects state transitions that require attention using the model.

[0032] When performing machine learning, for example, the management terminal 300, in response to an instruction from a user, sends a machine learning request to the machine learning system 100, which requests a machine learning model generation process or an inference process using a trained model. The machine learning request includes a dataset to be used for machine learning.

[0033] The machine learning system 100 generates a machine learning model or performs inference processing based on a learned model in response to a machine learning request from the management terminal 300. The machine learning system 100 transmits the processing results to the management terminal 300.

[0034] 3 is a diagram showing an example of hardware for a machine learning system. The entire machine learning system 100 is controlled by a processor 101. A memory 102 and multiple peripheral devices are connected to the processor 101 via a bus 109.

[0035] The processor 101 may be a multiprocessor. The processor 101 is, for example, a central processing unit (CPU), a micro processing unit (MPU), or a digital signal processor (DSP). At least some of the functions realized by the processor 101 executing a program may be realized by an electronic circuit such as an application specific integrated circuit (ASIC) or a programmable logic device (PLD).

[0036] The memory 102 is used as a main storage device of the machine learning system 100. The memory 102 temporarily stores at least a part of the OS (Operating System) program and application programs to be executed by the processor 101. The memory 102 also stores various data used in processing by the processor 101. As the memory 102, for example, a volatile semiconductor storage device such as a RAM (Random Access Memory) is used.

[0037] The peripheral devices connected to the bus 109 include a storage device 103, a graphics controller 104, an input interface 105, an optical drive device 106, a device connection interface 107, and a network interface 108.

[0038] The storage device 103 electrically or magnetically writes and reads data to and from a built-in recording medium. The storage device 103 is used as an auxiliary storage device for the machine learning system 100. The storage device 103 stores an OS program, application programs, and various data. Note that the storage device 103 may be, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0039] The graphics controller 104 is an arithmetic device that performs image processing. The graphics controller 104 is, for example, a GPU (Graphics Processing Unit). A monitor 21 is connected to the graphics controller 104. The graphics controller 104 displays an image on the screen of the monitor 21 in accordance with an instruction from the processor 101. The monitor 21 may be a display device using organic EL (Electro Luminescence) or a liquid crystal display device. When a GPU is used as the graphics controller 104, for example, the graphics controller 104 can also perform complex numerical calculations such as matrix calculations.

[0040] The input interface 105 is connected to a keyboard 22 and a mouse 23. The input interface 105 transmits signals sent from the keyboard 22 and the mouse 23 to the processor 101. The mouse 23 is an example of a pointing device, and other pointing devices can also be used. Examples of other pointing devices include a touch panel, a tablet, a touch pad, and a trackball.

[0041] The optical drive device 106 uses a laser beam or the like to read data recorded on an optical disc 24 or write data to the optical disc 24. The optical disc 24 is a portable recording medium on which data is recorded so that it can be read by reflected light. The optical disc 24 includes a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc Read Only Memory), a CD-R (Recordable) / RW (Rewritable), and the like.

[0042] The device connection interface 107 is a communication interface for connecting peripheral devices to the machine learning system 100. For example, a memory device 25 or a memory reader / writer 26 can be connected to the device connection interface 107. The memory device 25 is a recording medium equipped with a function for communicating with the device connection interface 107. The memory reader / writer 26 is a device for writing data to the memory card 27 or reading data from the memory card 27. The memory card 27 is a card-type recording medium.

[0043] The network interface 108 is connected to the network 20. The network interface 108 transmits and receives data to and from other computers or communication devices via the network 20. The network interface 108 is a wired communication interface connected by a cable to a wired communication device such as a switch or a router. The network interface 108 may also be a wireless communication interface connected by radio waves to a wireless communication device such as a base station or an access point.

[0044] The machine learning system 100 can realize the processing functions of the second embodiment by using the hardware described above. Note that the information processing device 10 shown in the first embodiment can also be realized by using the same hardware as the machine learning system 100 shown in FIG.

[0045] The machine learning system 100 realizes the processing functions of the second embodiment by executing a program recorded on, for example, a computer-readable recording medium. The program describing the processing to be executed by the machine learning system 100 can be recorded on various recording media. For example, the program to be executed by the machine learning system 100 can be stored in a storage device 103. The processor 101 loads at least a portion of the program in the storage device 103 into the memory 102 and executes the program. The program to be executed by the machine learning system 100 can also be recorded on a portable recording medium such as an optical disc 24, a memory device 25, or a memory card 27. The program stored on the portable recording medium becomes executable after being installed on the storage device 103, for example, under the control of the processor 101. The processor 101 can also read and execute the program directly from the portable recording medium.

[0046] In the above-described system, the machine learning system 100 automatically extracts features from a data set of a state transition process when generating a machine learning model. The following describes the difficulty of automatically extracting features from a state transition process.

[0047] The performance of a model generated in machine learning depends on what features are extracted from a dataset and used as training data. In many cases, features are created manually by someone with extensive knowledge of the data. To promote the use of machine learning, it is appropriate to be able to automatically generate features from data without relying on human intervention.

[0048] One type of data used in machine learning is called point-process data. Point-process data is data that does not have a set time interval for data acquisition. For example, earthquake data that records the region, time of occurrence, and magnitude is point-process data. In contrast, data acquired at regular intervals, such as hourly precipitation in a certain region, is called time-series data. Point-process data can be treated as time-series data by, for example, re-aggregating it into data at regular intervals.

[0049] Point process data also includes data obtained as a log of a state transition process. Data that can be represented by a state transition process includes, for example, data showing the status of a task using a computer system. For example, a state transition process that shows the status of a task starts at "Started," when the task is accepted, and progresses through states such as "Working," before reaching "Ended." By performing machine learning using data that represents such a state transition process, it is possible to use a trained model to detect, for example, events that require attention in the execution of the task.

[0050] If appropriate features for solving machine learning problems can be extracted from a state transition process, machine learning can be performed with high accuracy. However, the transitions between states in a state transition process can take a wide variety of patterns. Moreover, the words that represent states in a state transition process may have an order relationship in their meaning, but this is not strictly defined.

[0051] Fig. 4 is a diagram showing an example of a dataset showing the execution history of a state transition process. In dataset 31, a record is registered for each state of an executed process. Each record contains information such as the process ID of the corresponding process, the date and time of transition to the corresponding state, and the state after the transition. In the example of Fig. 4, the possible states of a process are five types: "Started," "Working," "Inquiry," "Resolved," and "Ended."

[0052] By sorting records with the same process ID in ascending order based on the date and time, you can see the state transition of the corresponding process. Then, you can see the state at each elapsed day since the process started.

[0053] The elapsed days table 32 shows the status of each process shown in the dataset 31 by the number of days elapsed. The solid line 32a shows the status of the "ID001" process by the number of days elapsed. The dashed line 32b shows the status of the "ID002" process by the number of days elapsed. The dashed line 32c shows the status of the "ID003" process by the number of days elapsed.

[0054] All of the processes shown in Table 32 begin in the "Start" state and end in the "End" state. However, the states through which each process passes to reach the "End" state differ for each process. Also, the number of days elapsed until each state transitions differs for each process, and the number of days elapsed until the "End" state also differs for each process.

[0055] As described above, a process undergoes state transitions as it progresses. The state transitions that a process undergoes vary from process to process, and the sequential relationship between states before and after a transition may contain process characteristics. When using a machine learning model to detect processes that require attention, it is appropriate to create features from dataset 31 that can extract features that can distinguish between processes that have undergone normal state transitions and processes that have undergone abnormal state transitions.

[0056] For example, the process "ID003" contains the transition "Finished → Working → Finished." As in this example, if the "Working" state occurs again despite the process having "Finished" once, the order of the states does not make sense. However, in actual business, it is possible that state transitions that do not make sense may occur.

[0057] Here, the state transitions that a process goes through, rearranged from oldest to newest on the time axis, are called a "path." A continuous partial state transition taken from a path is called a "pattern."

[0058] For example, dataset 31 contains data on processes with process IDs such as "ID001," "ID002," etc. The path of the process with "ID001" is "Start → Working → Inquiry → Resolution → Finish." Examples of patterns that can be extracted from the path of the process with "ID001" are "Start → Working," "Working → Inquiry → Resolution," etc.

[0059] Next, an example of a method for automatically creating feature amounts for each process and its problems will be described with reference to FIGS. FIG. 5 is a diagram showing a first example of a method for automatically creating features for each process. For example, it is conceivable to generate, as features, information that distinguishes between processes of paths that occur infrequently and other processes. A frequency summary table 34 is obtained by aggregating state transition paths for each process ID from a data set 33. In the frequency summary table 34, the occurrence frequency (number of times) of the process of the path is set in association with the path.

[0060] For example, the process path for "ID001" is "Start → Working → Resolved → Finished." Therefore, "1" is added to the occurrence frequency value of the path "Start → Working → Resolved → Finished" in the frequency summary table 34 in accordance with the process path for "ID001." The process path for "ID002" is "Start → Working → Inquiry → Resolved → Finished." Therefore, "1" is added to the occurrence frequency value of the path "Start → Working → Inquiry → Resolved → Finished" in the frequency summary table 34 in accordance with the process path for "ID002." The process path for "ID003" is "Start → Finished." Therefore, "1" is added to the occurrence frequency value of the path "Start → Finished" in the frequency summary table 34 in accordance with the process path for "ID003."

[0061] When referring to the frequency summary table 34, there may be paths that occur extremely rarely. In the example of Figure 5, the occurrence frequency of the "start → finish" path is "twice," which is an exceptional occurrence frequency. For each process, information on whether the path occurs exceptionally frequently can be used as a feature.

[0062] For example, in the feature information 35, information on whether the frequency of a path is exceptional or not is set in association with a process ID. If the path is exceptional, "True" is set, and if the path is not exceptional, "False" is set. In the example of FIG. 5, since the frequency of the path of the process "ID003" is exceptional, the path frequency of that process is set to exceptional "True."

[0063] In this way, if the information on whether the frequency of a path is exceptional or not is used as a feature, it is not possible to distinguish between a process of a path that is a normal state transition but occurs infrequently and a process of an abnormal state transition.

[0064] 6 is a diagram showing an example of a problem that occurs when information on whether the frequency of a path is exceptional is used as a feature. In the frequency summary table 36, the occurrence frequency of the path "Start → Finish → Work → Finish" is "2 times," and the occurrence frequency of the path "Start → Working → Inquiry → Working → Inquiry → Working → Inquiry → Working → Solution → Finish" is "1 time."

[0065] In this case, if the information that distinguishes processes with low frequency paths from others is used as a feature, the frequency of the path for a process with a path of "Start → Finish → Work → Finish" will be the exceptional "True." This path does not make sense of the state transition. In other words, when considering the reason for the state transition by taking into account the meaning of the terms that indicate the state, the corresponding state transition cannot be explained under normal circumstances. Therefore, processes with such paths should be taken into consideration.

[0066] Additionally, the path frequency for a process with the path "Start → Working → Inquiry → Working → Inquiry → Working → Inquiry → Working → Solution → End" is also an exceptional "True" result. This path occurs less frequently due to the large number of state transitions, but the meaning of the state transitions remains consistent throughout. In other words, this path is a path that can occur within the scope of normal work.

[0067] If the information on whether the frequency of a path is exceptional is used as a feature, then a process with a path whose state transition makes sense will have the same feature as a process with a path whose state transition makes no sense. Therefore, it is not suitable as a feature for machine learning to find processes with abnormal transitions (paths whose state transition makes no sense).

[0068] Furthermore, the more transitions there are, the less frequently the same path occurs. However, a large number of transitions does not necessarily mean that the information is meaningless. Therefore, if all such paths are used as features, the number of features increases. As the number of features increases, it takes longer to generate a model in machine learning and to make inferences using the model. Furthermore, if there are too many features, the accuracy of the generated model decreases due to overfitting and other factors.

[0069] FIG. 7 is a diagram showing a second example of a method for automatically creating features for each process. When all possible state transition patterns are enumerated from the state values ​​present in the dataset 37, each pattern becomes a column of feature information 38. In feature information 38, information on whether a pattern set in an item is included in a path is set as a feature of the process, in association with the process ID. For example, for a process whose path includes a state transition of the pattern set in the item, "True" is set in the corresponding item. For a process whose path does not include a state transition of the pattern set in the item, "False" is set in the corresponding item.

[0070] FIG. 8 is a diagram showing an example of a problem that occurs when information indicating whether a path passes through a pattern is used as a feature. When information indicating whether a path passes through a pattern is used as a feature, the number of feature items set in the feature information 38 increases as the number of states increases. If there are too many features, there is a high possibility that overlearning will occur in model generation in machine learning. Furthermore, if there are too many features, the processing load for model generation in machine learning will also be excessive.

[0071] Thus, in machine learning using a dataset related to state transition processes, it is inappropriate to use either information about whether the frequency of a path is exceptional or information about whether a path passes through a pattern as a feature. Furthermore, it is difficult to determine unambiguously what information should be used as a feature.

[0072] Therefore, in the machine learning system 100 according to the second embodiment, the features of a dataset are interpreted using large-scale language models (LLMs) and useful information is determined from the dataset as feature quantities. In the following description, the LLMs include language models trained by deep learning using neural networks and inference functions using the language models.

[0073] 9 is a block diagram showing an example of a function for performing machine learning on a state transition process. The business system 200 has a business management unit 210 and a storage unit 220. The business management unit 210 manages the performance of business using business terminals 30a, 30b, ..., and records information indicating the state of the business process in log data 221 each time the state changes. The storage unit 220 stores the log data 221.

[0074] The management terminal 300 includes a machine learning request unit 310. The machine learning request unit 310 sends a machine learning request using log data 221 to the machine learning system 100 in accordance with instructions from a user who is an administrator of the machine learning request unit. For example, the machine learning request unit 310 acquires at least a portion of the log data 221 of the business system 200 and generates a dataset 121 for machine learning. Then, the machine learning request unit 310 sends a machine learning request including the dataset 121 to the machine learning system 100. The machine learning request may be, for example, an instruction to generate a model or an instruction to make inference based on the model.

[0075] The machine learning system 100 includes a machine learning request receiving unit 110, a storage unit 120, a feature determining unit 130, an LLM 140, a data feature generating unit 150, and a machine learning unit 160. The machine learning request receiving unit 110 receives a machine learning request from the management terminal 300. The machine learning request receiving unit 110 stores a dataset 121 included in the machine learning request in the storage unit 120. The machine learning request receiving unit 110 also sends a feature determination request based on the dataset 121 to the feature determination unit 130. When the machine learning request receiving unit 110 receives a machine learning processing result from the machine learning unit 160, it sends the received processing result to the management terminal 300.

[0076] The storage unit 120 stores a dataset 121, feature quantity determination criterion information 122, and a model 123. The dataset 121 is information that records the state transitions of processes within a predetermined period. The feature quantity determination criterion information 122 is information that indicates what state transition information is to be used as a feature among the state transitions of each process shown in the dataset 121. The model 123 is a machine learning model generated by learning based on the dataset 121.

[0077] The feature determination unit 130 determines information to be extracted as features from the dataset 121 in accordance with the feature determination request. For example, the feature determination unit 130 interacts with the LLM 140 using sentences in natural language to generate feature determination criterion information 122 indicating criteria for determining the features. The feature determination unit 130 stores the generated feature determination criterion information 122 in the storage unit 120.

[0078] When the generation of the feature determination criterion information 122 is completed, the feature determination unit 130 instructs the data feature generation unit 150 to generate features. Note that if a trained model 123 already exists and feature determination criterion information 122 for the data set 121 to be used for that model has already been generated, the feature determination unit 130 instructs the data feature generation unit 150 to generate features without generating new feature determination criterion information 122.

[0079] The LLM 140 interprets the content of a sentence represented by a character string and generates a sentence as a response to a query based on that sentence. For example, the LLM 140 interprets the state transitions of each process represented in the dataset 121 and identifies a state transition pattern that represents a process characteristic that may be the cause of a problem. The LLM 140 then transmits the identified pattern and the reason for identifying the pattern to the feature determination unit 130. The LLM 140 is an example of the dialogue system 2 shown in FIG. 1.

[0080] In response to the feature generation instruction, the data feature generation unit 150 extracts process features from the data set 121. The data feature generation unit 150 determines which state transition pattern to use as a feature according to feature determination criterion information 122. The data feature generation unit 150 transmits feature information indicating the extracted features to the machine learning unit 160.

[0081] When a machine learning request specifies that a model be generated, the machine learning unit 160 generates a machine learning model 123 based on the feature amount information. The machine learning unit 160 stores the generated model 123 in the storage unit 120.

[0082] Furthermore, when an inference process is instructed in the machine learning request, the machine learning unit 160 executes the inference process using the feature information as input to the model 123. For example, the machine learning unit 160 detects processes that require attention among the processes indicated in the dataset 121. The machine learning unit 160 transmits the results of the machine learning to the machine learning request receiving unit 110.

[0083] The function of each element shown in FIG. 9 can be realized by, for example, having the processor 101 execute a program module corresponding to that element. This system can generate machine learning models for state transition processes and perform inference processing using those models. The type of information to extract as features from the state transition process is determined through interaction with the LLM140.

[0084] For example, the feature determination unit 130 causes the LLM 140 to list meaningful state transition patterns and the reasons for them. Then, the feature determination unit 130 determines, as the feature of each process, information indicating whether the patterns listed by the LLM 140 are included.

[0085] FIG. 10 is a diagram illustrating an example of a method for generating feature information used in machine learning targeting a state transition process. For example, the feature determination unit 130 inputs a character string of a question 41 to the LLM 140. The question 41 indicates, for example, a dataset 121 and includes a sentence such as, "Show me the transition that does not make sense and the reason why." In this case, the feature determination unit 130 may include, in the question 41, an explanation of the dataset 121, a description of the machine learning problem to be solved, and statistics and metadata obtained from the dataset 121. For example, the feature determination unit 130 may input the following sentence in " " to the LLM 140: "This dataset records the lifecycle of inquiries that occur in a call center. - Use machine learning to predict the expected time it will take to resolve future inquiries based on the current dataset. This dataset has five possible states: Started, Working, Inquiry, Solved, and Finished. The most common path in this dataset is: Start → Working → Solved → Finished. The path with the longest time required from start to finish in this dataset is shown on the right: "Start → Working → Inquiry → Working → Inquiry → Working → Solution → Finish." -Show the transitions that do not make sense as transitions between these states and the reasons why. The LLM 140 interprets the sentence shown in the question sentence 41 and generates an answer sentence 42. The answer sentence 42 indicates any meaningless patterns and the reasons for them. The LLM 140 transmits the generated answer sentence 42 to the feature determination unit 130. The answer sentence 42 includes meaningless patterns such as "start → end" and "start → reception → end."

[0086] The feature determination unit 130 generates feature determination criterion information 122a that indicates a pattern that does not make sense in the answer sentence 42. The feature determination criterion information 122a indicates a feature determination criterion, such as "Does it go from start to reception?" or "Does it go from start to reception to end?" The feature determination unit 130 transmits the feature determination criterion information 122a to the data feature generation unit 150.

[0087] The data feature quantity generation unit 150 determines a path for each process from the data set 121. Then, for each process, the data feature quantity generation unit 150 determines whether or not each of the feature quantity judgment criteria indicated in the feature quantity judgment criterion information 122a is met. The data feature quantity generation unit 150 generates feature quantity information 43 indicating the judgment result. In the feature quantity information 43, a feature quantity is set in association with the process ID of each process, with the feature quantity being "True" if the criterion is met, and "False" if the criterion is not met. The data feature quantity generation unit 150 transmits the generated feature quantity information 43 to the machine learning unit 160.

[0088] The machine learning unit 160 performs machine learning processing based on the acquired feature amount information 43. For example, the machine learning unit 160 generates a model 123. When the response sentence 42 obtained from the LLM 140 contains multiple meaningless patterns, the feature determiner 130 may cause the LLM 140 to classify the meaningless patterns into categories based on the reasons listed at the same time. Alternatively, the feature determiner 130 may perform clustering based on the sentences indicated by the LLM 140 to classify the meaningless patterns into categories. By classifying the meaningless patterns into categories, it becomes possible to treat patterns with similar listed reasons as a unified feature.

[0089] FIG. 11 shows an example of categorizing meaningless patterns. When a response sentence 42 from the LLM 140 to a question sentence 41 contains multiple patterns, the feature determination unit 130 transmits a pattern categorization instruction 44 to the LLM 140. The LLM 140, in accordance with the instruction, categorizes the meaningless pattern indicated in the response sentence 42 based on the reason for determining that the meaning is meaningless. At this time, the LLM 140 can also be instructed to generate category names for the classified categories. The LLM 140 transmits the classification result 45 to the feature determination unit 130.

[0090] For example, for each category, meaningless patterns included in the category are set in association with the category name in the classification result 45. In the example of Fig. 11, the patterns of "start → finish" and "start → reception → finish" are set for the category "early close".

[0091] The feature amount determination unit 130 generates feature amount determination criterion information 122b based on the classification result 45. For example, the feature amount determination criterion information 122b indicates a feature amount determination criterion such as "whether the process goes from start to finish or from start to reception to finish" as a category of early closure. Then, the feature amount determination unit 130 transmits the generated feature amount determination criterion information 122b to the data feature amount generation unit 150.

[0092] The data feature quantity generation unit 150 determines a path for each process from the dataset 121. Then, for each process, the data feature quantity generation unit 150 determines whether or not each of the feature quantity criteria for each category indicated in the feature quantity judgment criteria information 122b is met. The data feature quantity generation unit 150 generates feature quantity information 46 indicating the judgment results. In the feature quantity information 46, a feature quantity is set for each category in association with the process ID of each process, with the feature quantity being "True" if the criterion for that category is met, and "False" if the criterion is not met. The data feature quantity generation unit 150 transmits the generated feature quantity information 46 to the machine learning unit 160.

[0093] The machine learning unit 160 performs machine learning processing based on the acquired feature amount information 46. For example, the machine learning unit 160 generates a model 123. For example, the machine learning unit 160 generates a model 123 that detects a process requiring attention.

[0094] The feature information 46 is a feature for each category. When the two patterns "start → end" and "start → reception → end" are indicated by the LLM 140, they are treated as a single feature for the "early close" pattern. In other words, it is possible to create a unified feature for state transition patterns in which the meaning of the transition order is not clear for similar reasons. As a result, the number of types of features is reduced compared to when categorization is not performed.

[0095] 12 is a flowchart showing an example of the procedure of the machine learning process. The process shown in FIG. 12 will be explained below in order of step number. [Step S101] The machine learning request receiving unit 110 receives a machine learning request from the management terminal 300. The machine learning request includes a dataset 121. The machine learning request also includes a specification of whether the request is for a learning phase in which a model 123 is generated using the dataset 121, or an inference phase in which prediction or classification is performed using the generated model 123.

[0096] The machine learning request also includes the column name of the column in dataset 121 in which information to be used as a feature of the state transition process is set. The machine learning request also includes information such as a description of dataset 121 and the purpose of the machine learning. The machine learning request receiving unit 110 transmits the acquired machine learning request to the feature determining unit 130.

[0097] [Step S102] The feature determination unit 130 determines whether the feature determination criterion information 122 used to perform machine learning in response to the machine learning request has already been generated. For example, if the model 123 has already been generated, the feature determination unit 130 has also already generated the feature determination criterion information 122 used when generating the model 123. If the feature determination unit 130 has already generated the feature determination criterion information 122, the process proceeds to step S104. If the feature determination unit 130 has not yet generated the feature determination criterion information 122, the process proceeds to step S103.

[0098] [Step S103] The feature amount determining unit 130 executes a feature amount determination criterion generation process, which will be described in detail later (see FIG. 13). [Step S104] The data feature quantity generation unit 150 generates feature quantity information indicating the feature quantities of each process indicated in the data set 121, based on the feature quantity determination criteria indicated in the feature quantity determination criteria information 122. The data feature quantity generation unit 150 transmits the generated feature quantity information to the machine learning unit 160.

[0099] [Step S105] The machine learning unit 160 executes the machine learning process indicated in the machine learning request based on the feature information generated from the dataset 121. For example, when execution of the learning phase is requested, the machine learning unit 160 generates a model 123 based on the feature information. When execution of the inference phase is requested, the machine learning unit 160 uses the feature information as input to the trained model 123 and calculates the output of the model 123.

[0100] The machine learning unit 160 transmits the results of the machine learning process to the machine learning request receiving unit 110. For example, when the machine learning unit 160 generates a new model 123, it transmits information indicating the completion of generation to the machine learning request receiving unit 110 as the processing result. Furthermore, when the machine learning unit 160 executes a prediction or classification process using the model 123, it transmits information indicating the prediction result or classification result to the machine learning request receiving unit 110 as the processing result.

[0101] [Step S106] The machine learning request receiving unit 110 sends the processing result to the management terminal 300. In this way, appropriate features are extracted from the data set 121, and machine learning is performed using the features in the machine learning system 100. Next, the feature determination criterion generation process will be described in detail.

[0102] 13 is a flowchart showing an example of the procedure for generating a feature amount determination criterion. The process shown in FIG. 13 will be described below in order of step number. [Step S201] The feature determiner 130 extracts information to be used in the machine learning process from the machine learning request. For example, the feature determiner 130 extracts information such as the dataset 121 and column names in which information representing process state transitions is set from the machine learning request. For example, the feature determiner 130 extracts the process ID, date and time, and state as column names in which information representing process state transitions is set.

[0103] [Step S202] The feature determining unit 130 executes a process statistic calculation process, the details of which will be described later (see FIG. 14). [Step S203] The feature determination unit 130 presents the machine learning problem and the statistics acquired in step S202 to the LLM 140.

[0104] [Step S204] The feature determination unit 130 uses a question to instruct the LLM 140 to list patterns whose order relationships do not make sense and to explain the reason for this determination. The LLM 140 then responds by listing the patterns that do not make sense and the reasons why.

[0105] [Step S205] The feature determination unit 130 determines whether there are multiple meaningless patterns. If there are multiple meaningless patterns, the feature determination unit 130 proceeds to step S206. If there is only one meaningless pattern, the feature determination unit 130 proceeds to step S207.

[0106] [Step S206] The feature determination unit 130 instructs the LLM 140 to perform category classification. The feature determination unit 130 then obtains the classification result from the LLM 140. <Step S207> The feature determining unit 130 generates feature determination criterion information and stores the generated feature determination criterion information in the storage unit 120.

[0107] In this way, feature value determination criterion information indicating feature value determination criteria for extracting whether or not an incomprehensible pattern is included as a feature value is generated using the LLM 140. Next, the process statistics calculation process will be described in detail.

[0108] 14 is a flowchart showing an example of the procedure for the process statistics calculation process. The process shown in FIG. 14 will be described below in order of step number. [Step S301] The feature determining unit 130 acquires possible states as one of the statistical values. For example, if the character string set in the state column in the data set 121 is "Started," "In Progress," "Inquiry," "Resolved," or "Ended," the feature determining unit 130 determines these five states as possible states.

[0109] [Step S302] The feature determination unit 130 tallies the occurrence frequency of each state transition path. For example, the feature determination unit 130 determines the state transition path for each process shown in the data set 121. Then, for each path that occurs, the feature determination unit 130 counts the number of processes corresponding to that path and sets this as the occurrence frequency of that path.

[0110] [Step S303] The feature determination unit 130 acquires the path with the highest occurrence frequency as one of the statistics. The pattern included in the path with the highest occurrence frequency is expected to be a semantically understandable pattern. Therefore, by presenting the path with the highest occurrence frequency to the LLM 140, the LLM 140 can more easily correctly determine a semantically understandable pattern. For example, if the path with the highest occurrence frequency is "Start → Working → Inquiry → Solution → End," the LLM 140, upon being presented with this path, can more easily determine that patterns such as "Start → Working → Inquiry" and "Working → Inquiry" are semantically understandable.

[0111] [Step S304] The feature determiner 130 tallies the occurrence frequency for each state transition pattern. For example, the feature determiner 130 finds all state transition patterns for each process in the data set 121. Then, for each pattern obtained, the feature determiner 130 counts the number of processes in the path that includes that pattern, and sets this as the occurrence frequency of that pattern.

[0112] [Step S305] The feature determination unit 130 acquires the most frequently occurring pattern as one of the statistics. The most frequently occurring pattern is expected to be a meaningful pattern. Therefore, by presenting the most frequently occurring pattern to the LLM 140, the LLM 140 can more easily correctly determine the meaningful pattern.

[0113] [Step S306] The feature determination unit 130 examines the first and last recorded states for each process and counts the number of times each state is recorded as the first or last state of the process. For example, for each possible state, the frequency with which it is recorded as the first state in 100 processes is tallied, such as "Started: 90 times, Working: 5 times, Inquiry: 3 times, Resolved: 1 time, Finished: 1 time." Similarly, the frequency with which each possible state is recorded as the last state of the process is tallied.

[0114] [Step S307] The feature determiner 130 acquires the most frequently occurring state at the beginning or end of the process as one of the statistics. By presenting the most frequently occurring state at the beginning or end of the process to the LLM 140, the LLM 140 can more easily determine whether the transition patterns to or from those states are meaningful.

[0115] For example, a pattern of transitions from the most frequently occurring state at the beginning of a process to another state is likely to make sense, but a pattern of transitions from the most frequently occurring state at the end of a process to another state is likely to make no sense.Furthermore, a pattern of transitions to the most frequently occurring state at the beginning of a process is likely to make no sense, but a pattern of transitions to the most frequently occurring state at the end of a process is likely to make sense.

[0116] [Step S308] The feature determining unit 130 counts the number of state transitions for each process. [Step S309] The feature determiner 130 identifies the process with the largest number of transitions and obtains the occurrence frequency of each state of that process as one of the statistics. For example, assume that the path of the process with the most state transitions is "Start → Working → Inquiry → Working → Inquiry → Working → Inquiry → Working → Solved → Finished." In this case, the feature determiner 130 obtains "Start: 1 time, Working: 4 times, Inquiry: 3 times, Solved: 1 time, Finished: 1 time."

[0117] By presenting the occurrence frequency of each state of the process with the largest number of transitions to the LLM 140, the LLM 140 can use the presented statistics as a basis for determining whether or not the presented statistics are meaningful even when the number of transitions is large.

[0118] [Step S310] For each pair of a pre-state and post-state of a transition, the feature determiner 130 acquires the occurrence frequency of the state transition as one of the statistics. This allows the feature determiner 130 to obtain, for example, information that "the most frequently occurring state to which a solution state transition occurs is the end." The feature determiner 130 then presents this statistical value to the LLM 140. Based on the presented statistical value, the LLM 140 can determine that the state transition from the solution state to the end state is likely to be a meaningful pattern.

[0119] [Step S311] The feature determining unit 130 tallies the time required for each process. The time required for a process is the difference between the earliest and latest dates and times when the process ID of the process was assigned.

[0120] [Step S312] The feature determination unit 130 acquires the path of the process with the longest required time as one of the statistics. By presenting the path of the process with the longest required time to the LLM 140, the LLM 140 can use the presented statistics as information for determining whether the process is meaningful even when the required time is long.

[0121] Next, the feature amount determination process for the data set 121 will be specifically described with reference to FIGS. 15 is a diagram showing an example of a question sentence in response to a feature determination request. For example, feature determination request 50 indicates that the dataset is a "call center inquiry log" and that the columns that represent the state transition of a process in that dataset are "ID, timestamp, state."

[0122] The feature determination unit 130 generates a question 51 for the LLM 140 based on the feature determination request 50. The question 51 includes the following sentence in " ". "We manage call center operations. Inquiries that come into the call center go through several states before finally being completed. To improve the efficiency of this operation, we record logs. The log data has three columns, each with the following items: ---From here ID: A unique ID for each query timestamp: The date and time the log was recorded state: The state when the log was recorded. It takes the following values ​​(in alphabetical order). Accepted, resolved, answered, completed, in progress, inquiry, occurred ---to this point When doing so, please take into consideration the meaning of the words in each state and list the patterns that you should look out for carefully when managing them, and why. For example, the paths below are the most frequently occurring paths in the current dataset: Occurrence → Received → Working → Answered → Resolved → Completed ... Answer format: State 1 → State 2 → State 3 → ...' The feature determination unit 130 transmits the query 51 to the LLM 140. Having acquired the query 51, the LLM 140 recognizes that the dataset is a call center inquiry log and that the work of responding to inquiries at the call center is a process, and that the state of that process is recorded in the dataset. The LLM 140 then extracts state transition patterns in accordance with the instruction, "Please list patterns that require careful attention in management, taking into account the meaning of the words in each state, and the reasons for doing so." For example, the LLM 140 extracts state transition patterns in which the meaning of the words before and after the transition is incorrect, or in which attention should be paid to the meaning of the words even if the meaning of the words is correct.

[0123] 16 is a diagram showing an example of an answer sentence. The LLM 140 receives the dataset 121a and the question sentence 51 as input, and outputs an answer sentence 52. The answer sentence 52 from the LLM 140 includes the following sentence in " ".

[0124] "1. Occurrence → Received → Completed: In this pattern, the inquiry is completed immediately after it is generated, omitting steps such as work in progress, response, and resolution. This may occur when the inquiry is simple and can be resolved immediately, or when the inquiry is generated by mistake. However, if this pattern occurs frequently, it may be that staff are trying to end the inquiry early without responding appropriately. Therefore, the frequency and content of this pattern need to be investigated in detail."

[0125] 2. Reception → Working → Inquiry: This pattern indicates that the staff member needs additional information while handling the inquiry. This occurs when the inquiry is complex or the staff member does not have sufficient knowledge. If this pattern occurs frequently, you may need to review your staff training and information sharing methods.

[0126] 3. Working → Answered → Working: This pattern indicates that the staff member answered the question once, but then returned to the Working state. This occurs when the staff member was unable to resolve the issue the first time they answered, or when the customer has additional inquiries. If this pattern occurs frequently, you may need to review the staff member's problem-solving ability and the quality of customer service.

[0127] 4. Solution → Inquiry: This pattern indicates that an inquiry occurs again after the problem has been resolved. This occurs when the customer does not understand the solution or the solution is not effective. If this pattern occurs frequently, you may need to reconsider how you explain the solution and how you confirm its effectiveness.

[0128] 5. Reception → Inquiry: This pattern indicates that an additional inquiry occurred immediately after an inquiry was accepted. This occurs when the customer initially provided insufficient information or when the staff was unable to fully understand the content of the inquiry. If this pattern occurs frequently, you should review your inquiry acceptance method and your staff's ability to understand inquiries.

[0129] 6. Working → Ask: This pattern indicates that a staff member needed additional information while handling an inquiry. This occurs when the inquiry is complex or the staff member lacks sufficient knowledge. If this pattern occurs frequently, you may need to reassess your staff training and information sharing methods.

[0130] 7. Answer → Inquiry: This pattern indicates that additional inquiries occurred after a staff member answered. This occurs when the customer did not understand the answer provided or the answer did not solve the problem. If this pattern occurs frequently, the quality of the staff's answers and problem-solving ability should be reviewed.

[0131] 8. Resolved → Working: This pattern indicates that a problem appears to have been resolved, but then reverts to a working state. This occurs when the solution is ineffective or the customer discovers additional issues. If this pattern occurs frequently, you may need to review your methods for verifying the effectiveness of solutions and your staff's problem-solving capabilities. Response 52 lists eight patterns. For each pattern, the reason for identifying that pattern as something to look at carefully is explained. For example, the first pattern, "Occurrence → Received → Completed," is identified because it omits steps such as "Work in Progress," "Response," and "Resolution."

[0132] In addition, the suspected cause of the identified pattern is shown in response sentence 52. For example, for the first pattern, "occurrence → reception → completion," if the inquiry is simple and can be resolved immediately, or if the inquiry is incorrect, this is the suspected cause of the problem.

[0133] Furthermore, response 52 indicates points to be aware of and countermeasures for the identified patterns. For example, for the first pattern, "occurrence → reception → completion," it is pointed out that there is a possibility that staff members are not responding appropriately, and as a countermeasure, it is suggested that a detailed investigation be conducted into the frequency and content of the occurrence of that pattern.

[0134] A plurality of patterns are shown in the answer sentence 52. Therefore, the feature determining unit 130 that has acquired the answer sentence 52 transmits a category classification instruction to the LLM 140. 17 is a diagram showing an example of a category classification instruction 53. The following sentence in " " is written in the category classification instruction 53.

[0135] "Regarding the eight patterns I just listed, please divide them into multiple categories based on the meaning of the state transitions, and give them names based on their meaning." Upon receiving the category classification instruction 53, the LLM 140 performs category classification of the patterns contained in the answer sentence 52.

[0136] 18 is a diagram showing an example of the classification result. The classification result 54 includes the following sentences in " ". 1. Early Completion Pattern: Occurrence → Acceptance → Completion This pattern can be seen when inquiries are resolved quickly or occur accidentally, but if it occurs frequently, it may be a sign of a problem with staff response.

[0137] 2. Additional information request pattern: Reception → Working → Inquiry, Working → Inquiry This pattern occurs when staff need additional information while handling an inquiry, and if it occurs frequently, it may indicate an issue with staff training or information sharing practices.

[0138] 3. Reaction pattern: Working → Answer → Working, Resolved → Working This pattern is seen when a problem has been answered or resolved once, but then goes back to being in progress. If this occurs frequently, there may be a problem with the staff's problem-solving ability or the quality of customer service.

[0139] 4. Re-inquiry pattern: Solution → Inquiry, Answer → Inquiry This pattern is seen when inquiries are made again after the problem was thought to have been resolved or after a response was given. If this occurs frequently, there may be a problem with the way the solution is explained, the way the effectiveness is confirmed, or the quality of the staff's responses.

[0140] 5. Initial lack of understanding pattern: Reception → Inquiry This pattern is seen when additional inquiries occur immediately after an inquiry is accepted. If this occurs frequently, there may be a problem with the way inquiries are accepted or the staff's ability to understand inquiries. In the classification result 54, for example, the pattern "Reception → Working → Inquiry" and the pattern "Working → Inquiry" are grouped together in one category. This category groups together patterns that are common in terms of the staff member needing additional information.

[0141] A category name is given to each category shown in the classification result 54. For example, the category name of a category that includes the pattern "reception → working → inquiry" and the pattern "working → inquiry" is "additional information request pattern."

[0142] The feature determining unit 130, which has acquired the classification result 54, generates feature determination criterion information. 19 is a diagram showing an example of feature quantity determination criterion information. The feature quantity determination criterion information 55 indicates feature quantity determination criteria for each category indicated in the classification result 54. For example, for the category "Early Completion," it indicates that a feature quantity of "True" is set for processes that include the pattern "Generation → Reception → Completion," and a feature quantity of "False" is set for processes that do not include the corresponding pattern. Based on such feature quantity determination criterion, the feature quantity for each process is determined from the data set 121a, and feature quantity information is generated.

[0143] In this way, through text-based dialogue with LLM140, meaningless state transitions can be automatically detected based on the text describing the process state and used as process features. This prevents paths and patterns that occur infrequently but are meaningful from being used as features. As a result, the number of patterns used as features is prevented from increasing.

[0144] Other Embodiments In the second embodiment, the LLM 140 is located within the machine learning system 100, but the LLM 140 may be located on a cloud computing system separate from the machine learning system 100.

[0145] Although the embodiments have been described above, the configuration of each part shown in the embodiments can be replaced with other parts having similar functions. Also, any other components or processes may be added. Furthermore, any two or more configurations (features) of the above-described embodiments may be combined. [Explanation of symbols]

[0146] 1. Dataset 2. Dialogue System 2a Language Model 3 Questions 4 Answer text 5,6 Feature information 10. Information processing equipment 11 Storage section 12 Processing section

Claims

1. generating a query in natural language instructing extraction of a first state transition that satisfies a predetermined condition regarding the meaning of states before and after the transition from a data set indicating a series of transitions of states of a plurality of events; inputting the question sentence into a dialogue system that conducts dialogue in the natural language using a language model that outputs an answer in the natural language sentence to a question in the natural language sentence; determining, as a feature quantity for each of the plurality of events, information based on whether or not the first state transition indicated in the response sentence output by the dialogue system is included in a series of state transitions for each of the plurality of events; An information processing program that causes a computer to execute a process.

2. In the process of generating the question sentence, the question sentence is generated to instruct output of the answer sentence including an explanation of why the first state transition is determined to satisfy the condition; In the process of determining If a plurality of the first state transitions are indicated in the answer sentence, classifying the plurality of the first state transitions into categories based on the explanation sentence; determining that two or more second state transitions classified into the same category among the first state transitions are to be used as a unified feature quantity; The information processing program according to claim 1.

3. In the categorization process, an instruction indicating categorization of the plurality of first state transitions is input to the dialogue system, and a categorization result output by the dialogue system is obtained.

3. The information processing program according to claim 2.

4. In the process of generating the question, the question is generated to include statistical values ​​regarding a series of transitions of states of the plurality of events indicated in the dataset. The information processing program according to claim 1.

5. In the process of generating the question, the question is generated to instruct extraction of the first state transition that does not make sense as a normal series of transitions of the states of the plurality of events. The information processing program according to claim 1.

6. generating feature information indicating feature amounts of each of the plurality of events indicated in the dataset based on information based on whether the first state transition indicated in the response sentence output by the dialogue system is included in a series of state transitions of each of the plurality of events; 2. The information processing program according to claim 1, further causing the computer to execute a process.

7. generating a query in natural language instructing extraction of a first state transition that satisfies a predetermined condition regarding the meaning of states before and after the transition from a data set indicating a series of transitions of states of a plurality of events; inputting the question sentence into a dialogue system that conducts dialogue in the natural language using a language model that outputs an answer in the natural language sentence to a question in the natural language sentence; determining, as a feature quantity for each of the plurality of events, information based on whether or not the first state transition indicated in the response sentence output by the dialogue system is included in a series of state transitions for each of the plurality of events; An information processing method in which processing is performed by a computer.

8. a processing unit that generates a question sentence in a natural language instructing extraction of a first state transition that satisfies a predetermined condition regarding the meaning of states before and after the transition from a data set in which a series of transitions of states of a plurality of events are indicated, inputs the question sentence to a dialogue system that conducts a dialogue in the natural language using a language model that outputs an answer in the natural language sentence to the question in the natural language sentence, and determines, as a feature quantity for each of the plurality of events, information based on whether the first state transition indicated in the answer sentence output by the dialogue system is included in the series of transitions of states of each of the plurality of events; An information processing device having the above.

Images