Task processing system, control method, and control program

Abstract

[Problem] To provide a task processing system capable of solving various tasks, while reducing costs and preventing a decrease in accuracy, on the basis of various kinds of multi-modal data. [Solution] A task processing system comprising: a task reception unit that receives a task from a user; a language model that outputs an answer for the task; a determination unit that determines, on the basis of the task, a type of data to be extracted from stored data in preprocessing; a preprocessing unit that performs preprocessing for extracting data of the determined type from a data storage unit; and a multi-modal data integration unit that integrates extracted data and converts the extracted data such that the converted data can be input to the language model, wherein the language model outputs an answer to the task using the converted data converted by the multi-modal data integration unit.

Description

Task processing system, control method, and control program

[0001] The present invention relates to a task processing system, a control method, and a control program.

[0002] Fundamental models such as large-scale language models are trained on huge datasets and are therefore capable of solving a wide range of tasks in a generalized manner.

[0003] However, in manufacturing and healthcare settings, analysis and other tasks are performed based on a wide variety of multimodal data that is specific to each site over a relatively long period of time, making it difficult to utilize existing platform models. This is because each site uses specialized sensors, different lighting conditions, and unique technical terms, and there are no platform models that can handle such site-specific multimodal data. Furthermore, attempting to process the long-term and diverse data obtained at sites using existing platform models requires memory input, which increases costs, and the data may contain noise and unnecessary information, resulting in a decrease in specifications.

[0004] The following prior art is disclosed in Patent Document 1 below. The system includes a knowledge base module that stores task attributes and multiple task models corresponding to the task attributes, and a task processing device. The task processing device determines whether the inference task corresponding to an acquired input sample is unknown. If it is determined to be unknown, an inference model for the unknown inference task is generated based on the task attributes and the corresponding task model stored in the knowledge base module, and inference is performed on the input sample using the generated inference model.

[0005] Non-Patent Document 1 discloses the following prior art: Information related to a natural language processing task is acquired from an external database, and an answer to the natural language processing task is generated based on the acquired information.

[0006] Special Publication No. 2023-548201

[0007] Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel, Douwe Kiela, Retrieval-Augmented Generation forKnowledge-Intensive NLP Tasks, NeurIPS 2020,<https: / / proceedings.neurips.cc / paper / 2020 / file / 6b493230205f780e1bc26945df7481e5-Paper.pdf>

[0008] However, the prior art disclosed in Patent Literature 1 requires a machine learning model corresponding to each task, which makes the management of the machine learning models cumbersome. Furthermore, the prior art disclosed in Patent Literature 1 is inefficient due to its inability to perform multi-task learning, which tends to result in poor generalization performance for the model. The prior art disclosed in Non-Patent Literature 1 has a problem in that the database information required to solve a task is limited to natural language, making it incapable of handling tasks such as analysis based on multimodal data. Specifically, sentences related to a question are extracted from a database and input into a language model, but multiple types of data cannot be integrated and processed according to the task.

[0009] The present invention has been made to solve the above-mentioned problems, and aims to provide a task processing system, a control method, and a control program that are capable of solving various tasks based on various types of multimodal data, while reducing costs and preventing a decrease in accuracy.

[0010] The above-mentioned problems of the present invention are solved by the following means.

[0011] (1) A task processing system comprising: a task receiving unit that receives a task from a user; a language model that outputs an answer to the received task; a determination unit that determines, based on the received task, the type of data to be extracted in preprocessing from stored data stored in a data storage unit; a preprocessing unit that performs the preprocessing to extract data of the determined data type from the data storage unit; and a multimodal data integration unit that integrates the extracted data and converts it so that it can be input to the language model, wherein the language model outputs an answer to the task using the converted data converted by the multimodal data integration unit.

[0012] (2) The task processing system described in (1) above, wherein the determination unit further determines a specific processing to be performed on the extracted data extracted from the stored data, and the preprocessing unit extracts data of the determined data type from the data storage unit in the preprocessing, and performs the determined specific processing on the extracted data.

[0013] (3) The task processing system according to (2) above, wherein the preprocessing unit performs the specific processing on the extracted data extracted from the stored data.

[0014] (4) The task processing system according to (1) above, wherein the preprocessing unit encodes the stored data that has undergone the preprocessing into a multidimensional vector and outputs the vector to the multimodal data integration unit.

[0015] (5) The task processing system described in (1) above, wherein the data storage unit stores at least one of the measurement values ​​of various sensors installed at a specified site, various recognition results obtained by inputting the measurement values ​​of the sensors into a first machine learning model, and specialized knowledge accumulated at the site, log information obtained from site equipment, and records written by workers, each associated with time information.

[0016] (6) The task processing system described in (2) above, wherein the determination unit determines at least one of the type of data and the specific processing obtained by inputting a language vector into which the sentence of the task accepted by the task accepting unit is converted into a second machine learning model of a neural network.

[0017] (7) The task processing system according to (2) above, wherein the specific processing includes a process of sampling the data at a predetermined sampling period.

[0018] (8) The task processing system described in (6) above, wherein the second machine learning model is one of a Transformer model, a recurrent neural network model, and a gated recurrent neural network model, and the determination unit determines at least one of the type of data and the specific process from the task sentence converted into the language vector by the second machine learning model.

[0019] (9) The task processing system described in (6) above, wherein the output layer of the second machine learning model is a classification model or a regression model.

[0020] (10) The task processing system described in (4) above, wherein the preprocessing unit encodes the preprocessed stored data into the multidimensional vector by converting each of the preprocessed types of stored data into a language vector in a single language vector space.

[0021] (11) The task processing system described in (4) above, wherein the preprocessing unit encodes the preprocessed stored data into the multidimensional vector by converting the preprocessed stored data into a vector space that differs for each data.

[0022] (12) The task processing system described in (1) above, wherein the multimodal data integration unit is a third machine learning model that extracts information useful for solving the task from the preprocessed stored data based on the task accepted by the task accepting unit, integrates it into a multidimensional vector, and converts it so that it can be input to the language model.

[0023] (13) The task processing system described in (12) above, further comprising a combination processing unit that generates first combined data by combining the multidimensional vector output from the multimodal data integration unit with a language vector obtained by converting the sentence of the task, and the language model outputs an answer for the task based on the first combined data.

[0024] (14) The task processing system described in (12) above further includes a combination processing unit that decodes the multidimensional vector output from the multimodal data integration unit into a sentence, combines it with the sentence of the task, and generates second combination data that has been converted so that it can be input to the language model, and the language model outputs an answer for the task based on the second combination data.

[0025] (15) A task processing system comprising: a task receiving unit that receives a task from a user; a determination unit that determines, based on the received task, the type of data to be extracted in preprocessing from stored data stored in a data storage unit; a preprocessing unit that performs the preprocessing to extract data of the determined data type from the data storage unit; and a multimodal data integration unit that integrates the extracted data and converts it so that it can be input to a language model that outputs an answer to the task, wherein the language model uses the converted data converted by the multimodal data integration unit to output an answer to the task.

[0026] (16) A control method comprising: a step (a) of accepting a task from a user; a step (b) of deciding, based on the task accepted in step (a), by a decision unit that decides, based on the accepted task, the type of data to be extracted in preprocessing from stored data stored in a data storage unit; a step (c) of performing preprocessing to extract data of the type of data decided in step (b) from the data storage unit; a step (d) of integrating the extracted data extracted in step (c) and converting it by a multimodal data integration unit so that it can be input to a language model that outputs an answer to the task; and a step (e) of outputting an answer to the task by the language model using the converted data converted in step (d).

[0027] (17) A control program for causing a computer to execute a process comprising: (a) receiving a task from a user; (b) determining, based on the task received in (a), by a determination unit that determines, based on the received task, the type of data to be extracted in preprocessing from stored data stored in a data storage unit; (c) performing the preprocessing to extract data of the type of data determined in (b) from the data storage unit; (d) integrating the extracted data extracted in (c) and converting it by a multimodal data integration unit so that it can be input to a language model that outputs an answer to the task; and (e) using the converted data converted in (d) to output an answer to the task by the language model.

[0028] Based on the accepted task, the system determines the type of data to be extracted from the stored data in preprocessing. Then, the system extracts the determined type of data from the stored data, integrates the extracted data, converts it so that it can be input to a language model, and uses the converted data to output an answer to the task using the language model. This makes it possible to solve a variety of tasks based on diverse types of multimodal data, at low cost and while preventing a decrease in accuracy.

[0029] Advantages and features provided by one or more embodiments of the present invention will be more fully understood from the following detailed description and the accompanying drawings, which are for illustrative purposes only and are not intended to limit the present invention.

[0023] FIG. 1 is a diagram showing a schematic configuration of a task processing system.

[0024] FIG. 2 is a block diagram showing a hardware configuration of a task processing system.

[0025] FIG. 3 is an explanatory diagram for explaining an example of encoding into a multidimensional vector by an input data preprocessing unit.

[0026] FIG. 4 is an explanatory diagram for explaining another example of encoding into a multidimensional vector by an input data preprocessing unit.

[0027] FIG. 5 is an explanatory diagram for explaining an example of identification processing by an input data preprocessing unit and processing by a multimodal data integration neural network.

[0028] FIG. 6 is an explanatory diagram for explaining an identification processing by an input data preprocessing unit and another example of processing by a multimodal data integration neural network.

[0029] FIG. 7 is an explanatory diagram for explaining a combination processing unit that combines a multidimensional vector, which is converted data output from a multimodal data integration neural network, with a language vector obtained by converting a sentence of a task.

[0029] FIG. 8 is a flowchart showing the operation of the task processing system.

[0030] A task processing system, a control method, and a control program according to an embodiment of the present invention will be described below with reference to the accompanying drawings. However, the scope of the present invention is not limited to the disclosed embodiment. In the description of the drawings, the same elements are designated by the same reference numerals, and duplicated descriptions will be omitted. Furthermore, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0031] 1 is a diagram showing a schematic configuration of a task processing system 1. FIG. 2 is a block diagram showing a hardware configuration of the task processing system 1.

[0032] As shown in FIG. 1, the task processing system 1 includes a task receiving unit 110, an input data determination unit 120, an input data preprocessing unit 130, a multimodal data integration neural network 140, a large-scale language model 150, an answer output unit 160, and a memory unit 200. As shown in FIG. 2, the task processing system 1 may be constructed by a computer including a control unit 100, a memory unit 200, a display unit 300, an input unit 400, and a communication unit 500. The functions of the task receiving unit 110, the input data determination unit 120, the input data preprocessing unit 130, the multimodal data integration neural network 140, the large-scale language model 150, and the answer output unit 160 are realized by the control unit 100 executing a program. The memory unit 200 constitutes a data storage unit. The input data determination unit 120 constitutes a determination unit. The large-scale language model constitutes a language model.

[0033] As shown in FIG. 2, the control unit 100, storage unit 200, display unit 300, input unit 400, and communication unit 500 of the task processing system 1 are interconnected via a bus 600.

[0034] The control unit 100 is configured by a CPU (Central Processing Unit) and controls and processes each part of the task processing system 1 in accordance with a program. The function of the control unit 100 will be described later.

[0035] The storage unit 200 may be configured with RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. The RAM temporarily stores programs and data as a working area for the control unit 100. The ROM stores various programs and data in advance. The flash memory stores various programs and data, including the operating system.

[0036] The display unit 300 is, for example, a liquid crystal display, and displays various types of information.

[0037] The input unit 400 is configured with, for example, a touch panel and various keys, and is used for various operations and inputs.

[0038] The communication unit 500 is an interface for communicating with external devices. Network interfaces conforming to standards such as Ethernet (registered trademark), SATA, PCI Express, USB, and IEEE 1394 may be used for communication. Additionally, various local connection interfaces such as wireless communication interfaces such as Bluetooth (registered trademark) and IEEE 802.11 may also be used for communication.

[0039] The function of the control unit 100 will be described with reference to FIG.

[0040] The task receiving unit 110 receives tasks from the user. Specifically, the task receiving unit 110 receives tasks input by the user to the input unit 400. A task may be input by the user as a task sentence, which is a sentence, and may be received by the task receiving unit 110. An example of a task sentence may be, "Is there an abnormality in device X, and what is the cause of the abnormality?"

[0041] The task receiving unit 110 may convert the task sentence into a language vector and output it to the large-scale language model 150 and the input data determining unit 120. The large-scale language model 150 may be a generative model that generates an answer for the task sentence and outputs it as a language vector.

[0042] The input data determination unit 120 determines the type of data to be extracted in preprocessing from the stored data stored in the storage unit 200 based on the task sentence accepted by the task acceptance unit 110. The input data determination unit 120 is configured, for example, by a neural network machine learning model. The machine learning model constituting the input data determination unit 120 constitutes a second machine learning model. Hereinafter, the predetermined data stored in the storage unit 200 will also be simply referred to as "stored data." The input data determination unit 120 may further determine a specific process to be performed on extracted data extracted from the stored data. The process of extracting extracted data from the stored data and the specific process may be included in preprocessing. Preprocessing is a process of optimizing input data to the large-scale language model 150 in accordance with the task sentence, which is used to generate an answer to the task sentence. Hereinafter, the data extracted from the stored data in preprocessing will also be simply referred to as "extracted data."

[0043] The types of data extracted from the stored data as extracted data include, for example, detected behaviors, types and locations of detected objects, sensor measurements, log information from equipment, and accumulated expertise and written records from workers at the site. The types of data extracted from the stored data as extracted data may also include the number and clothing of detected people, people's vital signs, care records, temperature, weather, and news.

[0044] The identification process includes, for example, sampling the extracted data at a predetermined sampling period. In this case, the input data determination unit 120 determines the predetermined sampling period. The input data determination unit 120 may determine the predetermined sampling period for each type of data. The predetermined sampling period can be set to an appropriate value through experiments, taking into account factors such as the accuracy and processing time of the response by the task processing system 1 to the task of the task sentence, and the memory size of the task processing system 1.

[0045] The input data determination unit 120 may input the task sentences received by the task receiving unit 110 into a machine learning model, causing the machine learning model to estimate the type of data to be extracted from the stored data in preprocessing and the specific processing to be performed. The input data determination unit 120 may then determine the type of data to be extracted in preprocessing and the specific processing to be performed in preprocessing based on the estimated data type and specific processing. The machine learning model of the input data determination unit 120 may be a Transformer model. In this case, the input data determination unit 120 may determine at least one of the type of data to be extracted in preprocessing and the specific processing to be performed from the task sentences converted into language vectors using the attention mechanism of the Transformer model.

[0046] The machine learning model of the input data determination unit 120 may be a recurrent neural network model or a gated recurrent neural network model.

[0047] The output layer of the machine learning model that constitutes the input data determination unit 120 may be a classification model or a regression model. For example, when the input data determination unit 120 determines the type of extracted data, the input data determination unit 120 outputs at least one of the classes assigned to each data type. When the input data determination unit 120 determines the sampling period when extracting extracted data, the input data determination unit 120 outputs the value of the sampling period. Note that when the input data determination unit 120 determines the sampling period when sampling the extracted data in a specific process, it may output one of the classes assigned to each value of the sampling period.

[0048] The input data preprocessing unit 130 performs preprocessing to extract stored data of the data type determined by the input data determination unit 120 from the storage unit 200. The stored data may be assigned attribute data indicating the type of the stored data. The input data preprocessing unit 130 may perform preprocessing to extract stored data of the data type determined by the input data determination unit 120 from the storage unit 200 using the attribute data. If the stored data is natural language data, the input data preprocessing unit 130 converts the natural language data into a language vector and compares it with the language vector of the data type determined by the input data determination unit 120. Alternatively, the input data preprocessing unit 130 compares the language vector into which the task sentence input to the task receiving unit 110 is converted with the attribute data indicating the type of the stored data. By using any of these processes, the input data preprocessing unit 130 may determine whether each piece of stored data should be extracted as extracted data. In preprocessing, the input data preprocessing unit 130 may extract data of the data type determined by the input data determination unit 120 from the storage unit 200 and perform identification processing on the extracted data. The time period for extracting the extracted data may be set by the user, and the input data preprocessing unit 130 extracts the extracted data for the set time period.

[0049] The stored data stored in the memory unit 200 may include at least one of: measurements from various sensors installed at a specific site; various recognition results obtained by inputting the sensor measurements into a machine learning model; log information obtained from the equipment; and specialized knowledge accumulated at the site and written records written by workers. These stored data are multimodal data and may be stored in association with time information. The machine learning model that receives the sensor measurements and outputs the various recognition results constitutes a first machine learning model. The stored data may include, for example, detected behavior, the type and location of detected objects, sensor measurements, site knowledge, the number and clothing of detected people, people's vital signs, care records, temperature, weather, and news.

[0050] The input data preprocessing unit 130 may encode the preprocessed stored data into a multidimensional vector and output the encoded data to the multimodal data integration neural network 140. The multidimensional vector may be a vector of two or more dimensions.

[0051] FIG. 3 is an explanatory diagram for explaining an example of encoding into a multidimensional vector by the input data preprocessing unit 130. In FIG.

[0052] In the example of FIG. 3, the stored data is a code, and the stored data is data in a different vector space for each type of data. That is, for example, data on behavior detected by a sensor is data in a vector space that is different from other types of stored data. In FIG. 3, behavior detected by a sensor associated with a time stamp is expressed as a code. That is, the stored data contains the codes "0" and "1" as the detected behavior, which respectively represent "walking" and "sitting."

[0053] As described above, the input data preprocessing unit 130 encodes the preprocessed storage data into a multidimensional vector. As in the example of FIG. 3 , if the storage data is data in a different vector space for each type of data, the input data preprocessing unit 130 may encode the storage data into a multidimensional vector as data in a different vector space for each type of data. That is, the input data preprocessing unit 130 may encode the preprocessed storage data into a multidimensional vector by converting the preprocessed storage data into a different vector space for each type of data. More specifically, in the preprocessing, the input data preprocessing unit 130 may perform a specific process on extracted data extracted from the storage data and encode the extracted data after the specific process into a multidimensional vector. When encoding the storage data into a multidimensional vector as data in a different vector space for each type of data, an identifier capable of distinguishing the vector space of each data may be assigned to each data. The identifier may be included in the multidimensional vector.

[0054] Note that, as in the example of FIG. 3 , when the stored data is a code or the like and not a natural language, the input data preprocessing unit 130 may convert the extracted data extracted from the stored data into a natural language in preprocessing, and then perform a specification process on the converted extracted data. The input data preprocessing unit 130 may then encode the extracted data that has undergone the specification process into a multidimensional vector. More specifically, when the stored data is not a natural language, the input data preprocessing unit 130 may convert the extracted data extracted from the stored data into a natural language in preprocessing, and then convert it into a language vector, and then perform a specification process on the converted extracted data. The input data preprocessing unit 130 may then encode the extracted data that has undergone the specification process into a multidimensional vector.

[0055] The input data preprocessing unit 130 may encode the preprocessed storage data into multidimensional vectors by converting each of the preprocessed storage data of all types into language vectors in a single (common) language vector space.

[0056] FIG. 4 is an explanatory diagram for explaining another example of encoding into a multidimensional vector by the input data preprocessing unit 130. In FIG.

[0057] In the example of Fig. 4, the stored data is in natural language. In Fig. 4, the behavior detected by the sensor is expressed in natural language along with a time stamp. That is, the stored data includes detected behavior data in natural language such as "A walking behavior occurred at c o'clock on b day of a month" and "A sitting behavior occurred at z o'clock on y day of x month."

[0058] The input data preprocessing unit 130 encodes the preprocessed stored data into a multidimensional vector. As shown in Figure 4, if the stored data is in natural language, the input data preprocessing unit 130 may convert extracted data in the natural language extracted from the stored data into a language vector in the preprocessing, perform specific processing on the converted extracted data, and encode the extracted data after the specific processing into a multidimensional vector. In this case, all extracted data may be integrated and converted into a multidimensional vector in a single vector space.

[0059] The multimodal data integration neural network 140 integrates the extracted data and converts it so that it can be input to the large-scale language model 150. In other words, the multimodal data integration neural network 140 integrates the preprocessed stored data and converts it so that it can be input to the large-scale language model 150.

[0060] The multimodal data integration neural network 140 may be a machine learning model. The machine learning model of the multimodal data integration neural network 140 extracts information useful for solving a task from preprocessed stored data based on the task sentence, integrates it into a multidimensional vector, and converts it so that it can be input to the large-scale language model 150. The multimodal data integration neural network 140 may include an internal pooling function to extract information useful for a task based on the task sentence from the multidimensional vector. In this case, the language vector of the task sentence may be input to an intermediate layer of the multimodal data integration neural network 140. As a result, the multimodal data integration neural network 140 extracts information while taking into account the correlation between the feature vector calculated from the multidimensional vector obtained from the input data preprocessing unit 130 and the language vector of the task sentence. The machine learning model of the multimodal data integration neural network 140 constitutes a third machine learning model.

[0061] As described above, the stored data may be codes, each of which is data in a different vector space depending on the type of data (see FIG. 3). In this case, an identifier capable of distinguishing the vector space of each data may be assigned to each data and included in a multidimensional vector. The multimodal data integration neural network 140 may use the identifier included in the multidimensional vector to receive each piece of data through a different channel for each identifier. In this case, for each channel, at least a portion of the data included in the multidimensional vector is processed and integrated by a different neural network, and converted so that it can be input to the large-scale language model 150.

[0062] FIG. 5 is an explanatory diagram for explaining an example of the identification process by the input data preprocessing unit 130 and the process by the multimodal data integration neural network 140.

[0063] As shown in FIG. 5 , the input data preprocessing unit 130 performs a specification process to extract data from stored data (multimodal data) at a predetermined sampling period for each data type. The stored data is represented by symbols similar to those in FIG. 3 , and the stored data is represented as data in a different vector space for each data type. In FIG. 5 , the timing of sampling for each data type in the specification process is indicated by dots on a timing chart of the multimodal log for Person A. Data types shown include behavior, number of people, clothing, temperature, weather, news, behavior summary, and hobbies. The granularity and number of dimensions of the data acquired by sampling over time may differ for each data type. For example, behavioral data has relatively small granularity (high data density) and a small number of dimensions. Hobbies, for example, have relatively large granularity (low data density) and a large number of dimensions. The number of dimensions of the multidimensional vector output from the input data preprocessing unit 130 is the sum of the product of S (the number of samples) for each type of extracted data and D (the number of dimensions) for all types of extracted data.

[0064] Because the granularity of the extracted data after sampling varies depending on the type of data, the layer of the multimodal data integration neural network 140 to which the extracted data is input can be changed for each data type depending on the granularity. As shown in FIG. 5 , in a multidimensional vector, data with small granularity and a small number of dimensions can be input to a layer relatively closer to the input of the multimodal data integration neural network 140. In a multidimensional vector, data with large granularity and a large number of dimensions can be input to a layer relatively closer to the output of the multimodal data integration neural network 140. This can improve the computational efficiency of the multimodal data integration neural network 140 and reduce the memory size required for computation. Furthermore, the number of parameters in the multimodal data integration neural network 140 can be reduced, thereby improving learning efficiency.

[0065] 5, the language vector of the task sentence is input to the intermediate layer of the multimodal data integration neural network 140. An example of the task sentence is "Based on the log related to A, please list some points to improve QOL." In addition, an example of a response to the task sentence is output from the large-scale language model 150, which is "It seems that getting up early and doing your hobby of knitting has a positive effect on sleep. Try making it a habit."

[0066] FIG. 6 is an explanatory diagram for explaining another example of the identification process by the input data preprocessing unit 130 and the process by the multimodal data integration neural network 140. In FIG.

[0067] As shown in FIG. 6 , the input data preprocessing unit 130 performs a specification process to sample extracted data from stored data, which is multimodal data, at a predetermined sampling period for each data type. The stored data is represented by symbols, as in FIG. 3 , and is data in a vector space that differs for each data type. The stored data may be natural language, as in FIG. 4 . The stored data may include both symbols and natural language. In FIG. 6 , the timing of sampling for each data type in the specification process is indicated by dots on a timing chart of the multimodal log for Person A. Examples of data types include behavior, number of people, clothing, temperature, weather, news, behavior summary, and hobbies. For example, assume that the extracted data for "behavior" includes the symbols "2" and "1," which represent "wake up" and "walk," respectively, and are assigned timestamps of "March 13th, 7:00 AM" and "March 13th, 10:00 AM." In this case, the input data preprocessing unit 130 converts the extracted data into sentences such as "March 13th, 7:00 AM: wake up" and "March 13th, 10:00 AM: walk," respectively. Then, the input data preprocessing unit 130 concatenates the extracted data for each modal (data type) that has been converted into sentences, converts it into a language vector, and outputs it to the multimodal data integration neural network 140 .

[0068] The granularity and number of dimensions of the data acquired by sampling over time may differ for each type of data. For example, behavioral data has a relatively small granularity (high data density) and a small number of dimensions. Hobbies, for example, have a relatively large granularity (low data density) and a large number of dimensions. The number of dimensions of the multidimensional vector output from the input data preprocessing unit 130 is the sum of the products of S (number of samples) and D (number of dimensions) for each type of extracted data, for all types of extracted data.

[0069] 6, the language vector of the task sentence is input to the intermediate layer of the multimodal data integration neural network 140. An example of the task sentence is "Based on the log related to A, please list some points to improve QOL." In addition, as an answer to the task of the task sentence, the large-scale language model 150 provides an example of the language vector of the sentence "It seems that getting up early and doing your hobby of knitting has a positive effect on sleep. Try to make it a habit."

[0070] The large-scale language model 150 outputs an answer to the task sentence received by the task receiving unit 110. The large-scale language model 150 outputs an answer to the task of the task sentence using converted data obtained by converting the stored data after preprocessing by the multimodal data integration neural network 140. The answer may be a language vector. The answer may be natural language. The large-scale language model 150 may be composed of an encoding unit and a decoding unit. In this case, the language vector of the task sentence may be input in a manner combined with the output of the encoding unit, which is also the input of the decoding unit.

[0071] The large-scale language model 150 may be stored in advance in the storage unit 200. Note that a large-scale language model available on a cloud server may be used as the large-scale language model 150. In this case, the task processing system 1 does not need to include the large-scale language model 150.

[0072] The answer output unit 160 converts the answer of the language vector output from the large-scale language model 150 into natural language and outputs it. In the example shown in Fig. 1 , the answer output unit 160 outputs text data as an answer saying "Nighttime data has not been acquired since XX / XX, and this is thought to be due to XX."

[0073] The input data determination unit 120 and the learning method of the multimodal data integration neural network 140 will now be described.

[0074] The machine learning model of the input data determination unit 120 may be trained by machine learning using a combination of the language vector of the task sentence, the type of data extracted from the stored data as extracted data, and the sampling period for the specific process as training data. In this case, the language vector of the task sentence is the input, and the type of data and the sampling period for the specific process are the correct answer labels. The task sentence, the type of data, and the sampling period for the specific process used as training data may be determined based on the opinions of on-site experts, etc. The types of data may include measurements and recognition results required to solve the task sentence, log information obtained from the equipment, recorded documents written by workers, and specialized knowledge.

[0075] The multimodal data integration neural network 140 is trained with the components of the task processing system 1 shown in FIG. 1 interconnected. When training the multimodal data integration neural network 140, the input data determination unit 120 and the large-scale language model 150 may not be trained. Training of the multimodal data integration neural network 140 may be performed as follows, using a combination of the language vectors of task sentences and the language vectors of answers as training data. By inputting the language vectors of the task sentences, which are training data, to the task receiving unit 110, the large-scale language model 150 outputs the language vectors of answers. The difference between the language vectors of the answers output from the large-scale language model 150 and the language vectors of the answers in the training data is calculated as a loss, and the multimodal data integration neural network 140 is trained by backpropagation so as to minimize the loss.

[0076] (First Modification) In this modification, first combined data obtained by combining a multidimensional vector, which is the converted data output from the multimodal data integration neural network 140, with a language vector obtained by converting the task sentence is input to the large-scale language model 150. Then, the large-scale language model 150 outputs an answer based on the first combined data.

[0077] FIG. 7 is an explanatory diagram illustrating the combining unit 170, which combines the multidimensional vectors, which are the converted data output from the multimodal data integration neural network 140, with the language vectors obtained by converting the task sentences.

[0078] The combination processor 170 generates first combined data by combining the multidimensional vectors, which are the converted data output from the multimodal data integration neural network 140, with the language vectors obtained by converting the sentences of the task. The combination processor 170 outputs the first combined data to the large-scale language model 150.

[0079] The large-scale language model 150 outputs an answer to the task of the task sentence based on the first combined data.

[0080] (Second Modification) In this modification, the multidimensional vector, which is the converted data output from the multimodal data integration neural network 140, is decoded into a sentence. The decoded sentence is then combined with the task sentence, and the second combined data converted so as to be input to the large-scale language model 150 is input to the large-scale language model 150. The large-scale language model 150 then outputs an answer based on the second combined data.

[0081] 7 decodes the multidimensional vectors, which are the converted data output from the multimodal data integration neural network 140, into sentences. The combination processing unit 170 then combines the decoded sentences with the natural language task sentences to generate second combined data that has been converted so that it can be input to the large-scale language model 150. The combination processing unit 170 outputs the second combined data to the large-scale language model 150.

[0082] The large-scale language model 150 outputs an answer to the task of the task sentence based on the second combined data.

[0083] The operation of the task processing system 1 will now be described.

[0084] 8 is a flowchart showing the operation of the task processing system 1. This flowchart can be executed by the control unit 100 in accordance with a program.

[0085] The control unit 100 receives a task as a task sentence via the communication unit 500 or the input unit 400 (S101).

[0086] The control unit 100 determines the type of data to be extracted from the stored data in the preprocessing and the specific processing based on the task sentence (S102).

[0087] The control unit 100 extracts the stored data of the determined data type from the storage unit 200, and executes pre-processing for the determined specific processing (S103).

[0088] The control unit 100 integrates the preprocessed stored data and converts it so that it can be input to the large-scale language model 150 (S104).

[0089] The control unit 100 uses the converted data to output an answer to the task in the task sentence using the large-scale language model 150 (S105).

[0090] This embodiment has the following advantages.

[0091] Based on the accepted task, the system determines the type of data to be extracted from the stored data in preprocessing. Then, the system extracts the determined type of data from the stored data, integrates the extracted data, converts it so that it can be input to a language model, and uses the converted data to output an answer to the task using the language model. This makes it possible to solve various tasks based on various types of multimodal data, while reducing cost and preventing a decrease in accuracy. Furthermore, by inputting the extracted data into a language model, the system can take advantage of the task versatility of the language model.

[0092] Furthermore, the specific processing to be performed on the extracted data extracted from the stored data is further determined, and in the preprocessing, data of the determined data type is extracted from the stored data, and the determined specific processing is performed on the extracted data. This makes it possible to solve various tasks while further reducing costs and effectively suppressing a decrease in accuracy.

[0093] The extracted data is then converted into natural language and specific processing is performed on the converted extracted data. This allows various tasks to be solved flexibly and simply, regardless of the type of stored data, while reducing costs and preventing a decrease in accuracy.

[0094] In addition, the preprocessed stored data is encoded into a multidimensional vector, and the preprocessed stored data is integrated and converted so that it can be input to a language model. This makes it possible to solve various tasks more easily, at lower cost, and with less loss of accuracy.

[0095] In addition, the measurement values ​​of various sensors installed at each specified site, various recognition results obtained by inputting the sensor measurement values ​​into the first machine learning model, log information obtained from the equipment, and specialized knowledge accumulated at the site and records written by workers are all associated with time information and stored as stored data.This makes it possible to solve various tasks efficiently, at low cost and while preventing a decrease in accuracy, based on diverse types of multimodal data.

[0096] Furthermore, the system converts the received task text into a language vector and inputs it into a neural network machine learning model to determine at least one of the data type and specific processing. This makes it possible to easily and appropriately determine the data type and specific processing that are effective for solving various tasks efficiently, while reducing costs and preventing a decrease in accuracy.

[0097] Furthermore, the specific processing includes a process of sampling data at a predetermined sampling period, which allows various tasks to be solved at lower cost and with less accuracy degradation.

[0098] Furthermore, the machine learning model that determines at least one of the data type and specific processing is one suitable for language processing, such as a Transformer model, a recurrent neural network (RNN) model, or a gated RNN model such as Long Short-Term Memory (LSTM). For example, in a Transformer model, the attention mechanism of the Transformer model determines at least one of the data type and specific processing from the task text. This makes it possible to solve various tasks more easily, at lower cost, and while preventing a decrease in accuracy.

[0099] Furthermore, the output layer of the machine learning model, which determines at least one of the data type and specific processing, is a classification model or a regression model. This allows for solving various tasks with a simple configuration, at low cost, while preventing a decrease in accuracy.

[0100] In addition, by converting all types of preprocessed stored data into language vectors in a single language vector space, the preprocessed stored data is encoded into a multidimensional vector, which makes it possible to solve various tasks flexibly, at low cost, and while suppressing a decrease in accuracy, regardless of the type of stored data.

[0101] Furthermore, by converting the preprocessed stored data into a different vector space for each data, the preprocessed stored data is encoded into a multidimensional vector, which enables various tasks to be solved flexibly, at low cost, and with minimal loss of accuracy, regardless of the type of stored data.

[0102] Furthermore, the multimodal data integration unit that integrates the extracted data and converts it so that it can be input to the language model is defined as a machine learning model. This machine learning model extracts information useful for solving a task from preprocessed stored data based on the task, integrates it into a multidimensional vector, and converts it so that it can be input to the language model. This improves the accuracy of estimating answers to tasks using the language model.

[0103] The system also generates first combined data by combining the multidimensional vectors output from the multimodal data integration unit with language vectors obtained by converting the task sentences. Then, a language model is used to output an answer to the task based on the first combined data. This allows various tasks to be solved flexibly, at low cost, and with minimal accuracy degradation, depending on the specifications of the language model.

[0104] The multi-dimensional vectors output from the multimodal data integration unit are decoded into sentences, combined with the task sentences, and converted to generate second combined data that can be input to a language model. The language model then outputs an answer to the task based on the second combined data. This allows various tasks to be solved flexibly, at low cost, and with minimal loss of accuracy, depending on the specifications of the language model.

[0105] The present invention is not limited to the above-described embodiments.

[0106] For example, in the embodiments, some or all of the processing executed by the program may be replaced with hardware such as a circuit.

[0107] While embodiments of the present invention have been described and illustrated in detail, the disclosed embodiments are made for purposes of illustration and example only and are not intended to be limiting, and the scope of the present invention should be construed by the language of the appended claims.

[0108] This application is based on a Japanese patent application (Patent Application No. 2024-119529) filed on July 25, 2024, the disclosure of which is incorporated herein by reference in its entirety.

[0109] 1 Task processing system, 100 Control unit, 110 Task reception unit, 120 Input data determination unit, 130 Input data preprocessing unit, 140 Multimodal data integration neural network, 150 Large-scale language model, 160 Answer output unit, 200 Memory unit, 300 Display unit, 400 Input unit, 500 Communication unit, 600 Bus.

Claims

1. A task processing system comprising: a task receiving unit that receives tasks from users; a language model that outputs an answer to the received task; a determination unit that determines the type of data to be extracted in preprocessing from stored data stored in a data storage unit based on the received task; a preprocessing unit that performs the preprocessing to extract data of the determined data type from the data storage unit; and a multimodal data integration unit that integrates the extracted data and converts it so that it can be input to the language model, wherein the language model outputs an answer to the task using the converted data converted by the multimodal data integration unit.

2. The task processing system of claim 1, wherein the determination unit further determines a specific process to be performed on the extracted data extracted from the stored data, and the preprocessing unit extracts data of the determined data type from the data storage unit in the preprocessing, and performs the determined specific process on the extracted data.

3. The task processing system according to claim 2, wherein the preprocessing unit performs the specific processing on the extracted data extracted from the stored data.

4. The task processing system according to claim 1, wherein the preprocessing unit encodes the stored data that has undergone the preprocessing into a multidimensional vector and outputs the encoded data to the multimodal data integration unit.

5. The task processing system of claim 1, wherein the data storage unit stores at least one of the measurement values ​​of various sensors installed at a specified site, various recognition results obtained by inputting the measurement values ​​of the sensors into a first machine learning model, and specialized knowledge accumulated at the site, log information obtained from site equipment, and records written by workers, each associated with time information.

6. The task processing system of claim 2, wherein the determination unit determines at least one of the type of data and the specific processing obtained by inputting a language vector into which the sentence of the task accepted by the task accepting unit is converted into a second machine learning model of a neural network.

7. The task processing system according to claim 2, wherein the specific processing includes a process of sampling the data at a predetermined sampling period.

8. The task processing system of claim 6, wherein the second machine learning model is one of a Transformer model, a recurrent neural network model, and a gated recurrent neural network model, and the determination unit determines at least one of the type of data and the specific process from the task sentence converted into the language vector by the second machine learning model.

9. The task processing system according to claim 6, wherein the output layer of the second machine learning model is a classification model or a regression model.

10. The task processing system of claim 4, wherein the preprocessing unit encodes the preprocessed storage data into the multidimensional vector by converting each of the preprocessed types of storage data into a language vector in a single language vector space.

11. The task processing system of claim 4, wherein the preprocessing unit encodes the preprocessed stored data into the multidimensional vector by converting the preprocessed stored data into a vector space that differs for each piece of data.

12. The task processing system of claim 1, wherein the multimodal data integration unit is a third machine learning model that extracts information useful for solving the task from the preprocessed stored data based on the task accepted by the task acceptance unit, integrates the information into a multidimensional vector, and converts it so that it can be input to the language model.

13. A task processing system as described in claim 12, further comprising a combination processing unit that generates first combined data by combining the multidimensional vector output from the multimodal data integration unit with a language vector obtained by converting the sentence of the task, and the language model outputs an answer for the task based on the first combined data.

14. The task processing system of claim 12, further comprising a combination processing unit that decodes the multidimensional vector output from the multimodal data integration unit into a sentence, combines it with the sentence of the task, and generates second combination data that has been converted so that it can be input to the language model, and the language model outputs an answer for the task based on the second combination data.

15. A task processing system comprising: a task receiving unit that receives tasks from a user; a determination unit that determines the type of data to be extracted in preprocessing from stored data stored in a data storage unit based on the received task; a preprocessing unit that performs preprocessing to extract data of the determined data type from the data storage unit; and a multimodal data integration unit that integrates the extracted data and converts it so that it can be input to a language model that outputs an answer to the task, wherein the language model uses the converted data converted by the multimodal data integration unit to output an answer to the task.

16. A control method comprising: a step (a) of accepting a task from a user; a step (b) of deciding, based on the task accepted in step (a), by a decision unit that decides, based on the accepted task, the type of data to be extracted in preprocessing from stored data stored in a data storage unit; a step (c) of performing preprocessing to extract data of the data type decided in step (b) from the data storage unit; a step (d) of integrating the extracted data extracted in step (c) and converting it by a multimodal data integration unit so that it can be input to a language model that outputs an answer to the task; and a step (e) of outputting an answer to the task by the language model using the converted data converted in step (d).

17. A control program for causing a computer to execute a process comprising: (a) receiving a task from a user; (b) determining, based on the task received in step (a), by a determination unit that determines, based on the received task, the type of data to be extracted in preprocessing from stored data stored in a data storage unit; (c) performing preprocessing to extract, from the data storage unit, data of the type of data determined in step (b); (d) integrating the extracted data extracted in step (c) and converting it by a multimodal data integration unit so that it can be input to a language model that outputs an answer to the task; and (e) outputting an answer to the task by the language model using the converted data converted in step (d).

Images