Data processing method, data processing system, and data processing program
Patent Information
- Application Number
- JP2025026240
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-02-20
- Publication Date
- 2026-09-01
- Estimated Expiration
- 2045-02-20
AI Technical Summary
【0014】 本発明に係るデータ処理方法、データ処理システム、又はデータ処理用プログラムを用いることにより、複数の所定の項目のそれぞれに値が入力されたデータに対する処理を実行する際の利便性を向上することができる。
Smart Images

Figure 2026139501000001_ABST
Abstract
Description
[[Technical Field]]
[0001] The present invention relates to a technology for executing processing on data in which values are input to each of a plurality of predetermined data items. [[Background Art]]
[0002] Databases storing customer information and the like are used in various places such as companies and public offices. As a database, a relational database is widely used, which is configured by setting data items of information to be stored in one or more columns and associating a plurality of tables in which records are input in the row direction with each other. In a relational database, for example, a user can obtain information on a desired record by specifying a table and column to be searched, setting search conditions, and executing a search using an SQL (Structured Query Language) statement (see, for example, Patent Document 1). [[Prior Art Documents]] [[Patent Documents]]
[0003] [[Patent Document 1]] Japanese Patent Laid-Open No. 5-120100 [[Patent Document 2]] Japanese Patent Laid-Open No. 2022-91986 [[Non-Patent Documents]]
[0004] [[Non-Patent Document 1]] "Grasp the Image of Embedding (Vectorization)!", [online], Classmethod, Inc., retrieved February 20, 2025, Internet <URL: https: / / zenn.dev / peishim / articles / c696ff85a539bd> [[Non-Patent Document 2]] "IBM Granite", [online], IBM Japan, Ltd., accessed February 20, 2025, Internet<URL:https: / / www.ibm.com / jp-ja / topics / large-language-models> [Overview of the project] [Problems that the invention aims to solve]
[0005] To extract the desired records using the database search described above, the user must accurately specify the table name, the names of the data items set in each column, and the search conditions related to the attributes of the values of each data item (numeric, text, etc.) in the SQL statement. Therefore, there was a problem in that it was impossible to extract the desired records from the database without knowing the database structure. Although this explanation uses database search as an example, the same problem exists in processes such as inputting, updating, and deleting database records, as it is necessary to accurately specify the name of the table being processed, the names of the columns, etc. Furthermore, the same problem exists when processing data sets that do not take the form of a so-called database.
[0006] The problem that this invention aims to solve is to improve the convenience of performing processing on data in which values have been entered for each of several predetermined items. [Means for solving the problem]
[0007] One aspect of the present invention, made to solve the above problems, is a method used to perform processing on target data in which values are entered for each of a plurality of predetermined data items, We prepare a large-scale language model built using a predetermined machine learning method, which takes text data as input and outputs key words for data processing. Prepare a vectorization model that is constructed using a predetermined machine learning method and outputs vectorized input words. Using the vectorization model described above, the information identifying the multiple data items and the attribute information of each value of the multiple data items are vectorized. Prepare design information including information on the relationships between the multiple data items, information identifying the multiple vectorized data items, and attribute information of the values of each of the multiple vectorized data items. The user is instructed to input information that identifies the type of processing to the target data and text data that represents the content of the processing. By inputting text data representing the content of the aforementioned process into the large-scale language model, the large-scale language model outputs key words for the process. Key vectors are generated by inputting the words output from the aforementioned large-scale language model into the aforementioned vectorization model. By referring to the aforementioned design information data, the data items and processing content of the target data are identified from the key vector. Perform processing on the identified data item, or output information for performing such processing. It is characterized by the following:
[0008] Large Language Models (LLMs) are generative AIs specialized for Natural Language Processing (NLP), constructed using neural networks whose parameters are tuned by machine learning on vectors generated through tokenization (dividing various text data into the smallest units such as words and punctuation) and vectorization (converting the tokenized data into numerical values). By pre-training such large language models with the attributes of the words to be output, it is possible to output words according to the intended use (database search in this invention) from text data input as natural language. Vectorization models (also called embedding models) are pre-trained models constructed by deep learning using, for example, a large number of words and vectors. By using such vectorization models, it is possible to output vectors that have components that reflect the meaning of the input words.
[0009] The data processing method according to the present invention is typically used to process data (records) that constitute a database, but it can also be used to process other types of data. In the data processing method according to the present invention, a large-scale language model and a vectorization model are prepared in advance. Furthermore, information that identifies data items and attribute information of the record values entered for each data item are vectorized and combined with information on the relationships between data items to create design information data. The attribute information of values referred to here includes, for example, numerical values, free text, and selection options (those selected in a pull-down format, etc.). Note that the values of the target data may also include Null. Furthermore, information on the relationships between data items includes, for example, information that identifies the common table when multiple data items constitute one table, and information that when a value is entered for one data item, the value of another data item is updated based on that value.
[0010] The user inputs information that identifies the type of processing to be performed on the data, along with text data representing the content of that processing. Examples of data processing types include data retrieval, data item value input, updating, and deletion. The information regarding the type of processing is not limited to direct user input; it may also include indirect input (e.g., identifying the type of processing based on input through a specific screen, or identifying the type of processing based on text data being sent to a specific email address). The input text data is fed into a large-scale language model, which outputs key words for data processing. Examples of key words for data processing include words related to data items such as industry, address, and number of employees, as well as words related to search conditions such as "within one year," "within one month," and "Kyoto Prefecture." Furthermore, a vectorization model is used to vectorize the words output from the large-scale language model, generating key vectors. Then, by referring to design information data, the data item to be processed and the processing content are identified from the key vectors. This method of vectorizing the meaning of words and sentences and identifying similar meanings is called semantic search or vector search. By performing such searches, even if the words output from a large-scale language model, the information identifying the data items, and the attribute information of the values of each data item do not perfectly match, it is possible to identify the target and content of data processing based on the semantic similarity (degree of similarity of vector components) between the two. For example, the target and content of data processing can be identified based on having the vector that is closest to the processing key vector (smallest difference in components, shortest distance between vectors, smallest angle between vectors, etc.).
[0011] Once the data items to be processed and the content of the processing for those data items are determined, the system either executes the specified processing for the specified data items or outputs information for executing that processing.
[0012] Another aspect of the present invention is a system used to perform processing on data in which values are entered for each of a plurality of predetermined items, A large-scale language model storage unit stores a large-scale language model that was constructed using a predetermined machine learning method, which receives text data as input and outputs key words for data processing. A vectorization model memory unit stores a vectorization model that is constructed by a predetermined machine learning method and outputs vectorized input words, A design information data storage unit stores design information data including information on the relationships between the plurality of data items, information identifying the plurality of data items vectorized by the vectorization model, and attribute information of the values of each of the plurality of data items vectorized by the vectorization model. A processing key acquisition unit that prompts the user to input information identifying the type of processing to the target data and text data representing the content of the processing, and inputs the text data representing the content of the processing into the large-scale language model, thereby outputting a key word for the processing from the large-scale language model, A key vector generation unit generates key vectors by inputting words output from the large-scale language model into the vectorization model, A processing content identification unit identifies the data items and processing content of the target data from the key vector by referring to the design information data, A processing execution unit that performs processing on the specified data item or outputs information for performing said processing. It is characterized by having the following features.
[0013] Yet another aspect of the present invention is a program used to perform processing on data in which values are entered for each of a plurality of predetermined items, A large-scale language model storage unit stores a large-scale language model that was constructed using a predetermined machine learning method, which receives text data as input and outputs key words for data processing. a vectorization model storage unit storing a vectorization model constructed by predetermined machine learning, which vectorizes an input word and outputs the vectorized word; a design information data storage unit storing design information data including information on relevance between the plurality of data items, information specifying the plurality of data items vectorized by the vectorization model, and attribute information on respective values of the plurality of data items vectorized by the vectorization model; In a system comprising: a processing key acquisition unit that causes a user to input information specifying a type of processing for target data and text data representing content of the processing, and inputs the text data representing the content of the processing to the large-scale language model, thereby causing the large-scale language model to output a word serving as a key for the processing; a key vector generation unit that generates a key vector by inputting the word output from the large-scale language model to the vectorization model; a processing content specifying unit that specifies a data item of the target data and processing content from the key vector by referring to the design information data; a processing execution unit that executes the processing of the specified content on the specified data item, or outputs information for executing the processing , wherein the computer is caused to function as the above components. Effects of the Invention
[0014] Use of the data processing method, data processing system, or data processing program according to the present invention can improve convenience when executing processing on data in which values are input to each of a plurality of predetermined items. Brief Description of the Drawings
[0015] [Figure 1] FIG. 1 is a configuration diagram of essential parts of an embodiment of the data processing system according to the present invention. [Figure 2] FIG. 2 is a diagram explaining the configuration of design information data related to data processing in the present embodiment. [Figure 3] A diagram illustrating the structure of design information data related to knowledge processing in this embodiment. [Figure 4] Examples of design information for column processing groups (information groups) and design information for join information in this embodiment. [Figure 5] Examples of design information related to basic information and design information related to item information in this embodiment. [Figure 6] An example of design information related to the choice information in this embodiment. [Figure 7] An example of design information relating to sample value information in this embodiment. [Figure 8] An example of design information related to trigger information in this embodiment. [Figure 9] An example of design information related to update details information triggered by a trigger in this embodiment. [Figure 10] Examples of design information related to knowledge information and design information related to knowledge information documents in this embodiment. [Figure 11] An example of processing on the database in this embodiment. [Figure 12] An example of processing of data entered into a screen or the like in this embodiment. [Figure 13] An example of knowledge processing in this embodiment. [Figure 14] This image flow illustrates the process of integrating the data processing device of this embodiment into an existing business system to perform data processing. [Figure 15] An example of request information (structure) in this embodiment. [Figure 16] An example of decomposition result information (structure) in this embodiment. [Figure 17] An example of the returned information (structure) in this embodiment. [Figure 18] An example screen of the input form for customer visit reports in this embodiment. [Modes for carrying out the invention]
[0016] Embodiments of the data processing method, data processing system, and data processing program according to the present invention will be described below with reference to the drawings. In this embodiment, an example of processing information stored in a business system (corporate database) established in a company will be described.
[0017] Figure 1 shows the main components of the data processing system 1 of this embodiment. The data processing system 1 includes a business system 10 and a data processing device 20 that a company or other organization has within its private network.
[0018] Business system 10 includes a sales support system 11, a sales management system 12, a payroll system 13, and a human resources management system 14. Furthermore, the sales support system 11 has a customer management database (DB) 11D, the sales management system 12 has a financial management database (DB) 12D, the payroll system 13 has a payroll management database (DB) 13D, and the human resources management system 14 has a human resources management database (DB) 14D. While this example describes a scenario where each system has one database, a single system may have multiple databases.
[0019] The customer management database 11D, the financial management database 12D, the payroll management database 13D, and the human resources management database 14D are all relational databases that link together multiple tables in which data items are arranged in columns and records are arranged in rows.
[0020] The data processing device 20 includes a storage unit 21. The storage unit 21 is equipped with a design information data storage unit 211, a tuning data storage unit 212, a large-scale language model (LLM) storage unit 213, a vectorization model storage unit 214, and an input form storage unit 215.
[0021] The design information data storage unit 211 is one of the characteristic elements of this embodiment, and it stores information regarding the structure of various databases that are processed by the data processing device 20 as design information data.
[0022] Design information data is broadly composed of information related to data processing (including database processing and other natural language processing) and knowledge information (data processing performed based on accumulated data).
[0023] Figure 2 shows the structure of the design information data related to data processing. The design information related to data processing includes (1) processing groups to specify the content of the processing to be executed, (2) join information representing the join relationships between tables in the database and between multiple input screens, (3) basic information about the database tables and screens, (4) item information regarding columns in tables and items provided in screens, (5) choice information regarding the choices presented in the database columns and items in screens, (6) value sample information regarding the values entered in the database columns and items in screens, and (11)(12) trigger information.
[0024] Figure 3 shows the structure of design information data related to knowledge processing. The design information related to knowledge processing includes (7) knowledge information that has accumulated documents such as daily reports, and (8) knowledge information-related documents (templates, etc.) for generating new information such as instructions and advice based on the knowledge information.
[0025] Figure 4 shows examples of design information for (1) processing groups (information groups) and (2) join information. (1) Processing groups (information groups) do not directly handle information from database tables or screens, but refer to labels as a collection of information. Design information for processing groups (information groups) includes items related to system identification, function group, group code, target identification, name, meaning, vector value, and access rights, each with a value (character, numeric, vector array). (2) Join information refers to joiner information that allows a processing group to view several basic pieces of information as a single unit, such as which combination of tables in the database will be used for processing, or, in the case of screens, which collection of screens or objects will be used for processing. Design information for join information includes items related to system identification, function group, group code, integration sequence information, information ID, name, identifier, integration type, and integration source information conditions, each with a value (character, numeric).
[0026] Figure 5 shows examples of design information for (3) basic information and (4) item information. (3) Basic information refers to tables in databases, labels for the entire screen in screens, and labels for the entire process in processes. Design information for basic information includes items related to system identification, function group, target identification, information ID, name, meaning, vector value, and access rights, each with a value (character, numeric, vector array). (4) Item information refers to columns in databases, item objects in screens, and labels for process details in processes. Design information for item information includes items related to system identification, function group, target identification, information ID, item ID, selection reference category, selection reference item ID, name, meaning, vector value, attribute, number of digits, default value, item type, required item, automatic setting target, and access rights, each with a value (character, numeric, vector array).
[0027] Figure 6 shows an example of design information related to (5) choice information. (5) Choice information refers to information for each selection value when the attribute is a choice (such as a selection list or radio buttons). The design information related to choice information includes items related to system identification, function group, target identification, information ID, item ID, choice, name, meaning, vector value, attribute, and number of digits, and each of these is assigned a value (character, number, or vector array).
[0028] Figure 7 shows an example of design information related to (6) sample value information. (6) Sample value information refers to information for each sample value that represents the characteristics of an item when the attribute of the item is not a choice. The design information related to sample value information includes fields for system identification, function group, target identification, information ID, item ID, attribute, three types of sample values (character, number, date), name, meaning, and vector value, each of which is assigned a value (character, number, vector array). In addition to item-specific information, it is also possible to register and use public information that is common within the system or within a function group for choice information and sample information.
[0029] Figure 8 shows an example of design information related to (11) trigger information. (11) Trigger information refers to monitoring item information and specification information that links other items to be updated when there is input or change in that item. The design information related to (11) trigger information includes items related to system identification, function group, trigger management number, target information ID, target item ID, activation condition, activation specified value (character, number, date), action, and access rights, and each of these is set with a value (character, number, date).
[0030] Figure 9 shows an example of design information for (12) trigger detail information. (12) Trigger detail information refers to the source information and destination information after the trigger is activated. The design information for (12) trigger detail information includes items such as system identification, function group, trigger management number, classification for identifying the source / update destination, branch number (line), target information ID, target item ID, read / update conditions, prompt, specified value (character, number, date), and operation, each of which is set with a value (character, number, date).
[0031] Figure 10 shows examples of (7) design information for knowledge information and (8) design information for knowledge information-related documents. (7) The design information for knowledge information includes items for system identification, function group, information-specific number, related type, record document, vector value, keys 1-3, and name, each with a value (character, numeric, vector array). (8) The knowledge information-related documents include items for system identification, function group, information-specific number, related document sub-number, and related document, each with a value (character, numeric).
[0032] The tuning data storage unit 212 stores data for tuning the large-scale language model. Tuning the large-scale language model is performed using learning methods such as prompt tuning and fine tuning. This results in a large-scale language model that has learned industry-specific information (e.g., industry terms and their meanings, abbreviations). As tuning data, for example, in the manufacturing industry, information such as the industry term "yield rate," which means the probability that no defective products will occur, that the yield rate is expressed as a numerical value between 0 and 1 (or a percentage between 0 and 100), and that a yield rate closer to 1 means a higher proportion of good products, while a yield rate closer to 0 means a higher proportion of defective products, is used.
[0033] Furthermore, the memory unit 21 includes a large-scale language model memory unit 213 that stores a large-scale language model, which is a trained model constructed by machine learning that receives text data as input and outputs key words for data processing, and a vectorization model memory unit 214 that stores a vectorization model constructed by machine learning that vectorizes input words and outputs them.
[0034] The Large Language Models (LLMs) used in this embodiment are generative AI specialized for natural language processing (NLP). They are constructed using a neural network whose parameters are adjusted by machine learning on vectors generated through tokenization, which divides various text data into the smallest units such as words and punctuation, and vectorization, which converts the tokenized data into numerical values. In this embodiment, the attributes of the words to be output are pre-trained in such a large language model, and the tokenization and cleansing of text data input in natural language, etc., are performed to output words related to the processing of the data.
[0035] In this embodiment, a large-scale language model stored in a large-scale language model storage unit 213, which is only usable within the company (located within a private network with no external connections), is used. It is difficult for a single company to own a large-scale language model, and it is common to share a large-scale language model provided on the cloud with an unspecified number of people (or a specific number of people). In this invention, it is also possible to use a large-scale language model that is generally widely available (public), but if information related to confidential company matters such as customer information is trained on the large-scale language model, there is a risk that the information may be leaked to the outside.
[0036] Therefore, as in this embodiment, it is preferable to use a large-scale language model that is only available in a closed environment (private). In this embodiment, it is sufficient to extract words related to the processing of information stored in the database (search, data entry, data update, etc.) from the input natural language, and it is not necessary to use a machine learning model that has acquired all kinds of words. As such a large-scale language model, for example, IBM's Granite (Non-Patent Literature 2) can be suitably used. Granite is a large-scale language model designed to be lightweight and have a low computational load, and it can be owned and used by companies even if they are not large corporations. When using such an unofficial large-scale language model, it is preferable to perform tuning using the design information data stored in the design information data storage unit 211 when performing the above tuning. This makes it possible to output processing keys that can more accurately identify tables, columns, etc. from the large-scale language model.
[0037] The vectorization model used in this embodiment is a trained model constructed by deep learning using a large number of word-vector pairs, and constitutes part of a large-scale language model. In this embodiment, by using such a vectorization model, it is possible to output a vector having components that reflect the meaning of the input word. The vector array (vector value) included in the above design information can be generated by using such a vectorization model. The vectorization model is also called an embedding model (for example, Non-Patent Document 1) and is used to perform semantic search (also called vector search). Suitable software that can be used via a network such as the Internet may be used for the vectorization model, or software owned by a company may be used.
[0038] The memory unit 21 is further equipped with an input form memory unit 215 that stores input forms for various types of data. For example, input forms related to the customer management database 11D include a form for registering a new customer, a form for reporting the results of customer visits, and a form for entering the results of transactions with customers. These input forms are provided with data items corresponding to columns and display fields for entering the values of those columns. Furthermore, for specific data items, processing is associated with the input of a value in a particular data item, such as changing the value of another column (this is stored as trigger information; Figures 8 and 9). In addition, when text data such as natural language is entered into specific input / display fields of each input form, processing is performed by a large-scale language model and / or a vectorization model (this is also stored as trigger information).
[0039] The data processing device 20 comprises, as functional blocks, a design information data creation unit 31, a tuning execution unit 32, a processing key acquisition unit 33, a key vector generation unit 34, a processing content identification unit 35, a processing instruction generation unit 36, and a processing execution unit 37. The data processing device 20 is configured, for example, as a general personal computer or workstation, and these functional blocks are realized by executing a pre-installed data processing program on the processor. The data processing device 20 is also connected to an input unit 51 consisting of a keyboard and mouse, and a display unit 52 consisting of a liquid crystal display, etc.
[0040] Figures 11 to 13 show an overview of the data processing performed in the data processing device of this embodiment. Figure 11 relates to database processing, Figure 12 to data input on a screen, etc., and Figure 13 relates to knowledge processing.
[0041] As shown in Figure 11, there are four types of operations on a database: data retrieval (SELECT), data update (UPDATE), data creation (INSERT), and data deletion (DELETE). When performing any of these data operations, text data consisting of natural language is input, and output is obtained by processing using a large-scale language model (LLM) (Language processing using LLM 1) and vector database (DB) processing (Vector DB processing 1).
[0042] As shown in Figure 12, there are three types of processing for data entered on a screen, etc.: processing to generate a summary from a group of items, processing to generate a group of items from a sentence, and processing to generate a group of conditions from a sentence. In the process of generating a summary from a group of items, information on the structure of the group of items and values is input, and a summary document is obtained by performing vector database (DB) processing (Vector DB processing 1) and processing using a large-scale language model (LLM) (Language processing using LLM 2). In the processes of generating a group of items from a sentence and generating a group of conditions from a sentence, natural language text and information on the structure of the group of items to be set are input, and an array of items and set values is obtained by performing processing using a large-scale language model (LLM) (Language processing using LLM 1) and processing using a vector database (DB) (Vector DB processing 1).
[0043] As shown in Figure 13, there are three types of knowledge processing: storage (vector DB), summarization, and proposal. In storage processing, the structure information of the items and values from which the data is to be summarized, the related types, and related documents are input, and the data is stored by processing using a large-scale language model (LLM) (Language processing by LLM 1), vector database (DB) processing (Vector DB processing 1), processing using a large-scale language model (LLM) (Language processing by LLM 2), and vector database (DB) processing (Vector DB processing 2). (No output is produced in storage processing). In summarization processing, the structure information of the items and values from which the data is to be summarized is input, and the data is stored by processing using a large-scale language model (Language processing by LLM 1), vector database (DB) processing (Vector DB processing 1), and processing using a large-scale language model (LLM) (Language processing by LLM 2) to obtain a summary document as output. In the proposal processing, the structure information of the items and values from the source of the summary, as well as the return association type, are input, and processing is performed using a large-scale language model (LLM) (Language processing by LLM 1), vector database (DB) processing (Vector DB processing 1), processing using a large-scale language model (LLM) (Language processing by LLM 2), and vector database (DB) processing (Vector DB processing 2), resulting in the output being an associated document.
[0044] Next, the operation of the data processing system of this embodiment (one embodiment of the data processing method according to the present invention) will be described.
[0045] In the data processing system 1 of this embodiment, the processing described in Figures 11 to 13 is executed for each of the databases of the business system 10 that are to be processed, using the design information described with reference to Figures 2 to 10.
[0046] The data processing device 20 of this embodiment can be integrated into an existing business system 10. Figure 14 is an image flow illustrating the flow of execution of the above processing in a data processing system 1 in which the data processing device 20 is integrated into an existing business system 10. In Figure 14, the area enclosed by the dashed line shows the processing and operation in the business system 10, and the area enclosed by the dashed line shows the processing and operation by the data processing device 20. The business system 10 and the data processing device 20 are configured to send and receive information via an Application Programming Interface (API).
[0047] Before operating the data processing system 1 of this embodiment, the design information data creation unit 31 creates the design information described with reference to Figures 2 to 10 and stores it as a vector database (DB) in the design information data storage unit 211 of the data processing device 20. In addition, the tuning execution unit 32 performs learning such as prompt tuning and fine tuning of the large-scale language model using the tuning data stored in the tuning data storage unit 212 and stores the trained large-scale language model in the large-scale language model storage unit 213.
[0048] Generally, a database is associated with a program for processing data within that database, and through this program, users can perform various data operations on that database. In conventional data processing, users input commands into the program in a language that the program can process, thereby performing data processing.
[0049] On the other hand, when the data processing device 20 of this embodiment is incorporated, when a user inputs text data (request information) such as natural language, it is sent to the data processing device 20 via the API. In the data processing device 20, the processing key acquisition unit 33 inputs the text data as request information (structure) into the large-scale language model, and obtains the decomposition result information (structure) output by the large-scale language model as a processing key (language processing 1 by LLM).
[0050] As shown in Figure 15, the request information (structure) may include API basic information, target identification, request action, requested natural language statement, additional prompts, information ID, and input information array. Of these, the API basic information and target identification can be identified, for example, from the screen entered by the user or the email address to which the user sent text data. Also, as shown in Figure 16, the decomposition result information (structure) may include API basic information, target identification, information ID, input information array, and decomposition information array.
[0051] Next, the key vector generation unit 34 vectorizes the processing key using a vectorization model. Then, the processing content identification unit 35 uses that vector to perform a semantic search of specification information, values, and options in the design information database (vector DB) to identify the content of the data processing requested by the text data entered by the user (vector DB processing 1).
[0052] Here, if the processing content is processing on the database (Figure 11), the processing instruction generation unit 36 generates an instruction statement (an instruction statement written in a programming language associated with the target database) describing the content of the specified data processing as return information (structure), and sends it to the business system 10 via the API. As shown in Figure 17, the return information (structure) may include result status information and a result information array.
[0053] If the processing content is a process that generates a summary from a group of items (Figure 12) or knowledge processing (Figure 13), the processing execution unit 37 executes processing using the large-scale language model again (such as creating a summary sentence) (Language processing using LLM 2). Also, if the processing content is summarization processing or suggestion processing (Figure 13), the processing execution unit 37 executes a search of the vector DB using the vectorization model again (Vector DB processing 2). After these processes, the processing instruction generation unit 36 generates an instruction statement (an instruction statement written in a programming language associated with the target database) describing the content of the identified data processing as return information (a structure) and sends it to the business system 10 via the API.
[0054] The data processing device 20 outputs information in a format pre-associated with the content of each data processing operation after the above processing has been performed.
[0055] The information output from the data processing device 20 is input to the business system 10 via an API. In the business system 10, programs associated with each database process this information. This allows for data processing based on request information entered by the user in text data such as natural language.
[0056] In conventional data processing systems, in order for users to perform desired data processing (such as extracting records), they had to accurately specify the table name, the names of the data items set in each column, the attributes of the values of each data item (numeric, text, etc.), and the processing content in an SQL statement. This presented a problem: users could not extract the desired records from the database without knowing the database structure or programming language.
[0057] In contrast, in this embodiment, even if the processing key (the word) does not match the names set in the table information, column information, etc., that constitute the database, the table and column to be searched can be identified based on the semantic similarity determined by comparing vector values, and further, the processing content for the values of those columns can be identified. This type of processing is called semantic search or vector search. By performing a semantic search in this way, the desired information can be extracted even if the user does not have knowledge of the structure of the database being processed. In addition, the user only needs to input the content of the desired data processing in text data such as natural language, and no knowledge of programming languages is required.
[0058] Furthermore, in business systems, it is common for various employees to add records or change or update the values of data items. However, not all employees know the input format for entering data into data items. As a result, even values with the same meaning may be mixed in different formats. Examples of different formats include the mixing of Arabic numerals and Chinese numerals, and the presence or absence of units (entering units causes the data to be recognized as text data and excluded from statistical processing of numerical data). In conventional data processing systems, when values in various formats are entered in this way, the entered values are reflected as they are. For example, when performing statistical processing on numerical data, values entered with Chinese numerals or values entered with units are not processed statistically, which was a problem.
[0059] In contrast, in the data processing system 1 of this embodiment, data processing is performed on text data entered by the user in various formats, in a format specified in the design information. It can perform statistical processing on numerical data and extract data accurately.
[0060] The data processing system 1 of this embodiment can implement the following functions.
[0061] Support functions: For example, automatically filling in remaining items based on some input data such as request natural language text. Collecting and summarizing individual input values to create a summary document (e.g., the body of a daily report). Analyzing the content of natural language text written in emails, etc., and calling up the processing flow that should be performed afterward.
[0062] Automatic creation function: By attaching audio, memos, photos, etc., to emails from outside the office and sending them to a designated email address that has been pre-associated with API basic information, target identification, and requested actions, daily reports and other documents are automatically created.
[0063] Response function: The system searches for past inquiries with similar meanings to the inquiry received via the inquiry form or the email address set as the contact address, and proposes a response using a pre-prepared template.
[0064] Data extraction (search) function: Automatically sets appropriate search conditions based on the requested text. Accumulation function: Accumulate data on successful and unsuccessful patterns and utilize it as knowledge. For example, summarize reports and suggestions can be generated from accumulated data that has similar meaning to the data entered, such as daily reports. Application: Read text from business card images and automatically extract customer and contact person information.
[0065] The following describes some specific examples of data processing in the data processing system 1 of this embodiment.
[0066] <Database operation: SELECT> When a user inputs the text data "List of customers who have not been visited recently" from a screen pre-associated with the customer management database 11D, the large-scale language model outputs the words "customer," "list," and "not recently visited." A semantic search of the design information data identifies that "customer" refers to the target table, "list" refers to the target column group, and "not recently visited" refers to the search condition. The processing command generation unit 36 then generates a command statement to execute this data search and sends it to the business system 10 as return information.
[0067] <Database operation: UPDATE> When a user enters the text data "Enable direct mail (DM) delivery to prospective customers in negotiations within a 5km radius" from a screen pre-mapped to the customer management database 11D, the large-scale language model outputs the phrases "In negotiations; Customer," "DM delivery; Enabled," and "Within a 5km radius; Prospective customer." A semantic search of the design information data then identifies that the phrase "In negotiations; Customer" refers to the target table. The system also searches for column information corresponding to the phrase "DM delivery; Enabled" and searches for the specification related to "Enabled," which represents the processing for that column. The system then identifies the option information with the closest meaning. Furthermore, even when a specific numerical value such as "Within a 5km radius" is entered, a semantic search of the design information identifies the option information with the closest meaning. As a result, for customers that match the search condition "Within a 5km radius; Prospective customer," the option for the data item "DM delivery" is updated to the value "Enabled."
[0068] <Database processing: INSERT & Processing of items on screens, etc.: Conditional statements from text> The user accesses the screen to add a "customer visit report" to the customer management database 11D. The processing key acquisition unit 33 then reads the customer visit report input form stored in the storage unit 21 and displays it on the screen of the display unit 52.
[0069] Figure 18 shows an example of the input form screen for a customer visit report. This screen has fields for customer code, customer name, status report, ABC rank, and probability. Of these, the user enters text data in the customer name and status report fields (fields shown with solid lines in Figure 18). When the user enters the customer name, the customer code is retrieved from the database and displayed in the customer code field.
[0070] The status report field is used to input natural language text data. When the user inputs natural language text data into the status report field, the processing key acquisition unit 33 inputs the input natural language into the large-scale language model. The large-scale language model outputs words related to data processing (processing keys) from the input natural language, such as "order amount," "high value," "promising," "competitor," "slightly unfavorable," and "situation."
[0071] The key vector generation unit 34 inputs the text output from the large-scale language model into the vectorization model and generates key vectors. The processing content identification unit 35 refers to the design information data and identifies the content of the data processing corresponding to the generated vector values. In this example, the design information includes trigger information that generates values for priority and confidence data items when a value is input for the status report data item. Referring to this design information and the attribute information of the values input to priority and confidence respectively, the processing content identification unit 35 determines the input values for the priority and confidence data items from the output word group (in the example in Figure 18, the priority value is "A" and the confidence value is "50%").
[0072] The processing command generation unit 36 generates a command statement to add the processing content (data item and its value pair) identified by the processing content identification unit 35 as a record to the customer management database 11D, and sends it to the business system 10 as return information (structure). The business system 10 executes the processing command statement generated by the processing command generation unit 36 and adds the record to the database specified by the user (in this case, the customer management database 11D).
[0073] In conventional data processing systems, information entered by users in natural language, such as the situation report in the example above, is simply entered as database values, and meaningful statistical processing cannot be performed. In contrast, in the data processing system 1 of this embodiment, predetermined data item values (in this case, priority and accuracy) are determined from the natural language entered by the user, making it possible to perform meaningful statistical processing using these values. Furthermore, with natural language input, even if the content is the same, the length and expression of the text can differ depending on the person entering it, making it difficult to correctly grasp the situation without carefully reading the entered natural language text. In contrast, in the data processing system 1 of this embodiment, a large-scale language model can be used to evaluate natural language written by various users using common criteria, and values obtained from this can be registered in the database.
[0074] Patent Document 2 describes a technique for generating SQL statements from natural language, which involves pre-matching the question information to be processed with the table content of a target table to obtain pre-matching results, extracting the word segment with the highest degree of matching to the question, determining the target attribute value semantically associated with the extracted word segment, and generating an SQL query statement using that target attribute value. According to Patent Document 2, this technique makes it possible to generate an SQL statement even if the question information presented by the user does not perfectly match the column names or attribute values of the table.
[0075] Patent Document 2 only conceptually presents the above technology, and does not show any specific examples of its application. This technology generates SQL statements in response to interactive requests from a human user and outputs the data resulting from executing those statements; it cannot use any search methods other than SQL statements. Therefore, it can only perform database searches that conform to the input format of the SQL statement. Furthermore, it is not easy to accurately output the elements necessary for data processing from a large-scale language model and generate SQL statements. In addition, the meaning of the term "attribute" is not clear in Patent Document 2. Moreover, it is unclear how to apply this technology when there are multiple tables in the database (such as in a so-called relational database).
[0076] On the other hand, while SQL statements can be used in the data processing system 1 of this embodiment, it essentially outputs the elements necessary for database searching through semantic search, and can be used for search methods other than database searching using SQL statements. Furthermore, in this embodiment, as explained with reference to Figures 2 to 10, the attributes of each item are defined in the database design information data, thereby enabling accurate data processing in line with the user's intentions. Moreover, the data processing system 1 of this embodiment can perform diverse data processing by using the large-scale language model and vectorization model not only for database searching, but also for registering records in the database, updating and deleting data item values, and processing input to items on screens, etc. (Figure 12) and knowledge processing (Figure 13).
[0077] As described above, the data processing method, data processing system, and data processing program of this embodiment combine artificial intelligence (AI) with business systems (including existing, vast business systems). By using the data processing method, data processing system, and data processing program of this embodiment, existing business systems can be improved into flexible and efficient business systems that are compatible with AI with minor modifications, without the need to redesign or redevelop existing business systems. Furthermore, the data processing method, data processing system, and data processing program of this embodiment can accumulate more design information as it is used, thereby improving the AI's understanding of the design information. As a result, it is anticipated that in the future, it will be possible to have the AI generate various data input / output screens and reports themselves according to user needs, and the programs that handle such processing will become unnecessary.
[0078] The above embodiments are examples and can be modified as appropriate in accordance with the spirit of the present invention.
[0079] In the above embodiment, we described a case where the user identifies the database to be processed or inputs the data processing content from an entry point (a specific screen, a specific email address) corresponding to the data processing content. However, the entry point for executing all databases and all types of data processing may be common. In that case, the processing content identification unit 35 refers to the design information data of all databases and identifies the database to be processed and the type and content of data processing in that database based on the key vector generated by the key vector generation unit 34 from the text data such as natural language input by the user. However, by separating the entry point, such as a screen for the user to input instructions, for at least one database, the load related to semantic search is reduced, and more accurate data processing becomes easier.
[0080] In some cases, it may not be possible to extract all the elements necessary for the instruction statement to execute data processing from the text data entered by the user. In such cases, the processing content identification unit 35 should notify the user by displaying the missing elements on the screen of the display unit 52, or by outputting an error message.
[0081] In the above embodiment, an example was described in which all the necessary configurations for searching information in the database and inputting data into the database are provided inside the data processing device 20. However, some of these configurations may be provided in other devices (including other devices in a private network, or cloud servers provided on an open network with access restricted by encryption or passwords). For example, in the above embodiment, the large-scale language model storage unit 213 and the vectorization model storage unit 214 are provided inside the data processing device 20. However, large-scale language models and vectorization models provided in other devices (same as above) connected to the data processing device 20 via a network may be used. Furthermore, when using public large-scale language models and vectorization models, those provided in other devices (such as cloud computers) connected via an open network (such as the Internet) may be used. In addition, functional blocks related to database searching and functional blocks related to data input may be provided in separate devices (same as above), and these devices may be connected to the network inside the data processing system 1.
[0082] In the above embodiment, an example of data processing for a customer management database 11D provided in a company's business system 10 was described, but the same configuration can be adopted when processing information in various types of databases provided in public institutions such as government agencies. Furthermore, the specific examples of data processing described in the above embodiment are only a part of what can be executed in the data processing system 1 of this embodiment, and can be appropriately modified depending on the contents of the database and its intended use. In addition, in the above embodiment, a configuration in which the business system 10 and the data processing device 20 are provided within a single private network was described, but they may be connected via an open network (such as the Internet).
[0083] In the above embodiment, we described a case where access to all data in the database is permitted. However, it is also possible to restrict access to certain data (for example, management data, financial data, or personnel data) to only those with access privileges. For example, when using the data processing system 1, users can log in using a user ID and password, and the scope of access can be determined according to the authenticated user ID.
[0084] In the above embodiment, an example of processing data in a database was described, but the same configuration can be used when processing data other than in a database. For example, the process of adding a customer visit report described in the above embodiment can also be used for purposes such as submitting the customer visit report in writing (i.e., for purposes that do not involve adding it as a record in a database). Also, in the above embodiment, the values of data items such as priority and probability were generated from natural language text entered by the user, but conversely, the system could be configured so that the user inputs the priority and probability, and natural language text is generated from these values using a large-scale language model.
[0085] Furthermore, multiple trigger information in the above embodiment can be combined to execute flowchart-like processing. For example, if the number of records with priority A in the customer management database 11D exceeds 10, those records can be extracted and output as a list of data. Various processes can be combined and executed in this way.
[0086] [Pattern] It will be obvious to those skilled in the art that the exemplary embodiments described above are specific examples of the following embodiments.
[0087] (Section 1) One aspect of the present invention is a method used to perform processing on target data in which values have been entered for each of a plurality of predetermined data items, We prepare a large-scale language model built using a predetermined machine learning method, which takes text data as input and outputs key words for data processing. Prepare a vectorization model that is constructed using a predetermined machine learning method and outputs vectorized input words. Using the vectorization model described above, the information identifying the multiple data items and the attribute information of each value of the multiple data items are vectorized. Prepare design information data including information on the relationships between the multiple data items, information identifying the multiple vectorized data items, and attribute information of the values of each of the multiple vectorized data items. The user is instructed to input information that identifies the type of processing to the target data and text data that represents the content of the processing. By inputting text data representing the content of the aforementioned process into the large-scale language model, the large-scale language model outputs key words for the process. Key vectors are generated by inputting the words output from the aforementioned large-scale language model into the aforementioned vectorization model. By referring to the aforementioned design information data, the data items and processing content of the target data are identified from the key vector. Perform processing on the identified data item, or output information for performing such processing. It is characterized by the following:
[0088] (Section 7) Another aspect of the present invention is a system used to perform processing on data in which values are entered for each of a plurality of predetermined items, A large-scale language model storage unit stores a large-scale language model that was constructed using a predetermined machine learning method, which receives text data as input and outputs key words for data processing. A vectorization model memory unit stores a vectorization model that is constructed by a predetermined machine learning method and outputs vectorized input words, A design information data storage unit stores design information data including information on the relationships between the plurality of data items, information identifying the plurality of data items vectorized by the vectorization model, and attribute information of the values of each of the plurality of data items vectorized by the vectorization model. A processing key acquisition unit that prompts the user to input information identifying the type of processing to the target data and text data representing the content of the processing, and inputs the text data representing the content of the processing into the large-scale language model, thereby outputting a key word for the processing from the large-scale language model, A key vector generation unit generates key vectors by inputting words output from the large-scale language model into the vectorization model, A processing content identification unit identifies the data items and processing content of the target data from the key vector by referring to the design information data, A processing execution unit that performs processing on the specified data item or outputs information for performing said processing. It is characterized by having the following features.
[0089] (Section 9) Yet another aspect of the present invention is a program used to perform processing on data in which values are entered for each of a plurality of predetermined items, A large-scale language model storage unit stores a large-scale language model that was constructed using a predetermined machine learning method, which receives text data as input and outputs key words for data processing. A vectorization model memory unit stores a vectorization model that is constructed by a predetermined machine learning method and outputs vectorized input words, A design information data storage unit stores design information data including information on the relationships between the plurality of data items, information identifying the plurality of data items vectorized by the vectorization model, and attribute information of the values of each of the plurality of data items vectorized by the vectorization model. In a system having a computer, A processing key acquisition unit that prompts the user to input information identifying the type of processing to the target data and text data representing the content of the processing, and inputs the text data representing the content of the processing into the large-scale language model, thereby outputting a key word for the processing from the large-scale language model, A key vector generation unit generates key vectors by inputting words output from the large-scale language model into the vectorization model, A processing content identification unit identifies the data items and processing content of the target data from the key vector by referring to the design information data, Processing execution unit that performs processing on the specified data items or outputs information for performing such processing. It is characterized by being designed to function as such.
[0090] The data processing method described in paragraph 1, the data processing system described in paragraph 7, and the data processing program described in paragraph 9 are typically used to process data (records) that constitute a database, but they can also be used to process other types of data. In the data processing method according to the present invention, a large-scale language model and a vectorization model are prepared in advance. Furthermore, information that identifies data items and attribute information of the record values entered for each data item are vectorized and combined with information on the relationships between data items to create design information data. The attribute information of values referred to here includes, for example, numerical values, free text, and selection options (those selected in a pull-down format, etc.). Note that the values of the target data may also include Null. Furthermore, information on the relationships between data items includes, for example, information that identifies the common table when multiple data items constitute one table, and information that when a value is entered for one data item, the value of another data item is updated based on that value.
[0091] The user inputs information that identifies the type of processing to be performed on the data, along with text data representing the content of that processing. Examples of data processing types include data retrieval, data item value input, updating, and deletion. The information regarding the type of processing is not limited to direct user input; it may also include indirect input (e.g., identifying the type of processing based on input through a specific screen, or identifying the type of processing based on text data being sent to a specific email address). The input text data is fed into a large-scale language model, which outputs key words for data processing. Examples of key words for data processing include words related to data items such as industry, address, and number of employees, as well as words related to search conditions such as "within one year," "within one month," and "Kyoto Prefecture." Furthermore, a vectorization model is used to vectorize the words output from the large-scale language model, generating key vectors. Then, by referring to design information data, the data item to be processed and the processing content are identified from the key vectors. This method of vectorizing the meaning of words and sentences and identifying similar meanings is called semantic search or vector search. By performing such searches, even if the words output from a large-scale language model, the information identifying the data items, and the attribute information of the values of each data item do not perfectly match, it is possible to identify the target and content of data processing based on the semantic similarity (degree of similarity of vector components) between the two. For example, the target and content of data processing can be identified based on having the vector that is closest to the processing key vector (smallest difference in components, shortest distance between vectors, smallest angle between vectors, etc.).
[0092] Once the data items to be processed and the content of the processing for those data items are determined, the system either executes the specified processing on the identified data items or outputs information for executing the processing. This improves the convenience of processing data in which values have been entered for each of multiple predetermined items.
[0093] (Section 2) The data processing method relating to paragraph 2 is, in the data processing method relating to paragraph 1, The aforementioned large-scale language model has learned a predetermined set of words and their meanings, which are included in a pre-prepared library.
[0094] The data processing method described in paragraph 2 can correctly process abbreviations and colloquialisms commonly used within the industry using a large-scale language model.
[0095] (Section 3) The data processing method relating to paragraph 3 is, in the data processing method relating to paragraph 1 or 2, The aforementioned large-scale language model is located within a private network, along with the aforementioned database.
[0096] The data processing method described in paragraph 3 can prevent confidential internal information of companies and other organizations from being leaked to the outside.
[0097] (Section 4) The data processing method relating to paragraph 4 is, in the data processing method relating to paragraph 3, The aforementioned large-scale language model has further been trained on the aforementioned design information data.
[0098] The data processing method described in paragraph 4 uses a large-scale language model that has learned the structure and attributes of the database, thereby enabling the output of processing keys that can more accurately identify tables and columns.
[0099] (Section 5) The data processing method relating to paragraph 5 is a data processing method relating to any of paragraphs 1 to 4, further comprising: The design information includes trigger information that, when input is made to a predetermined first data item, executes data processing for a predetermined second data item.
[0100] The data processing method described in paragraph 5 allows information entered by the user in the first data item to be automatically reflected in the second data item.
[0101] (Section 6) The data processing method relating to paragraph 6 is, in addition to the data processing method relating to paragraph 5, The design information data includes conversion information for converting the value input to the first data item into an input value for the second data item according to a predetermined standard. When information is entered into the first data item, the input value obtained by converting the entered value based on the conversion information is entered as the value of the second data item.
[0102] The data processing method described in paragraph 6 allows for the automatic determination of importance rankings, accuracy values, and other parameters from text input by the user in natural language and output from a large-scale language model.
[0103] (Section 8) The data processing system relating to paragraph 8 is, in the data processing system relating to paragraph 7, The aforementioned data is stored in a database that is provided independently of the data processing system and operates according to a pre-prepared program. The processing execution unit outputs a command statement in a language executable by the program to perform the processing of the specified content. It is characterized by the following:
[0104] The data processing system described in paragraph 8 can be used by incorporating it into an existing database. [Explanation of Symbols]
[0105] 1…Data processing system 10…Business Systems 11…Sales support system 11D... Customer Management Database 12…Sales Management System 13…Payroll system 14…Human Resources Management System 20...Data Processing Unit 21...Storage section 211...Design information data storage unit 212... Tuning data storage unit 213...Large-scale language model memory 214... Vectorized Model Memory Unit 215... Input form storage unit 31…Design Information Data Creation Department 32...Tuning Execution Unit 33... Processing key acquisition unit 34...Key vector generation unit 35... Processing Content Identification Unit 36... Processing instruction generation unit 37... Processing Execution Unit 51...Input section 52...Display section
Claims
1. A method used to perform processing on target data in which values have been entered for each of several predetermined data items, We prepare a large-scale language model built using a predetermined machine learning method, which takes text data as input and outputs key words for data processing. Prepare a vectorization model that is constructed using a predetermined machine learning method and outputs vectorized input words. Using the vectorization model described above, the information identifying the multiple data items and the attribute information of each value of the multiple data items are vectorized. Prepare design information data including information on the relationships between the multiple data items, information identifying the multiple vectorized data items, and attribute information of the values of each of the multiple vectorized data items. The user is instructed to input information that identifies the type of processing to the target data and text data that represents the content of the processing. By inputting text data representing the content of the aforementioned process into the large-scale language model, the large-scale language model outputs key words for the process. Key vectors are generated by inputting the words output from the aforementioned large-scale language model into the aforementioned vectorization model. By referring to the aforementioned design information data, the data items and processing content of the target data are identified from the key vector. Perform processing on the identified data item, or output information for performing such processing. A data processing method characterized by the following features.
2. The database search method according to claim 1, wherein the large-scale language model has learned predetermined words and their meanings, which are included in a pre-prepared library.
3. The database search method according to claim 1, wherein the large-scale language model is provided within a private network together with the database.
4. The database search method according to claim 3, wherein the large-scale language model has further learned the design information data.
5. moreover, The database search method according to claim 1, wherein the design information includes trigger information that performs data processing on a predetermined second data item when input is made to a predetermined first data item.
6. moreover, The design information data includes conversion information for converting a value input to the first data item into an input value for the second data item according to a predetermined standard. The database search method according to claim 1, wherein when information is entered into the first data item, the input value obtained by converting the entered value based on the conversion information is entered as the value of the second data item.
7. A system used to perform processing on data in which values have been entered for each of several predetermined items, A large-scale language model storage unit stores a large-scale language model that was constructed using a predetermined machine learning method, which receives text data as input and outputs key words for data processing. A vectorization model memory unit stores a vectorization model that is constructed by a predetermined machine learning method and outputs vectorized input words, A design information data storage unit stores design information data including information on the relationships between the plurality of data items, information identifying the plurality of data items vectorized by the vectorization model, and attribute information of the values of each of the plurality of data items vectorized by the vectorization model. A processing key acquisition unit that prompts the user to input information identifying the type of processing to the target data and text data representing the content of the processing, and inputs the text data representing the content of the processing into the large-scale language model, thereby outputting a key word for the processing from the large-scale language model, A key vector generation unit generates key vectors by inputting words output from the large-scale language model into the vectorization model, A processing content identification unit identifies the data items and processing content of the target data from the key vector by referring to the design information data, A processing execution unit that performs processing on the specified data item or outputs information for performing said processing. A data processing system characterized by comprising the following features.
8. The aforementioned data is stored in a database that is provided independently of the data processing system and operates according to a pre-prepared program. The processing execution unit outputs a command statement in a language executable by the program to perform the processing of the specified content. The data processing system according to claim 7, characterized in that...
9. A program used to perform processing on data in which values have been entered for each of several predetermined items, A large-scale language model storage unit stores a large-scale language model that was constructed using a predetermined machine learning method, which receives text data as input and outputs key words for data processing. A vectorization model memory unit stores a vectorization model that is constructed by a predetermined machine learning method and outputs vectorized input words, A design information data storage unit stores design information data including information on the relationships between the plurality of data items, information identifying the plurality of data items vectorized by the vectorization model, and attribute information of the values of each of the plurality of data items vectorized by the vectorization model. In a system having a computer, A processing key acquisition unit that prompts the user to input information identifying the type of processing to the target data and text data representing the content of the processing, and inputs the text data representing the content of the processing into the large-scale language model, thereby outputting a key word for the processing from the large-scale language model, A key vector generation unit generates key vectors by inputting words output from the large-scale language model into the vectorization model, A processing content identification unit identifies the data items and processing content of the target data from the key vector by referring to the design information data, Processing execution unit that performs processing on the specified data items or outputs information for performing such processing. A data processing program characterized by its function as such.
Citation Information
Patent Citations
Resource defining system
JP1993120100A
Intelligent interactive method, device, electronic apparatus and storage medium
JP2022091986A