Natural language generation device, historical data generation device, natural language generation method, and program

Abstract

To provide technology that enables the verbalization of the context behind a person's actions, using a low-cost and rich amount of information. [Solution] A natural language generation device comprising a control unit that generates natural language indicating the attributes of a target person's actions based on historical data showing the target person's action history.

Description

Technical Field

[0001] The present invention relates to the technologies of a natural language generation device, a history data generation device, a natural language generation method, and a program.

Background Art

[0002] Behind the actions of people in various scenes of daily life, there is often a context that serves as the motivation for the action, such as some purpose or intention. Conventionally, research aimed at verbalizing such actions has been conducted. Technologies for understanding the context of actions that involve "movement" among people's actions lead to a wide range of applications, such as navigation in art museums and theme parks and assistance to users with difficulties in movement.

[0003] The verbalization of human activities has been actively pursued in the fusion area of natural language processing and computer vision. Especially in recent research, an approach of learning a large language model (LLM) based on a large number of image-text pairs collected on the web has been mainstream (see, for example, Non-Patent Document 1). On the other hand, when verbalizing human activities for real-world scenes that cannot be obtained from the web, due to concerns about the human cost spent on setting up the measurement environment and the risks of privacy and security violations, it becomes difficult to implement from the stage of measuring and annotating the data required for model learning. To address such issues, in the field of ubiquitous computing, radio wave-based movement measurement technologies using Wi-Fi (registered trademark) and BLE (registered trademark) signals have been studied (see, for example, Non-Patent Document 2).

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

[0005] However, even if the aforementioned movement measurement technology can acquire the location information of a moving person, and even if metadata about the person's surroundings (such as landmarks and products) can be obtained from map information, it is difficult to grasp the purpose or intention behind that person's movement. Furthermore, methods for understanding the context (purpose, intention, etc.) behind such a person's actions often involve high-cost methods such as analyzing data from various perspectives as described above, or conducting surveys.

[0006] In view of the above circumstances, the present invention aims to provide a technology that enables the context behind a person's actions to be verbalized at low cost and with a wealth of information. [Means for solving the problem]

[0007] One aspect of the present invention is a natural language generation device comprising a control unit that generates natural language indicating the attributes of a target person's actions based on historical data showing the history of the target person's actions.

[0008] One aspect of the present invention is the above-described natural language generation device, wherein the history of the actions represents the history of the target person's movements in real space.

[0009] One aspect of the present invention is the above-described natural language generation device, wherein the real space is a space within a predetermined environment in which a plurality of articles are arranged, and the device further comprises a storage unit that stores area information that associates the location of an article arrangement area within the predetermined environment with the attributes of the articles arranged in the article arrangement area, and the control unit generates natural language indicating the attributes of the actions of the target person related to the articles based on the area information and the history data.

[0010] One aspect of the present invention is the above-described natural language generation device, wherein the history of the actions represents the history of the target person's access to websites.

[0011] One aspect of the present invention is the above-described natural language generation device, wherein the history of the behavior includes at least one of the following: the history of items used or purchased by the subject person and the history of services enjoyed by the subject person.

[0012] One aspect of the present invention is the above-described natural language generation device, which includes a control unit that generates historical data indicating the history of a target person's actions based on natural language indicating the attributes of the target person's actions.

[0013] One aspect of the present invention is a natural language generation method in which a computer performs a process to generate natural language that indicates the attributes of a person's actions based on historical data that indicates the history of the person's actions.

[0014] One aspect of the present invention is a program for causing a computer to execute a process of generating a natural language indicating an attribute of the behavior of a target person based on history data indicating the history of the behavior of the target person.

Effects of the Invention

[0015] According to the present invention, it becomes possible to verbalize the context behind a person's behavior at low cost and with a large amount of information.

Brief Description of the Drawings

[0016] [Figure 1] It is a diagram showing an example of the system configuration of the behavior history analysis system 1 of the embodiment. [Figure 2] It is a diagram showing a configuration example of the behavior history analysis apparatus 100 of the embodiment. [Figure 3] It is a diagram showing a configuration example of the learning apparatus 200 of the embodiment. [Figure 4] It is a diagram showing an example of the flow of a process of introducing the description generation model 121 and the behavior history generation model 122 of the embodiment into the behavior history analysis apparatus 100. [Figure 5] It is a diagram (Part 1) for explaining an example of a method of obtaining a description and a behavior history using the learned LLM 211. [Figure 6] It is a diagram (Part 2) for explaining an example of a method of obtaining a description and a behavior history using the learned LLM 211. [Figure 7] It is a diagram (Part 3) for explaining an example of a method of obtaining a description and a behavior history using the learned LLM 211. [Figure 8] It is a diagram (Part 4) for explaining an example of a method of obtaining a description and a behavior history using the learned LLM 211.

Modes for Carrying Out the Invention

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0018] FIG. 1 is a diagram showing an example of the system configuration of the action history analysis system 1 according to the embodiment. The action history analysis system 1 generates a description (an example of natural language) indicating the attributes of the actions of the target person based on the history data indicating the history of the actions of the person to be analyzed (target person), and based on the description (an example of natural language) indicating the attributes of the actions of the target person, generates history data indicating the history of the actions of the target person. Thereby, the user of the action history analysis system 1 can understand the purpose and intention behind the actions from the action history obtained about the target person. Also, the user of the action history analysis system 1 can estimate the action history from the natural language explaining the purpose and intention of the actions of the target person. According to the action history analysis system 1 of the embodiment, a user (for example, a marketing person in a company) can efficiently analyze the actions of the target person.

[0019] The action history analysis system 1 includes, for example, an action history analysis device 100, a learning device 200, and a user terminal device 300. The action history analysis device 100 and the user terminal device 300 can communicate with each other via a network NW. The network NW may be a network using wireless communication or a network using wired communication. The network NW may be configured using, for example, the Internet or a local area network (LAN). The network NW may be configured by combining a plurality of networks.

[0020] In this embodiment, the learning device 200 is assumed to exist stand-alone and include an input means for inputting data necessary for generating an analysis model and an output means for supplying the generated analysis model to the action history analysis device 100. The learning device 200 may be connected to the network NW and configured to communicate with the action history analysis device 100 and the user terminal device 300 via the network NW.

[0021] The user terminal device 300 is a terminal device used by user U of the behavioral history analysis system 1. For example, the user terminal device 300 may be a smartphone, tablet, personal computer, or other terminal device. A user interface operates on the user terminal device 300. By operating the user interface, user U can input data (history data) indicating the behavioral history of a target person to the behavioral history analysis device 100, and can also obtain an explanatory text indicating the attributes of the behavior from the behavioral history analysis device 100 as an analysis result of the behavioral history. In addition, by operating the user interface, user U can input data (explanatory text) indicating the attributes of the behavior of a target person to the behavioral history analysis device 100, and can also obtain a history of the behavior from the behavioral history analysis device 100 as an analysis result of the explanatory text. The user interface may be a dedicated application program or a web application provided via a web browser.

[0022] Figure 2 shows an example configuration of the behavioral history analysis device 100 according to the embodiment. The behavioral history analysis device 100 comprises, for example, a communication unit 110, a storage unit 120, and a control unit 130. The control unit 130 is configured using, for example, a processor such as a CPU (Central Processing Unit) and memory. The control unit 130 functions when the processor executes a program. Note that all or part of the functions of the control unit 130 may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, semiconductor storage devices (e.g., SSDs: Solid State Drives), and storage devices such as hard disks and semiconductor storage devices built into computer systems. The above program may be transmitted via a telecommunications line.

[0023] The communication unit 110 is a communication device that connects the behavioral history analysis device 100 to the network NW. The communication unit 110 is, for example, a network interface. The network interface may be a device that performs wireless communication or a device that performs wired communication.

[0024] The memory unit 120 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The memory unit 120 stores, for example, an explanatory text generation model 121 and an action history generation model 122. The explanatory text generation model 121 is a trained model that takes the action history of a target person as input and outputs an explanatory text of that action. The action history generation model 122 is a trained model that takes the explanatory text of a target person's action as input and outputs a history of that action. The explanatory text generation model 121 and the action history generation model 122 may be configured as a single cross-modal trained model. For example, the explanatory text generation model 121 and the action history generation model 122 are generated by fine-tuning an existing trained LLM (Large Language Model) with training data (training data) generated by collecting diverse action patterns and explanatory texts of people. The explanatory text generation model 121 and the action history generation model 122 are generated by the learning device 200 and stored in the memory unit 120 in advance.

[0025] The control unit 130 receives a request from the user terminal device 300 for analysis regarding the target person's behavior, executes the analysis process, and supplies the results of the analysis process to the user terminal device 300. More specifically, the control unit 130 receives the target person's behavior history along with a request for analysis of the behavior history, inputs the behavior history into the explanatory text generation model 121, and obtains an explanatory text for the behavior as its output. The control unit 130 also receives the explanatory text for the target person's behavior along with a request for analysis of the explanatory text, inputs the explanatory text into the behavior history generation model 122, and obtains the behavior history for the behavior as its output.

[0026] Figure 3 shows an example configuration of the learning device 200 according to the embodiment. The learning device 200 comprises, for example, a storage unit 210 and a control unit 220. The control unit 220 is configured using, for example, a processor such as a CPU (Central Processing Unit) and memory. The control unit 220 functions as an explanation text generation unit 221, an action history generation unit 222, a learning data generation unit 223, and a model generation unit 224 when the processor executes a program. Note that all or part of the functions of the control unit 220 may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, semiconductor storage devices (e.g., SSDs: Solid State Drives), and storage devices such as hard disks and semiconductor storage devices built into computer systems. The above program may be transmitted via a telecommunications line.

[0027] The memory unit 210 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The memory unit 210 stores an existing trained LLM 211, an explanatory text generation model 212, and an action history generation model 213. For example, ChatGPT may be used as the trained LLM 211. The explanatory text generation model 212 and the action history generation model 213 are trained models generated by the control unit 220. The generated explanatory text generation model 212 and the action history generation model 213 are supplied to the action history analysis device 100 and stored in the memory unit 120 as explanatory text generation model 121 and action history generation model 122.

[0028] The control unit 220 generates training data for generating the explanatory text generation model 212 and the behavioral history generation model 213 using machine learning, and generates the explanatory text generation model 212 and the behavioral history generation model 213 using machine learning with the generated training data. More specifically, the control unit 220 comprises an explanatory text generation unit 221, a behavioral history generation unit 222, a training data generation unit 223, and a model generation unit 224.

[0029] The description generation unit 221 generates description texts as training data using an existing pre-trained LLM. The action history generation unit 222 mechanically generates action histories that correspond to the description texts generated by the description generation unit 221. The training data generation unit 223 generates a large amount of training data by generating a large number of description text and action history pairs based on the description texts generated by the description generation unit 221 and the action history generated by the action history generation unit 222. The model generation unit 224 generates the description text generation model 212 and the action history generation model 213 by performing machine learning using the training data generated by the training data generation unit 223.

[0030] Figure 4 shows an example of the processing flow for introducing the description generation model 121 and the behavior history generation model 122 of the embodiment into the behavior history analysis device 100. First, the description generation unit 221 of the learning device 200 acquires a description of the behavior history using the trained LLM 211 (S101). Next, the behavior history generation unit 222 of the learning device 200 mechanically generates a behavior history corresponding to the description acquired in S101 using the trained LLM 211 (S102). More specifically, the behavior history generation unit 222 instructs the trained LLM 211 to generate an abstract behavior plan that is consistent with the acquired description.

[0031] Figures 5 to 8 illustrate an example of how to obtain descriptive text and behavioral history using a trained LLM211. For example, consider a situation where a person performs an activity involving movement (e.g., shopping) in a supermarket (an example of an environment with a specific purpose). In this embodiment, a descriptive text is generated that verbalizes the context behind the movement, based on the semantic information of the environment and the movement trajectory of the person.

[0032] The environment in which a person moves is defined by a set of items (objects) C = {c1, ..., cN} present within it (for example, products displayed in a supermarket and their categories). On the other hand, the trajectory of a person's movement is expressed as a sequence of items present in the vicinity P = {p1, ..., pT}, pt ∈ C (t = 1, ..., T). For example, a movement trajectory in a supermarket can be expressed in the form [fruit, vegetable, vegetable, meat, meat, ...]. Furthermore, it is assumed that the set of items that the person actually touched and selected (contact item set) C' ⊂ P from the set of items included in P is known separately from P. In practice, the item sequence P can be mechanically obtained by matching the location information of a person measured by any positioning technology with map information of the environment. The item sequence P may be associated with the attributes and placement locations (object placement areas) of each item.

[0033] Furthermore, by utilizing behavioral recognition based on movement information or metadata such as a person's purchase history, it is also possible to estimate the contact items C. As a descriptive text for the movement trajectory, consider a concise paragraph that includes context such as the attributes, intentions, and motivations of the person moving, such as, "They value product quality and compare various products to purchase the best one. They purchase small quantities at a time and visit stores frequently." Computationally, the descriptive text can be represented as a token sequence S=(s1,…,sK) transformed by an existing tokenizer. In this case, verbalizing the movement trajectory can be defined as a task of generating the descriptive text S based on the movement trajectory P and the set of contact items C'.

[0034] Here, Figures 5 and 6 show examples of instructions given to the trained LLM 211 to generate an explanatory text S about the context of a movement trajectory. For example, the explanatory text generation unit 221 can obtain an explanatory text S (customer intent) about various actions a customer might take in a supermarket by inputting an instruction text containing roles, instructions, rules, and examples, as shown in Figures 5 and 6, to the prompt of the trained LLM 211.

[0035] Figure 7 shows an example of instructions given to the trained LLM 211 to generate an abstract action plan (action history) consistent with the explanatory text S. For example, the action history generation unit 222 can obtain an action history (action plan such as a shopping plan) that is in line with the customer's intentions by inputting instructions, including roles, instructions, rules, customer intentions (explanatory text), and a list of product categories, as shown in Figure 7, into the prompts of the trained LLM 211.

[0036] Figure 8 shows an example of instructions given to the trained LLM 211 to generate the item candidate C and contact item set C' that constitute the movement trajectory P. For example, the behavior history generation unit 222 can obtain a behavior history (movement trajectory) that corresponds to the customer's intention by inputting instruction sentences, including roles, instructions, and policies as shown in Figure 8, into the prompts of the trained LLM 211. For example, the behavior history generation unit 222 can generate a movement trajectory P by randomly sampling items from the item candidate C obtained by the instructions in Figure 8.

[0037] Next, the learning data generation unit 223 of the learning device 200 paraphrases and diversifies the explanatory text S acquired in S101 (S103). Then, the learning data generation unit 223 constructs a larger learning dataset by expanding the combination of movement trajectories and their explanatory texts by the number of paraphrases (S104).

[0038] Next, the model generation unit 224 of the learning device 200 fine-tunes the trained LLM 211 using the dataset generated in S101 to S103 to generate an explanatory text generation model 212 and an action history generation model 213 (S105). For example, the model generation unit 224 learns a language model that takes a movement trajectory P and a set of contact items C' as input and outputs an explanatory text S. More specifically, the model generation unit 224 converts P and C' into sentences that can be input to the language model according to certain rules. The model generation unit 224 learns the task of outputting an explanatory text S by using these input sentences as input to sequence transformation models such as T5. For example, as a simple example, it is conceivable to convert P=(p1,…,pT) and C'={c1,…,cM} into the format “Trajectory is Stay at p1.…Stay at pT. And customer purchases item list is [“c1”,…,“cM”]”. Then, the explanatory text generation model 212 and the behavioral history generation model 213 generated in S105 are introduced into the behavioral history analysis device 100 (S106).

[0039] According to the behavioral history analysis system 1 of the embodiment described above, the context behind a person's actions can be verbalized at low cost and with a wealth of information.

[0040] <Variation> In the above embodiment, we described a case where the behavioral analysis system 1 represents the behavioral history of a person shopping in a supermarket by the permutation of products present in the supermarket. However, the space to be analyzed and the method of representing behavioral history by the behavioral analysis system 1 are not limited to this. The space to be analyzed is not limited to a specific environment; it can be any environment in which items capable of representing a person's behavioral history are placed. Furthermore, the items placed in the environment can be of any use, purpose, size, or shape, as long as they are capable of representing a person's behavioral history. For example, if the environment is an art museum or museum, the items may be exhibits or stored items within the museum. Also, for example, if the environment is a library, the items may be books or other items held in the library. Also, for example, if the environment is a commercial facility, the items may be goods displayed within the facility. Also, for example, if the environment is an event venue, the items may be things displayed within the venue, equipment, or locations. The purpose for which items are selected in the target environment may be arbitrary. For example, the purpose of selecting items may be for use, purchase, transfer, etc.

[0041] In the above embodiment, a case was described in which a movement trajectory P is generated by randomly sampling an item from candidate items C present in the environment in which a person moves. However, the manner in which the movement trajectory is represented is not limited to this. For example, the movement trajectory may be represented as time-series data of the coordinate information of the person's position. In this case, for example, the movement trajectory may be represented in the form of [(t=0,(x,y)=(10,20)), (t=1,(x,y)=(15,25)), ..., (t=n,(x,y)=(60,110))].

[0042] Furthermore, the movement trajectory may be represented as time-series data recording the time or order in which specific events occurred. For example, the movement trajectory can be represented as time-series data combining the checkpoint locations in an event and the time at which each checkpoint was passed. In this case, for example, the movement trajectory may be represented in the format [(20:10, passed point A), (20:50, passed point B), (21:30, passed point C)]. Alternatively, the movement trajectory may be represented even more simply as data representing the order in which the checkpoint locations were visited. For example, if locations A, B, and C were visited in that order, the movement trajectory may be represented in the format [location A, location B, location C].

[0043] Furthermore, the movement trajectory may be represented as data showing the history of actions (browsing history) within a specific website, with actions within that website treated as specific events. For example, if the specific website is an e-commerce site, the history may be represented in the format of [visit to e-commerce site, top page, category selection page, beverage page, click "add to cart", ...], starting from a visit to that e-commerce site. In other words, the history of actions may be represented by the history of destinations accessed by the subject on the web. Alternatively, the history of actions may be represented by the history of services enjoyed by the subject.

[0044] While embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Industrial applicability]

[0045] This invention is applicable to applications that support the creation of moving images having a predetermined presentation effect. [Explanation of Symbols]

[0046] 1. Behavioral History Analysis System 100 Behavioral History Analysis Device 110 Communications Department 120 Storage section 121 Description Generation Model 122 Behavioral History Generation Model 130 Control Unit 200 Learning Devices 210 Storage section 211 Pre-trained LLM 212 Description Generation Models 213 Behavioral History Generation Model 220 Control Unit 221 Description Generation Unit 222 Action History Generation Unit 223 Training Data Generation Unit 224 Model Generation Unit 300 User terminal devices

Claims

1. The system includes a control unit that generates natural language indicating the attributes of a person's actions based on historical data showing the person's behavioral history. Natural language generator.

2. The aforementioned history of actions represents the history of the subject person's movement in real space. The natural language generation device according to claim 1.

3. The aforementioned physical space is a space within a predetermined environment in which multiple articles are arranged. The system further includes a storage unit that stores area information relating the location of an item placement area within the predetermined environment to the attributes of the items placed in the item placement area. The control unit generates natural language indicating the attributes of the actions of the person concerned related to the item, based on the area information and the history data. The natural language generation device according to claim 2.

4. The aforementioned history of actions represents the history of the target person's web access. The natural language generation device according to claim 1.

5. The history of the aforementioned actions includes at least one of the following: the history of items used or purchased by the person concerned, and the history of services enjoyed by the person concerned. The natural language generation device according to claim 1.

6. A history data generation device comprising a control unit that generates history data indicating the history of a person's actions based on natural language indicating the attributes of the person's actions.

7. Computers Based on historical data showing the history of the target person's actions, a process is performed to generate natural language that indicates the attributes of the target person's actions. Natural language generation methods.

8. On the computer, To execute a process that generates natural language indicating the attributes of a person's actions based on historical data showing the history of the person's actions, program.

Images