Natural language generation device, history data generation device, natural language generation method, and program

Abstract

To provide a technology that enables the context behind a person's behavior to be verbalized at low cost with a rich amount of information as a background. [Solution] A natural language generation device includes a control unit that generates natural language indicating attributes of the behavior of a target person based on history data indicating the behavior history of the target person.

Description

[Technical Field]

[0001] The present invention relates to a natural language generation device, a history data generation device, a natural language generation method, and a program technology. [Background technology]

[0002] Behind human behavior in various situations in daily life, there is often a context that motivates the behavior, such as some kind of purpose or intention. Previous research has aimed to verbalize such behavior. Technology that can understand the context of human behavior that involves "movement" could have a wide range of applications, such as navigation in museums and theme parks, or assistance for users who have difficulty moving around.

[0003] Verbalization of human activities is an active area of ​​research focused on the intersection of natural language processing and computer vision. In particular, recent research has focused on training large language models (LLMs) based on a large number of image-text pairs collected from the web (see, for example, Non-Patent Document 1). However, when attempting to verbalize human activities from real-world scenes not available on the web, the personnel costs required to set up the measurement environment and concerns about privacy and security risks make it difficult to even begin measuring and annotating the data required for model training. To address these challenges, radio wave-based mobile measurement technologies using Wi-Fi (registered trademark) and BLE (registered trademark) signals are being researched in the ubiquitous computing field (see, for example, Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundational models defining a new era in vision: A survey and outlook. ArXiv e-prints (arXiv:2307.13721), 2023. [Non-patent document 2] Faheem Zafari, Athanasios Gkelias, and Kin K Leung. A survey of indoor localization systems and technologies. IEEE Communications Surveys & Tutorials, Vol. 21, No. 3, pp. 2568-2599, 2019. Summary of the Invention [Problem to be solved by the invention]

[0005] However, even if the above-mentioned movement measurement technology can acquire the location information of a moving person and meta-information (landmarks, products, etc.) around that person can be acquired from map information, etc., it is difficult to understand the purpose and intention of that person's movement. Furthermore, the means for understanding the context (purpose, intention, etc.) behind such a person's behavior often involve high-cost methods such as analyzing data from the above-mentioned various perspectives or conducting questionnaire surveys.

[0006] In view of the above circumstances, the present invention aims to provide a technology that enables the context behind a person's behavior to be verbalized at low cost and with a large amount of information as a background. [Means for solving the problem]

[0007] One aspect of the present invention is a natural language generation device including a control unit that generates natural language indicating attributes of a target person's behavior based on history data indicating the target person's behavior history.

[0008] One aspect of the present invention is the natural language generation device described above, wherein the behavior history represents a history of movement of the target person in real space.

[0009] One aspect of the present invention is the above-mentioned natural language generation device, wherein the real space is a space within a predetermined environment in which a plurality of items are placed, and the device further includes a memory unit that stores area information that associates the position of an item placement area within the predetermined environment with the attributes of the items placed in the item placement area, and the control unit generates natural language that indicates the attributes of the target person's behavior related to the items based on the area information and the history data.

[0010] One aspect of the present invention is the natural language generation device described above, wherein the behavior history represents a history of the target person's access destinations on the web.

[0011] One aspect of the present invention is the above-mentioned natural language generation device, wherein the behavioral history includes at least one of a history of items used or purchased by the target person and a history of services enjoyed by the target person.

[0012] One aspect of the present invention is the above-mentioned natural language generation device, which includes a control unit that generates history data indicating the behavioral history of a target person based on natural language indicating attributes of the target person's behavior.

[0013] One aspect of the present invention is a natural language generation method in which a computer performs a process of generating natural language indicating attributes of a target person's behavior based on history data indicating the history of the target person's behavior.

[0014] One aspect of the present invention is a program for causing a computer to execute a process of generating natural language indicating attributes of a target person's behavior based on historical data indicating the target person's behavior history. [Effects of the Invention]

[0015] The present invention makes it possible to verbalize the context behind a person's behavior at low cost and with a rich amount of information as a background. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating an example of a system configuration of a behavior history analysis system 1 according to an embodiment. [Figure 2] 1 is a diagram illustrating an example of the configuration of a behavior history analysis device 100 according to an embodiment. [Figure 3] FIG. 2 illustrates an example of the configuration of a learning device 200 according to an embodiment. [Figure 4] FIG. 2 is a diagram showing an example of a processing flow for introducing an explanatory sentence generation model 121 and a behavior history generation model 122 according to the embodiment into the behavior history analysis device 100. [Figure 5] This is a diagram (part 1) explaining an example of a method for obtaining explanatory text and behavioral history using the trained LLM211. [Figure 6] This is a diagram (part 2) explaining an example of a method for obtaining explanatory text and behavioral history using the trained LLM211. [Figure 7] This is a diagram (part 3) explaining an example of a method for obtaining explanatory text and behavioral history using the trained LLM211. [Figure 8] This is a diagram (part 4) explaining an example of a method for obtaining explanatory text and behavioral history using the trained LLM211. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0018] FIG. 1 is a diagram illustrating an example of a system configuration of a behavior history analysis system 1 according to an embodiment. The behavior history analysis system 1 generates explanatory text (an example of natural language) indicating attributes of the behavior of a person to be analyzed (a target person) based on history data indicating the behavioral history of the target person, and generates history data indicating the behavioral history of the target person based on the explanatory text (an example of natural language) indicating the attributes of the behavior of the target person. This allows a user of the behavior history analysis system 1 to understand the purpose and intention behind the behavior of the target person from the behavioral history obtained for the target person. Furthermore, the user of the behavior history analysis system 1 can infer the behavioral history of the target person from the natural language explaining the purpose and intention of the behavior. The behavior history analysis system 1 according to an embodiment allows a user (e.g., a marketing person in a company) to efficiently analyze the behavior of the target person.

[0019] The behavior history analysis system 1 includes, for example, a behavior history analysis device 100, a learning device 200, and a user terminal device 300. The behavior history analysis device 100 and the user terminal device 300 can communicate with each other via a network NW. The network NW may be a network using wireless communication or a network using wired communication. The network NW may be configured using, for example, the Internet or a local area network (LAN). The network NW may also be configured by combining multiple networks.

[0020] In this embodiment, the learning device 200 exists as a standalone device and includes an input means for inputting data necessary for generating an analytical model and an output means for supplying the generated analytical model to the behavior history analysis device 100. The learning device 200 may be configured to be connected to a network NW and to be able to communicate with the behavior history analysis device 100 and the user terminal device 300 via the network NW.

[0021] The user terminal device 300 is a terminal device used by a user U of the behavior history analysis system 1. For example, the user terminal device 300 may be a terminal device such as a smartphone, a tablet, or a personal computer. A user interface operates on the user terminal device 300. By operating the user interface, the user U can input data (history data) indicating the behavior history of a target person to the behavior history analysis device 100 and obtain an explanatory text indicating the attributes of the behavior from the behavior history analysis device 100 as an analysis result of the explanatory text. Furthermore, by operating the user interface, the user U can input data (explanatory text) indicating the attributes of the behavior of the target person to the behavior history analysis device 100 and obtain the behavior history from the behavior history analysis device 100 as an analysis result of the explanatory text. The user interface may be a dedicated application program or a web application provided via a web browser.

[0022] FIG. 2 is a diagram illustrating an exemplary configuration of a behavior history analysis device 100 according to an embodiment. The behavior history analysis device 100 includes, for example, a communication unit 110, a storage unit 120, and a control unit 130. The control unit 130 is configured using, for example, a processor such as a CPU (Central Processing Unit) and a memory. The control unit 130 functions when the processor executes a program. Note that all or part of the functions of the control unit 130 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a semiconductor storage device (e.g., a solid-state drive (SSD)), as well as storage devices such as a hard disk or semiconductor storage device built into a computer system. The program may be transmitted via a telecommunications line.

[0023] The communication unit 110 is a communication device that connects the behavior history analysis device 100 to the network NW. The communication unit 110 is, for example, a network interface. The network interface may be a device that performs wireless communication or a device that performs wired communication.

[0024] The storage unit 120 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 120 stores, for example, a description generation model 121 and a behavior history generation model 122. The description generation model 121 is a trained model trained to receive a behavior history of a target person and output a description of the behavior. The behavior history generation model 122 is a trained model trained to receive a description of the behavior of the target person and output the history of the behavior. The description generation model 121 and the behavior history generation model 122 may be configured as a single cross-modal trained model. For example, the description generation model 121 and the behavior history generation model 122 are generated by fine-tuning an existing trained large language model (LLM) using training data (teacher data) generated by collecting various behavior patterns and descriptions of people. The description generation model 121 and the behavior history generation model 122 are generated by the learning device 200 and stored in advance in the storage unit 120.

[0025] The control unit 130 receives a request for analysis of the target person's behavior from the user terminal device 300, executes analysis processing, and provides the results of the analysis processing to the user terminal device 300. More specifically, the control unit 130 receives a request for analysis of the behavior history of the target person together with the behavior history, inputs the behavior history to the explanation generation model 121, and obtains an explanation of the behavior as its output. The control unit 130 also receives a request for analysis of the explanation of the target person's behavior together with the explanation, inputs the explanation to the behavior history generation model 122, and obtains the behavior history of the behavior as its output.

[0026] FIG. 3 is a diagram illustrating an exemplary configuration of a learning device 200 according to an embodiment. The learning device 200 includes, for example, a storage unit 210 and a control unit 220. The control unit 220 is configured using, for example, a processor such as a CPU (Central Processing Unit) and a memory. The control unit 220 functions as an explanatory sentence generation unit 221, a behavioral history generation unit 222, a learning data generation unit 223, and a model generation unit 224 when the processor executes a program. Note that all or part of the functions of the control unit 220 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a semiconductor storage device (e.g., a solid-state drive (SSD)), as well as storage devices such as a hard disk or semiconductor storage device built into a computer system. The above program may be transmitted via a telecommunications line.

[0027] The storage unit 210 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 210 stores an existing trained LLM 211, an explanation generation model 212, and a behavior history generation model 213. For example, ChatGPT may be used as the trained LLM 211. The explanation generation model 212 and the behavior history generation model 213 are trained models generated by the control unit 220. The generated explanation generation model 212 and behavior history generation model 213 are supplied to the behavior history analysis device 100 and stored in the storage unit 120 as the explanation generation model 121 and the behavior history generation model 122.

[0028] The control unit 220 generates learning data for generating the explanatory sentence generation model 212 and the behavior history generation model 213 by machine learning, and generates the explanatory sentence generation model 212 and the behavior history generation model 213 by machine learning using the generated learning data. More specifically, the control unit 220 includes an explanatory sentence generation unit 221, a behavior history generation unit 222, a learning data generation unit 223, and a model generation unit 224.

[0029] The explanatory text generation unit 221 generates explanatory text as training data using an existing trained LLM. The behavioral history generation unit 222 mechanically generates a behavioral history in accordance with the explanatory text generated by the explanatory text generation unit 221. The training data generation unit 223 generates a large amount of training data by generating a large number of pairs of explanatory text and behavioral history based on the explanatory text generated by the explanatory text generation unit 221 and the behavioral history generated by the behavioral history generation unit 222. The model generation unit 224 generates the explanatory text generation model 212 and the behavioral history generation model 213 by performing machine learning using the training data generated by the training data generation unit 223.

[0030] 4 is a diagram showing an example of a processing flow for introducing the explanatory sentence generation model 121 and the behavior history generation model 122 of the embodiment into the behavior history analysis device 100. First, the explanatory sentence generation unit 221 of the learning device 200 acquires an explanatory sentence of the behavior history using the trained LLM 211 (S101). Next, the behavior history generation unit 222 of the learning device 200 mechanically generates a behavior history in accordance with the explanatory sentence acquired in S101 using the trained LLM 211 (S102). More specifically, the behavior history generation unit 222 instructs the trained LLM 211 to generate an abstract behavior plan that is consistent with the acquired explanatory sentence.

[0031] 5 to 8 are diagrams illustrating an example of a method for acquiring explanatory text and behavioral history using the trained LLM 211. For example, consider a situation in which a person is performing an activity (e.g., shopping) that involves movement in a supermarket (an example of an environment where a specific purpose is assumed). In this embodiment, an explanatory text that verbalizes the context behind the movement is generated based on semantic information of the environment and the movement trajectory of the person.

[0032] The environment in which a person moves is defined by a set of items (objects) C = {c1,...,cN} (e.g., products and their categories in a supermarket). The trajectory of a person's movement is expressed as a sequence P = {p1,...,pT}, pt∈C(t=1,...,T) of nearby items. For example, a trajectory in a supermarket can be expressed in the form [fruit, vegetable, vegetable, meat, meat,...]. Furthermore, we assume that the set of items that the person actually touched or selected (contact item set) C' ⊂ P, among the set of items included in P, is known separately from P. In practice, the item sequence P can be automatically obtained by matching the person's location information measured using any positioning technology with the map information of the environment. The item sequence P may be associated with the attributes and location (item placement area) of each item.

[0033] Furthermore, by utilizing behavioral recognition based on movement information or metadata such as a person's purchase history, it is also possible to estimate the items C that the person has interacted with. Consider a brief paragraph describing a movement trajectory, such as "The person values ​​product quality, compares various products, and tries to purchase the best one. The amount of purchases is small, and the person visits the store frequently." Computationally, the description can be expressed as a token sequence S = (s1, ..., sK) converted by an existing tokenizer. In this case, verbalizing a movement trajectory can be defined as the task of generating the description S based on the movement trajectory P and the set of interacting items C'.

[0034] 5 and 6 are diagrams showing examples of instructions to be given to the trained LLM 211 to generate an explanatory text S relating to the context of a movement trajectory. For example, the explanatory text generation unit 221 can obtain explanatory text S (customer intentions) about various actions that a customer may take in a supermarket by inputting instructions including roles, instructions, rules, and examples as shown in FIGS. 5 and 6 into the prompts of the trained LLM 211.

[0035] 7 is a diagram showing an example of instructions to be given to the trained LLM 211 to generate an abstract behavior plan (behavior history) consistent with the explanatory text S. For example, the behavior history generation unit 222 can acquire a behavior history (behavior plan such as a shopping plan) that is consistent with the customer's intention by inputting instructions including roles, instructions, rules, customer intentions (explanatory text), and a list of product categories, as shown in FIG. 7, into the prompt of the trained LLM 211.

[0036] 8 is a diagram showing an example of instructions to be given to the trained LLM 211 to generate the candidate item C and contact item set C' that make up the trajectory P. For example, the behavior history generation unit 222 can acquire a behavior history (trajectory) according to the customer's intention by inputting an instruction statement including a role, instruction, and policy as shown in FIG. 8 into the prompt of the trained LLM 211. For example, the behavior history generation unit 222 can generate the trajectory P by randomly sampling items from the candidate item C obtained by the instruction in FIG. 8.

[0037] Next, the learning data generation unit 223 of the learning device 200 diversifies the explanatory sentence S acquired in S101 by paraphrasing it (S103).Then, the learning data generation unit 223 constructs a larger-scale learning dataset by expanding the combinations of the trajectories and their explanatory sentences by the number of paraphrases (S104).

[0038] Next, the model generation unit 224 of the model of the learning device 200 generates an explanation generation model 212 and an action history generation model 213 by fine-tuning the trained LLM 211 using the dataset generated in S101 to S103 (S105). For example, the model generation unit 224 receives a trajectory P and a contact item set C' as input and trains a language model that outputs an explanation S. More specifically, the model generation unit 224 converts P and C' into sentences that can be input to the language model according to certain rules. The model generation unit 224 uses this input sentence as input to a sequence transformation model such as T5 to train the task of outputting an explanation S. For example, as a simple example, it is conceivable to convert P = (p1, ..., pT), C' = {c1, ..., cM} into the format "Trajectory is Stay at p1. ... Stay at pT. And customer purchases item list is ["c1", ..., "cM"]". Then, the explanation generation model 212 and the behavior history generation model 213 generated in S105 are introduced into the behavior history analysis device 100 (S106).

[0039] According to the behavior history analysis system 1 of the embodiment described above, the context behind a person's behavior can be verbalized at low cost and with a rich amount of information as the background.

[0040] <Modification> In the above embodiment, the behavior analysis system 1 has been described as representing the behavior history of a person shopping at a supermarket by the permutation of products in the supermarket. However, the space to be analyzed by the behavior analysis system 1 and the method of representing the behavior history are not limited to this. The space to be analyzed is not limited to a specific environment, and may be any environment space as long as it is a space in which objects capable of representing the person's behavior history are arranged. Furthermore, the objects arranged in the environment may have any use, purpose, size, shape, etc., as long as they are capable of representing the person's behavior history. For example, if the environment is an art gallery or museum, the objects may be exhibits or stored items within the museum. For example, if the environment is a library, the objects may be books, etc., held in the library. For example, if the environment is a commercial facility, the objects may be products displayed within the facility. For example, if the environment is an event venue, the objects may be objects displayed within the venue, equipment, or a location. The purpose for selecting an item in the target environment may be arbitrary. For example, the purpose for selecting an item may be for use, purchase, transfer, etc.

[0041] In the above embodiment, a case has been described in which a movement trajectory P is generated by randomly sampling items from candidate items C present in the environment in which a person moves, but the manner in which a movement trajectory is represented is not limited to this. For example, the movement trajectory may be represented as time-series data of coordinate information of the person's position. In this case, for example, the movement trajectory may be represented in a format such as [(t=0,(x,y)=(10,20)), (t=1,(x,y)=(15,25)), ..., (t=n,(x,y)=(60,110))].

[0042] Furthermore, a movement trajectory may be expressed as time-series data that records the time or order in which a specific event occurred. For example, a movement trajectory can be expressed as time-series data that combines checkpoint locations in an event with the time at which the checkpoint locations were passed. In this case, for example, the movement trajectory may be expressed in a format such as [(20:10, passed point A), (20:50, passed point B), (21:30, passed point C)]. Furthermore, a movement trajectory may be expressed more simply as data indicating the order in which the checkpoint locations were visited. For example, if points A, B, and C were visited in this order, the movement trajectory may be expressed in a format such as [point A, point B, point C].

[0043] Furthermore, the movement trajectory may be expressed as data representing a behavioral history (traffic history) of a specific website, with behavior within the specific website being a specific event. For example, if the specific website is an e-commerce site, the movement trajectory may be expressed in the form of [e-commerce site visit, top page, category selection page, beverage page, clicking "add to cart," ...], starting from a visit to the e-commerce site. In other words, the behavioral history may be represented by a history of the destinations accessed by the target person on the web. In another aspect, the behavioral history may be represented by a history of the services enjoyed by the target person.

[0044] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Industrial Applicability]

[0045] The present invention is applicable to applications that support the creation of moving images having a predetermined presentation effect. [Explanation of symbols]

[0046] 1. Behavioral history analysis system 100 Behavioral history analysis device 110 Communications Department 120 Storage section 121 Explanatory Text Generation Model 122 Behavioral History Generation Model 130 control section 200 Learning Device 210 Storage section 211 Learned LLM 212 Explanatory Text Generation Model 213 Behavioral History Generation Model 220 Control Unit 221 Description Generation Unit 222 Behavioral History Generation Unit 223 Learning Data Generation Unit 224 Model Generation Unit 300 User terminal device

Claims

1. An acquisition unit that acquires history data indicating a history of a target person's behavior from a storage unit that stores the history data; a control unit that generates an explanatory sentence indicating attributes of the behavior of the target person by inputting the history data into a large-scale language model; Equipped with The large-scale language model is a first step of acquiring an explanatory sentence indicating attributes of the behavior related to the history using an existing large-scale language model; a second step of mechanically generating a history of behavior in accordance with the acquired description; a third step of paraphrasing the narrative using an existing large-scale language model; a fourth step of fine-tuning an existing large-scale language model based on training data in which the behavioral history generated in the second step is associated with the explanatory sentences amplified in the third step; The learning method includes Natural language generator.

2. The fine tuning includes: a large-scale language model is trained to output the explanatory sentences using as input a movement trajectory of the target person and a contact item set, which is a set of items that the target person has come into contact with on the movement trajectory; The natural language generation device according to claim 1 .

3. The behavior history represents a history of movement of the target person in real space. The natural language generation device according to claim 1 .

4. the real space is a space within a predetermined environment in which a plurality of items are arranged, The method further includes a storage unit that stores area information that associates a position of an item placement area in the predetermined environment with an attribute of an item placed in the item placement area, the control unit generates an explanation indicating attributes of the target person's behavior related to the item based on the area information and the history data. The natural language generation device according to claim 3 .

5. The behavior history represents a history of the target person's access destinations on the web. The natural language generation device according to claim 1 .

6. The behavior history includes at least one of a history of items used or purchased by the target person and a history of services received by the target person; The natural language generation device according to claim 1 .

7. an acquisition unit that acquires an explanatory text indicating an attribute of a behavior of a target person from a storage unit that stores the explanatory text; a control unit that generates history data indicating a history of the target person's behavior by inputting the description into a large-scale language model; Equipped with The large-scale language model is a first step of acquiring an explanatory sentence indicating attributes of the behavior related to the history using an existing large-scale language model; a second step of mechanically generating a history of behavior in accordance with the acquired description; a third step of paraphrasing the narrative using an existing large-scale language model; a fourth step of fine-tuning an existing large-scale language model based on training data in which the behavioral history generated in the second step is associated with the explanatory sentences amplified in the third step; The learning method includes Historical data generator.

8. A computer comprising: acquiring history data indicating the behavior history of the target person from a storage unit that stores the history data; generating a description indicating attributes of the target person's behavior by inputting the history data into a large-scale language model; and The large-scale language model is a first step of acquiring an explanatory sentence indicating attributes of the behavior related to the history using an existing large-scale language model; a second step of mechanically generating a history of behavior in accordance with the acquired description; a third step of paraphrasing the narrative using an existing large-scale language model; a fourth step of fine-tuning an existing large-scale language model based on training data in which the behavioral history generated in the second step is associated with the explanatory sentences amplified in the third step; The learning method includes Natural language generation methods.

9. A computer, acquiring history data indicating the behavior history of the target person from a storage unit that stores the history data; generating a description indicating attributes of the target person's behavior by inputting the history data into a large-scale language model; A program for executing The large-scale language model is a first step of acquiring an explanatory sentence indicating attributes of the behavior related to the history using an existing large-scale language model; a second step of mechanically generating a history of behavior in accordance with the acquired description; a third step of paraphrasing the narrative using an existing large-scale language model; a fourth step of fine-tuning an existing large-scale language model based on training data in which the behavioral history generated in the second step is associated with the explanatory sentences amplified in the third step; The learning method includes program.

Images