Information processing device, information processing method, and recording medium

Abstract

This information processing device comprises: a first acquisition means that acquires a moving image; an object identification means that identifies a target from the moving image; a second acquisition means that acquires a target state that is a state of the target; a linguistic expression means that generates a sentence indicating the target state of the target from the target and the target state; and a learning means that causes a large language model to learn the sentence generated by the linguistic expression means.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing method, and a recording medium.

[0002] As an example of this type of device, a device has been proposed that analyzes video captured by a surveillance camera and acquires verbalized data relating to the user's behavior (see Patent Document 1).

[0003] JP 2010-238204 A

[0004] An object of this disclosure is to provide an information processing device, an information processing method, and a recording medium that aim to improve upon the techniques described in prior art documents.

[0005] One aspect of the information processing device includes a first acquisition means for acquiring a moving image, a target identification means for identifying a target from the moving image, a second acquisition means for acquiring a target state that is the state of the target, a verbalization means for generating a sentence indicating the target state related to the target from the target and the target state, and a learning means for training a large-scale language model with the sentence generated by the verbalization means.

[0006] One aspect of the information processing method includes acquiring a video image, identifying an object from the video image, acquiring an object state that is the state of the object, generating a sentence indicating the object state related to the object from the object and the object state, and training the generated sentence into a large-scale language model.

[0007] One aspect of the recording medium has recorded thereon a computer program for causing a computer to execute an information processing method of acquiring a moving image, identifying an object from the moving image, acquiring an object state that is the state of the object, generating a sentence indicating the object state related to the object from the object and the object state, and training a large-scale language model with the generated sentence.

[0008] Fig. 1 is a block diagram showing an example of the configuration of an information processing device; Fig. 2 is a flowchart showing the operation of the information processing device according to the present disclosure; Fig. 3 is a block diagram showing another example of the configuration of an information processing device; Fig. 4 is a block diagram showing another example of the configuration of an information processing device; Fig. 5 is a diagram for explaining person tracking processing; Fig. 6 is a block diagram showing another example of the configuration of an information processing device.

[0009] Examples of such devices include a temperature measuring device that measures the temperature inside a room, and a security device that analyzes images captured by a surveillance camera to detect the intrusion of a suspicious person.

[0010] These devices use information processing to observe real-world events using devices such as temperature sensors and cameras, and then apply the output values ​​to some kind of mathematical model to analyze them and understand the situations and events occurring in the real world.Furthermore, there are also air conditioning devices that control the room temperature based on the temperature they understand, and security devices that notify security companies based on the detection results.

[0011] For example, in a temperature measurement device that uses a thermistor as a temperature sensor, a constant voltage is applied to the thermistor in an electrical circuit to measure the current, and the output value corresponding to the current is input to information processing. Here, thermistors have the characteristic that their resistance changes depending on the temperature, and the resistance value can be calculated from Ohm's law if the voltage and current are known. Therefore, in information processing, the input value corresponding to the current can be applied to a mathematical model to obtain an output corresponding to the real-world temperature. Hereinafter, the variables input to the mathematical model will be called explanatory variables, and the variables output by the mathematical model will be called target variables.

[0012] In the case of a security device that analyzes video captured by a surveillance camera to detect the intrusion of a suspicious individual, the video output by the camera's image sensor is the explanatory variable of the mathematical model, and the value representing the presence or absence of a suspicious individual's intrusion as true or false is the dependent variable. Here, for example, when analyzing one frame of video using a surveillance camera equipped with an image sensor with 2 million pixels, the number of explanatory variables is 2 million and the number of dependent variables is 1. Furthermore, in the case of a security device that detects not only the intrusion of a suspicious individual but also the outbreak of a fire, the number of dependent variables is 2. In this way, the number of explanatory variables and dependent variables handled by a mathematical model is called the number of dimensions.

[0013] When observing and understanding real-world phenomena using an information processing device, due to the nature of information processing, which involves fitting explanatory variables to a mathematical model to obtain a target variable, it is necessary to design a mathematical model in advance to derive the target variable from the explanatory variables and implement it as information processing.As a result, there is an issue that the situations and events that can be understood are limited, and users' expectations are often not met.

[0014] For example, in the case of a temperature measurement device that measures the temperature inside a room, the real-world temperature at the installation location can be understood, but in the case of an air conditioning device that uses this device, users often seek comfort that takes into account not only temperature but also humidity, and a mathematical model that does not implement humidity as an explanatory variable and comfort as a target variable will not meet users' expectations.

[0015] Furthermore, the level of comfort that users require from an air conditioning unit will differ depending on the context of the user's past situation, such as "just returning home from the hot sun" or "exercising indoors." Unless these are also implemented as mathematical models, it will not be possible to control the air conditioning taking these into account.

[0016] As described above, the real world is an open-ended environment with an infinite number of explanatory variables, and user expectations are also an open-ended environment with an infinite number of objective variables. Therefore, a mathematical model of a limited environment in which the explanatory variables and objective variables are limited in advance will not satisfy the diverse expectations of users.

[0017] For example, an information processing device can be considered that inputs image information of a person using an ATM in the real world into information processing, verbalizes and outputs the "motion state of the person" in an image analysis stage in a first stage, and compares the verbalized "motion state of the person" with "pre-specified conditions" in a second stage to detect "whether or not the person is likely to become a victim of a crime." However, even if an attempt is made to respond to the diverse real world by using image information with a huge number of dimensions as the explanatory variable in the first stage and to express diverse motion states by using a language combining nouns and verbs as the objective variable in the first stage, the conditions for comparison in the second stage become a limited environment that is restricted in advance, making it impossible to understand the diverse motion states of people in the real world.

[0018] For example, an information processing device can be considered that inputs real-world image information into information processing, recognizes faces, objects, etc., and verbalizes their states. When outputting the results, the device presents the user with an image of the recognized face or object, along with a verbalized description of its state, thereby promoting the user's understanding of the real world. However, in this device, the objective variables are the image of the recognized face or object itself, and the verbalized description of its state, which allows for a huge number of dimensions and allows for the diverse real world to be handled, but it is humans, not technology, that understand this.

[0019] First Embodiment A first embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to Fig. 1 and Fig. 2. In the following, the first embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 10.

[0020] In FIG. 1 , the information processing device 10 includes a first acquisition unit 11 , a target identification unit 12 , a second acquisition unit 13 , a languageization unit 14 , and a learning unit 15 .

[0021] The first acquisition unit 11 acquires moving images. For example, the first acquisition unit 11 may acquire one or more moving images captured by one or more cameras. The camera is not limited to a camera installed in a fixed position such as a surveillance camera, but may also be a portable terminal device (e.g., a smartphone) with a camera function.

[0022] The target identification unit 12 identifies a target from a video. The "target" may include at least one of a person and an object. For example, the target identification unit 12 may identify a target from a video using image recognition technology. Specifically, the target identification unit 12 may identify a target from a video using pattern recognition processing, which is an example of image recognition technology. The target identification unit 12 may identify a target from a video using a computational model, which is another example of image recognition technology, that outputs a recognition result of the target when a video is input. An example of the computational model is a computational model using a neural network (e.g., a convolutional neural network (CNN)). When the target is a person, the target identification unit 12 may identify the person as the target through biometric authentication using the video (e.g., at least one of face recognition and iris recognition).

[0023] The moving image may be a moving image including a reader that reads an identification tag (e.g., an employee ID card, an entry pass, etc.) held by a person as a target, and the surrounding area of ​​the reader. The target identification unit 12 may identify the person as a target by linking the detected person with the read identification information based on the time when the person (i.e., the target) is detected from the moving image and the time when the reader reads the identification information indicated by the identification tag.

[0024] The second acquisition unit 13 acquires a target state, which is the state of the target. The second acquisition unit 13 may link the acquired target state to the corresponding target (i.e., the target identified by the target identification unit 12). For example, the second acquisition unit 13 may acquire at least one of the target's motion and behavior from a video image as the target state using video analysis technology. For example, the second acquisition unit 13 may acquire an output value of a sensor attached to the target as the target state. For example, if the target is a person and the person is wearing a biosensor, the second acquisition unit 13 may acquire the output value (e.g., body temperature, pulse rate, etc.) of the biosensor as the target state. The second acquisition unit 13 may acquire the person's instantaneous state (e.g., "body temperature is 38 degrees") as the target state, or may acquire a change in the person's state over a predetermined period (e.g., "body temperature rises by 2 degrees in one hour") as the target state. Examples of biosensors include a blood pressure sensor, a pulse wave sensor, a temperature sensor, a motion sensor, and an electrooculography sensor. For example, the subject's state can be acquired from a biosensor of a wearable device worn by the subject.

[0025] The linguistic unit 14 generates a sentence indicating an object state related to an object from an object and an object state associated with the object. For example, the linguistic unit 14 may generate a sentence that conforms to the grammar of a natural language as a sentence indicating the object state. A "sentence that conforms to the grammar of a natural language" may be a sentence in which, for example, at least one of a state and a situation for a subject indicating an object is described by arranging multiple units corresponding to a subject and a predicate according to a predetermined grammar. A sentence that conforms to the grammar of a natural language may include not only a subject and a predicate, but also at least one of an object, a complement, a modifier, a particle, and a conjunction, for example.

[0026] The expression format of the sentence based on natural language grammar does not have to be expressed in a natural language actually used by humans, such as Japanese or English. For example, if the sentence generated by the linguistic unit 14 is not presented to a user (i.e., a human) of the information processing device 10, the linguistic unit 14 may express the sentence by replacing at least some of the words contained in the natural language with symbols. As long as there is a correspondence between at least one of a real-world object, state, or situation and at least one of a word and a symbol representing the at least one, the linguistic unit 14 does not need to predefine at least one of the replacement words and the ontology (i.e., new words may be added arbitrarily). Note that the correspondence is not strict, and as with natural languages, there may be at least one of synonyms and spelling variations, for example.

[0027] For example, if the target identified by the target identification unit 12 is "Suzuki-san" and the target state acquired by the second acquisition unit 13 is "collapsed," the verbalization unit 14 may generate the sentence "Suzuki-san collapsed." For example, if the targets identified by the target identification unit 12 are "Suzuki-san" and an "elderly person" and the target states acquired by the second acquisition unit 13 are "collapsed," "bathroom," and "6:30 PM," the verbalization unit 14 may generate the sentence "At 6:30 PM, elderly person Suzuki-san collapsed in the bathroom."

[0028] The learning unit 15 trains a large-scale language model on the sentences generated by the language generation unit 14. "Learning" may mean, for example, a process of interpreting the content of a sentence based on the meaning of each word contained in the sentence, the relationship between words, and the like.

[0029] The large-scale language model may be a language model generated by inputting a large amount of sentences (e.g., a huge amount of text on the web, past records similar to the subject of analysis, etc.) into a model having a model structure capable of handling a large number of variables and parameters, such as a Transformer, and then machine learning the model using unsupervised learning such as self-attention.

[0030] The operation of the information processing device 10 will be further described with reference to the flowchart in FIG. 2. In FIG. 2, the first acquisition unit 11 of the information processing device 10 acquires a video image (step S101). Next, the object identification unit 12 of the information processing device 10 identifies an object from the video image (step S102). Next, the second acquisition unit 13 of the information processing unit 10 acquires an object state, which is the state of the object (step S103). Next, the verbalization unit 14 of the information processing unit 10 generates a sentence indicating the object state related to the object from the object and the object state (step S104). Next, the learning unit 15 trains the sentence generated in the processing of step S104 into a large-scale language model (step S105).

[0031] In this way, the information processing device 10 performs an information processing method in which a moving image is acquired, an object is identified from the moving image, an object state that is the state of the object is acquired, a sentence indicating the object state related to the object is generated from the object and the object state, and the generated sentence is trained in a large-scale language model.

[0032] The information processing device 10 described above may be realized by a computer reading a computer program recorded on a recording medium. The recording medium can be said to have recorded thereon a computer program for causing a computer to execute an information processing method that acquires video, identifies an object from the video, acquires a target state that is a state of the object, generates a sentence indicating the target state related to the object from the object and the target state, and trains a large-scale language model on the generated sentence.

[0033] (Technical Effect) For example, devices that only have an object recognition function, such as an IC (Integrated Circuit) card reader or a face authentication device, can observe the subject corresponding to the subject of a sentence, but the word corresponding to the predicate of the sentence is limited to a specific word that depends on the installation location of the device, such as "entry." For this reason, devices that only have an object recognition function cannot respond to events expressed by, for example, a combination of a subject and a verb.

[0034] For example, devices that only have a status acquisition function, such as a motion sensor or an intrusion detector, can observe the behavior corresponding to the predicate of a sentence, but cannot identify the subject corresponding to the subject of the sentence. For this reason, a device that only has a status acquisition function has an unspecified number of subjects, and therefore cannot respond to events expressed by a combination of a subject and a verb, for example.

[0035] Even if a device with only object recognition functionality is combined with a device with only status acquisition functionality, data description languages ​​that do not require any particular order, such as XML, JSON, or YAML, do not have grammar, particles, or conjunctions, and therefore cannot describe the relationships between words.

[0036] In response to these, the verbalization unit 14 of the information processing device 10 performs verbalization using both the target identified by the target identification unit 12 (corresponding to the target recognition function) and the target state acquired by the second acquisition unit 13 (corresponding to the state acquisition function). As a result, the verbalization unit 14 can generate sentences that conform to the grammar of natural language (for example, SV-type sentences that include at least a subject and a verb).

[0037] For example, at least one of the particles, conjunctions, and syntax may be set in advance in the verbalization unit 14. For example, the verbalization unit 14 may generate a sentence by inserting a word representing the target identified by the target identification unit 12 and a word representing the target state acquired by the second acquisition unit 13 into a syntax set in advance.

[0038] For example, attribute information indicating the attributes of at least one of a person and an object as a target may be registered in advance in the target identification unit 12. The target identification unit 12 may assign attribute information indicating the attributes of the identified target. For example, the verbalization unit 14 may include at least one of a complement and a modifier that supplementarily explain the target in a sentence based on the attribute information assigned to the target.

[0039] For example, location information indicating at least one of the installation location of the information processing device 10 and the location where the moving image is captured may be registered in advance in the second acquisition unit 13. The second acquisition unit 13 may assign location information to the acquired target state. For example, the verbalization unit 14 may include at least one of a complement, a modifier, and an adverb that supplements and explains the verb expressing the target state, in the sentence based on the location information assigned to the target state.

[0040] Based on the time information, the verbalization unit 14 may include at least one of a phrase indicating the date and time and a phrase indicating a time attribute such as during working hours / outside working hours in the sentence.

[0041] The large-scale language model can handle an extremely large number of variables and parameters. Therefore, the learning unit 15 trains the large-scale language model with the sentences generated by the linguistic unit 14, and the large-scale language model can analyze the meaning and context of the sentences. As a result, the large-scale language model can analyze, with a high degree of accuracy, the real world, where causes and effects exist infinitely and are intricately intertwined.

[0042] For example, when a large-scale language model is used for at least one of result prediction and cause estimation, the large-scale language model can understand the meaning of the sentence generated by the languageization unit 14 and the meaning of each sentence including the context of multiple sentences arranged in chronological order, and can make predictions such as the possibility of a certain result occurring in the current real world and estimation of the cause that caused a certain result. For example, when a user of the information processing device 10 inputs a question to the large-scale language model, the large-scale language model can output an appropriate answer to the input question.

[0043] For example, suppose the learning unit 15 has trained the large-scale language model with sentences generated by the language unit 14, such as "Mr. Suzuki is an 85-year-old man" and "Mr. Suzuki has severe heart disease." If a user then inputs an instruction such as "What are the risks of Mr. Suzuki taking a bath early in the morning?" into the large-scale language model, the large-scale language model can output a response indicating the risks associated with a specific person, "Mr. Suzuki," based on the state of "Mr. Suzuki" indicated by the sentences "Mr. Suzuki is an 85-year-old man," "Mr. Suzuki has severe heart disease," and "Mr. Suzuki takes a bath early in the morning," as well as general knowledge and the like. For example, the large-scale language model can output a response such as "Since Mr. Suzuki is an elderly man in his 85s, the following risks are considered: 1. Falls, 2. Myocardial infarction, 3. cerebral infarction, 4. Loss of consciousness."

[0044] Second Embodiment A second embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to FIG. 3 . Hereinafter, the second embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 10a. In the second embodiment, part of the configuration of the information processing device 10a differs from the information processing device 10 according to the first embodiment described above. Other parts may be the same as those in the first embodiment described above. Therefore, for the second embodiment, descriptions that overlap with those in the first embodiment will be omitted as appropriate.

[0045] 3 , the information processing device 10a includes a first acquisition unit 11, a target identification unit 12, a second acquisition unit 13, a languageization unit 14, and a learning unit 15. The second acquisition unit 13 includes a state acquisition unit 131 and a surrounding situation acquisition unit 132.

[0046] The state acquisition unit 131 acquires a target state, which is a state of a target, from a video image, similar to the second acquisition unit 13 according to the first embodiment described above. The state acquisition unit 131 may link the acquired target state to the corresponding target (i.e., the target identified by the target identification unit 12). For example, the state acquisition unit 131 may acquire at least one of the motion and behavior of the target from the video image as the target state using video analysis technology. For example, the state acquisition unit 131 may acquire an output value of a sensor attached to the target as the target state.

[0047] The surrounding situation acquisition unit 132 acquires the surrounding situation of the target from the video. The "surrounding situation" may mean, for example, the state of at least one of people and objects present around the target when the target is identified by the target identification unit 12, and the state of at least one of people and objects present around the target. The surrounding situation acquisition unit 132 may link the acquired surrounding situation to the corresponding target (i.e., the target identified by the target identification unit 12).

[0048] The verbalization unit 14 may generate a sentence indicating a target state related to one target from one target, a target state linked to the one target, and a peripheral situation linked to the one target. Note that the verbalization unit 14 may generate a sentence indicating a target state related to one target from one target and the target state or peripheral situation linked to the one target. In other words, the verbalization unit 14 may generate a sentence indicating a target state related to one target from one target and at least one of the target state and peripheral situation linked to the one target.

[0049] (Technical Effect) For example, the surrounding circumstances acquired by the surrounding circumstances acquisition unit 132 may be at least one of an action and a behavior in which one target is at least one of another person and an object, such as "having a conversation with another person" or "lifting an object." The verbalization unit 14 may generate a word corresponding to an object from the acquired surrounding circumstances. As a result, the verbalization unit 14 can generate an SVO-type sentence including at least a subject, a verb, and an object.

[0050] For example, the surrounding conditions acquired by the surrounding condition acquisition unit 132 may be the environment around a target that may affect the state of the target, such as "air temperature," "light intensity," or "whether the lights are on or off." The verbalization unit 14 may generate words corresponding to at least one of a complement and a modifier (in other words, elements for supplementing the meaning of a sentence) from the acquired surrounding conditions. As a result, the verbalization unit 14 can generate an SVC-type or SV-type sentence that includes at least a subject, a verb, and a complement or a modifier.

[0051] For example, if the object identified by the object identification unit 12 is "Mr. Sato," the object state acquired by the state acquisition unit 131 is "bought" and "2 p.m.," and the surrounding conditions acquired by the surrounding condition acquisition unit 132 are "red" and "notebook personal computer," the verbalization unit 14 may generate the sentence "Mr. Sato bought a red notebook personal computer at 2 p.m."

[0052] Third Embodiment A third embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to FIG. 4 . Hereinafter, the third embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 20. In the third embodiment, part of the configuration of the information processing device 20 differs from the information processing device 10a according to the second embodiment described above. Other parts may be the same as those of the first and second embodiments described above. Therefore, for the third embodiment, descriptions that overlap with those of the first and second embodiments will be omitted as appropriate.

[0053] 4, the information processing device 20 includes a calculation device 21, a storage device 22, and a communication device 23. The information processing device 20 may further include an input device 24 and an output device 25. In the information processing device 20, the calculation device 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26. Note that the information processing device 20 does not necessarily include at least one of the input device 24 and the output device 25.

[0054] The arithmetic device 21 may include, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a TPU (Tensor Processing Unit), and a quantum processor.

[0055] The storage device 22 may include, for example, at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and an optical disk array. In other words, the storage device 22 may include a non-transitory recording medium. The storage device 22 is capable of storing desired data. For example, the storage device 22 may temporarily store a computer program executed by the arithmetic device 21. The storage device 22 may temporarily store data that is temporarily used by the arithmetic device 21 when the arithmetic device 21 is executing a computer program.

[0056] The communication device 23 may be capable of communicating with devices external to the information processing device 20. The communication device 23 may perform wired communication or wireless communication.

[0057] The input device 24 is a device capable of accepting information input to the information processing device 20 from outside. The input device 24 may include an operation device (e.g., a keyboard, a mouse, a touch panel, etc.) that can be operated by an operator of the information processing device 20. The input device 24 may include a recording medium reading device that can read information recorded on a recording medium that is detachable from the information processing device 20, such as a USB (Universal Serial Bus) memory. Note that when information is input to the information processing device 20 via the communication device 23 (in other words, when the information processing device 20 acquires information via the communication device 23), the communication device 23 may function as an input device.

[0058] The output device 25 is a device capable of outputting information to the outside of the information processing device 20. The output device 25 may output visual information such as text or images, auditory information such as sound, or tactile information such as vibration, as the information. The output device 25 may include, for example, at least one of a display, a speaker, a printer, and a vibration motor. The output device 25 may be capable of outputting information to a recording medium detachable from the information processing device 20, such as a USB memory. Note that when the information processing device 20 outputs information via the communication device 23, the communication device 23 may function as the output device.

[0059] The computing device 21 may have, as logically realized functional blocks or as physically realized processing circuits, a first acquisition unit 211, a target identification unit 212, a second acquisition unit 213, a language generation unit 214, an external input unit 215, a large-scale language model unit 216, and an answer display unit 217. The second acquisition unit 213 may have a state acquisition unit 2131 and a surrounding situation acquisition unit 2132.

[0060] At least one of the first acquisition unit 211, the target identification unit 212, the second acquisition unit 213, the linguistic unit 214, the external input unit 215, the large-scale language model unit 216, and the answer display unit 217 may be realized in a format in which a logical functional block and a physical processing circuit (i.e., hardware) are mixed. When at least some of the first acquisition unit 211, the target identification unit 212, the second acquisition unit 213, the linguistic unit 214, the external input unit 215, the large-scale language model unit 216, and the answer display unit 217 are functional blocks, at least some of the first acquisition unit 211, the target identification unit 212, the second acquisition unit 213, the linguistic unit 214, the external input unit 215, the large-scale language model unit 216, and the answer display unit 217 may be realized by the calculation device 21 executing a predetermined computer program.

[0061] The arithmetic device 21 may acquire (in other words, read) the predetermined computer program from the storage device 22. The arithmetic device 21 may read the predetermined computer program stored in a computer-readable, non-transitory recording medium using a recording medium reading device (not shown) included in the information processing device 20. The arithmetic device 21 may acquire (in other words, download or read) the predetermined computer program from a device (not shown) external to the information processing device 20 via the communication device 23. Note that the recording medium for recording the predetermined computer program executed by the arithmetic device 21 may be at least one of an optical disk, a magnetic medium, a magneto-optical disk, a semiconductor memory, and any other medium capable of storing a program.

[0062] The "first acquisition unit 211," "target identification unit 212," "second acquisition unit 213," "linguisticization unit 214," and "large-scale language model unit 216" are components corresponding to the "first acquisition unit 11," "target identification unit 12," "second acquisition unit 13," "linguisticization unit 14," and "learning unit 15" in the first embodiment described above, respectively. The "state acquisition unit 2131" and the "surrounding situation acquisition unit 2132" are components corresponding to the "state acquisition unit 131" and the "surrounding situation acquisition unit 132" in the second embodiment described above, respectively.

[0063] (Operation of Information Processing Device) The external input unit 215 acquires, for example, at least one of information input by a user (or an operator) via the input device 24 and information from a device external to the information processing device 20. The information acquired by the external input unit 215 will hereinafter be referred to as "external input information" as appropriate. The external input information may include, for example, text data written in words and grammar common to the natural language used in the sentences generated by the linguistic unit 214. In other words, the external input information may include sentences.

[0064] The words expressing a single event may not be the same in the sentence generated by the verbalization unit 214 and the sentence included in the external input information. In other words, as in natural language, at least one of spelling variations and synonyms may exist. Furthermore, the sentence included in the external input information may have irregular grammar.

[0065] The large-scale language model unit 216 uses the large-scale language model to learn the sentences generated by the verbalization unit 214 (i.e., sentences indicating the target state related to the target) and the external input information acquired by the external input unit 215. In other words, the large-scale language model unit 216 causes the large-scale language model to learn the sentences generated by the verbalization unit 214 and the external input information acquired by the external input unit 215.

[0066] When a user inputs a directive via the input device 24, the large-scale language model unit 216 outputs a response sentence to the input directive using a large-scale language model. The large-scale language model unit 216 may output a response sentence to the input directive based on a result of learning at least one of the sentence generated by the languageization unit 214 and the external input information acquired by the external input unit 215.

[0067] The answer display unit 217 controls the output device 25 to display the answer sentence output from the large-scale language model unit 216. As a result, for example, the answer sentence may be displayed on a display as an example of the output device 25.

[0068] (Application Example of Information Processing Device) For example, the operation of the information processing device 20 when the information processing device 20 is applied to process management at a construction site will be specifically described.

[0069] It is assumed that the external input unit 215 acquires, for example, construction technical information and construction plan documents as external input information. The construction technical information may be acquired from at least one of construction-related academic journals and industry papers. The construction technical information may also be acquired via the Internet. The construction plan documents may be input by the site supervisor via the input device 24, or may be acquired from a terminal device carried by the site supervisor via the communication device 23. The construction plan documents may be updated as appropriate. The external input unit 215 may acquire the updated construction plan documents as external input information. The external input information may include at least one of a worker's work report and an inspection report showing the inspection results by a qualified person. The external input information may also include a proper name unique to the construction site.

[0070] It is assumed that a plurality of cameras are installed to capture images of a plurality of locations on the construction site. At least one of the cameras may be installed to capture images near an entrance / exit of the construction site. The first acquisition unit 211 acquires video images captured by each of the plurality of cameras.

[0071] When identifying a person as a target, the target identification unit 212 may perform face recognition using the moving image acquired by the first acquisition unit 211. Note that, since the behavior of workers is an important subject of analysis at a construction site, the target identification unit 212 may identify only people as targets. In this case, the target identification unit 212 may be referred to as a "person identification unit."

[0072] The target identification unit 212 may identify the name of the target person (in other words, the person to be authenticated) through facial recognition. The target identification unit 212 may identify at least one of the name of the company to which the target person belongs, the occupation type, and the qualifications of the target person, in addition to the name, through facial recognition. The target identification unit 212 may identify at least one of the name of the company to which the target person belongs, the occupation type, and the qualifications of the target person by performing image recognition processing on a video including the appearance of the target person. With this configuration, it is possible to specifically identify the target person and obtain information about the person.

[0073] The target identification unit 212 may perform a tracking process to track a person as a target based on the moving image acquired by the first acquisition unit 211. The tracking process will now be described with reference to FIG.

[0074] 5 are images including a person P1 as a target (in other words, images in which the person P1 is captured). The image IMG1 may be an image captured by a camera at time t1. The image IMG2 may be an image captured by a camera at time t2, which is later than time t1.

[0075] Each of images IMG1 and IMG2 may correspond to one frame of a video. Image IMG2 does not have to correspond to the frame immediately following the frame corresponding to image IMG1. In other words, image IMG2 may correspond to a frame two or more frames after the frame corresponding to image IMG1.

[0076] The target identification unit 212 may detect the head of person P1 included in image IMG1 from image IMG1. Note that existing technology is applied to the method of detecting a person's head from an image, so detailed description thereof will be omitted. The target identification unit 212 may set an area including the head of person P1 as target tracking area TA1 based on the detected head of person P1. Note that the target identification unit 212 may detect the upper body or the entire body of person P1 included in image IMG1. The target identification unit 212 may set an area including the upper body or the entire body of person P1 as target tracking area TA1 based on the detected upper body or the entire body of person P1.

[0077] When the target identification unit 212 sets the target tracking area TA1, it sets a tracking ID, which is identification information for identifying the person P1 related to the target tracking area TA1, to the person P1. The target identification unit 212 calculates the position of the person P1 (for example, the position of the feet of the person P1) based on the target tracking area TA1. Note that, since existing technology can be applied to a method for calculating the position of a subject included in an image from the image, a detailed description thereof will be omitted. The target identification unit 212 may associate the tracking ID and the position of the person P1 with each other and store them in the storage device 22.

[0078] The target identification unit 212 may use the image IMG2 and the image IMG1 to identify the person P1 included in the image IMG2. Identifying the person P1 included in the image IMG2 is synonymous with associating the person P1 included in the image IMG1 with the person P1 included in the image IMG2. Therefore, at least one of a matching method and an optical flow method related to image association can be applied to identifying the person P1 included in the image IMG2. Note that various existing methods can be applied to both the matching method and the optical flow method, and therefore detailed description thereof will be omitted.

[0079] When person P1 included in image IMG2 is identified, the target identification unit 212 may detect the head of person P1. Based on the detected head of person P1, the target identification unit 212 may set an area including the head of person P1 as target tracking area TA2. Since person P1 included in image IMG1 and person P1 included in image IMG2 are the same person, the tracking ID of person P1 related to the target tracking area TA2 is the same as the tracking ID of person P1 related to the target tracking area TA1. The target identification unit 212 calculates the position of person P1 based on the target tracking area TA2. Note that the target identification unit 212 may also detect the upper body or the entire body of person P1 included in image IMG2. Based on the upper body or the entire body of the detected person P2, the target identification unit 212 may set an area including the upper body or the entire body of person P1 as target tracking area TA2.

[0080] The target identification unit 212 may associate the position of the person P1 with the tracking ID related to the person P1 and store it in the storage device 22. Note that, when the position of the person P1 calculated based on the target tracking area TA1 is stored in the storage device 22, the target identification unit 212 stores the position of the person P1 calculated based on the target tracking area TA2 in the storage device 22, thereby updating the position of the person P1.

[0081] However, it may be difficult for the target identification unit 212 to continue tracking the person P1, for example, because the head of the person P1 is hidden behind another person. When tracking of the person P1 is interrupted, the target identification unit 212 may perform the following process. After tracking of the person P1 is interrupted, the target identification unit 212 may determine whether a new person has been detected from the moving image acquired by the first acquisition unit 211. A "new person" means a person for whom a tracking ID has not been set.

[0082] When a new person is detected, the target identification unit 212 may determine whether the person P1 and the new person are the same person by comparing the feature amount of the tracking area (e.g., at least one of the tracking areas TA1 and TA2) related to the person P1 with the feature amount of the tracking area related to the new person. If it is determined that the person P1 and the new person are the same person, the target identification unit 212 may set the tracking ID related to the person P1 to the new person. As a result, the target identification unit 212 can track the person P1 again.

[0083] The feature amount may be a feature amount related to the person's head, a feature amount related to the person's upper body, or a feature amount related to the person's entire body. In other words, the tracking area (for example, at least one of the tracking areas TA1 and TA2) may include the person's head, the person's upper body, or the person's entire body. The feature amount may be obtained, for example, by person re-identification technology.

[0084] The state acquisition unit 2131 acquires a target state, which is a state of the target, from the moving image. For example, the state acquisition unit 2131 may acquire a state, such as when a person as the target picks up an object, swings the object around, or lets go of the object, as the target state. The state acquisition unit 2131 may assign at least one of the time and place at which the target state is detected and the proper name of the construction site to the acquired target state.

[0085] The surrounding situation acquisition unit 2132 acquires the surrounding situation from the video. For example, the surrounding situation acquisition unit 2132 may acquire at least one of materials, tools, and heavy machinery that are the target of an action by a person as a target as the surrounding situation. The surrounding situation acquisition unit 2132 may acquire the state (e.g., whether interior construction has been completed, etc.) of objects (e.g., framework, concrete walls, etc.) around the person as a target as the surrounding situation.

[0086] For example, the target identification unit 212 may transmit target information indicating a single person as a target to the status acquisition unit 2131. The status acquisition unit 2131 may assign status information indicating the status of the single person to the target information. In other words, the status acquisition unit 2131 may link the single person as a target with the target status indicating the status of the single person by assigning the target status to the target information. The status acquisition unit 2131 may transmit the target information to which the target status has been assigned to the surrounding situation acquisition unit 2132. The surrounding situation acquisition unit 2132 may assign the surrounding situation to the target information to which the target status has been assigned. In other words, the surrounding situation acquisition unit 2132 may link the single person as a target with the target status indicating the status of the single person and the surrounding situation by assigning the surrounding situation to the target information to which the target status has been assigned.

[0087] The verbalization unit 214 may generate a sentence indicating a target state related to one person from one person as a target, the target state, and a surrounding situation.

[0088] For example, if the target identified by the target identification unit 212 is "Worker P," the target state acquired by the state acquisition unit 2131 is "Date and time: 9:00, May 4, 2023," "Location: near the center of construction site A," and "Action: operating heavy machinery," and the surrounding conditions acquired by the surrounding condition acquisition unit 2132 are "surrounding area: earth and rocks" and "heavy machinery: large backhoe," the languageization unit 214 may generate the sentence "At 9:00, May 4, 2023, near the center of construction site A, worker P is operating a large backhoe in the midst of earth and rocks in the surrounding area."

[0089] For example, if the target identified by the target identification unit 212 is "Worker Q," the target state acquired by the state acquisition unit 2131 is "Date and time: 9:00, May 4, 2023," "Location: near the center west of construction site A," and "Action: driving a car," and the surrounding conditions acquired by the surrounding condition acquisition unit 2132 are "near a large backhoe," "the vehicle is a dump truck," and "earth and stones in the loading compartment," the languageization unit 214 may generate the sentence "At 9:00, May 4, 2023, near the large backhoe, near the center west of construction site A, worker Q is loading earth and stones into a dump truck."

[0090] For example, if the target identified by the target identification unit 212 is "Worker Q," the target state acquired by the state acquisition unit 2131 is "Date and time: 10:00, May 4, 2023," "Location: front gate of construction site A," and "Action: Exit by car," and the surrounding conditions acquired by the surrounding condition acquisition unit 2132 are "The vehicle is a dump truck" and "There is earth and stones in the loading area," the languageization unit 214 may generate the sentence "At 10:00, May 4, 2023, Worker Q loaded earth and stones into a dump truck and exited through the front gate of construction site A."

[0091] The large-scale language model unit 216 uses the large-scale language model to learn the sentences generated by the language unit 214 and the external input information acquired by the external input unit 215 .

[0092] Known methods for training large-scale language models include pretraining, fine-tuning, and in-context learning, and at least one of these methods may be used.

[0093] The large-scale language model unit 216 may sequentially learn the sentences generated by the languageization unit 214 and the external input information acquired by the external input unit 215, or may store them in a storage device 22 or the like and learn them all at once.

[0094] For example, when training a large-scale language model using in-context learning, the data may be stored in the storage device 22 or the like, and when processing an instruction sentence, the data may be added as context information for the site, and the instruction sentence may be processed while training is being performed.

[0095] For example, the large-scale language model unit 216 may learn sentences such as "At 9:00 on May 4, 2023, near the center of construction site A, worker P is operating a large backhoe in the midst of the surrounding area, surrounded by earth and rocks," "At 9:00 on May 4, 2023, worker Q is loading earth and rocks into a dump truck near the large backhoe, towards the west of the center of construction site A," and "At 10:00 on May 4, 2023, worker Q loaded earth and rocks into a dump truck and left through the front gate of construction site A," as well as a process plan that reads "Process 2: Foundation work; Period: From May 1 to May 31, 2023; Work: Leveling, piling, reinforcing bars, formwork, pouring concrete, piping" as external input information.

[0096] As part of its learning, the large-scale language model unit 216 may estimate that each sentence generated by the verbalization unit 214 represents "civil engineering work." As part of its learning, the large-scale language model unit 216 may estimate, based on a construction plan as external input information, that each sentence generated by the verbalization unit 214 represents "ground leveling work in the foundation work process" of "civil engineering work." As part of its learning, the large-scale language model 216 may compare information such as "Period: May 1 to May 31, 2023" included in the construction plan as external input information with the date and time included in each sentence generated by the verbalization unit 214, and estimate that the progress of the construction work is progressing smoothly.

[0097] For example, when a site supervisor inputs a question such as "Is the construction at construction site A going smoothly?" via the input device 24, the large-scale language model unit 216 may output the answer sentence "It's going smoothly" using the large-scale language model. The answer display unit 217 may control the output device 25 to display the character string "It's going smoothly" as the answer sentence output from the large-scale language model unit 216.

[0098] The site supervisor may input a prompt for the question, such as "Is the construction at construction site A going smoothly? Please answer using the following examples as a reference: (a) It's going smoothly. (b) A certain task in a certain process is behind schedule. (c) The task in a certain process differs from the process plan." via the input device 24. In other words, the user of the information processing device 20 may input information that restricts the format of the answer to the question in addition to the prompt.

[0099] (Technical Effect) Conventional mathematical models can only handle a relatively small number of predefined explanatory variables. Therefore, conventional mathematical models cannot learn information described in natural language (e.g., sentences generated by the linguistic unit 214). In contrast, large-scale language models can handle a very large number of variables and parameters, and therefore can learn information described in natural language. Therefore, large-scale language models can learn, for example, external input information.

[0100] Natural languages ​​have a higher degree of freedom than, for example, programming languages. Therefore, natural languages ​​can be used to appropriately describe real-world phenomena that change over time or that generate new concepts. Therefore, the large-scale language model unit 216, which learns sentences that conform to the grammar of natural languages ​​(e.g., sentences generated by the languageization unit 214), can appropriately learn real-world phenomena.

[0101] As described above, large-scale language models are capable of unsupervised learning, eliminating the need to assign teacher labels to information written in natural language. In other words, large-scale language models can learn information written in natural language as is. Therefore, the information processing device 20 can improve user convenience.

[0102] In the real world, there are countless possible causes of an event included in a video (in other words, an observed event) and countless possible consequences of that event. If the large-scale language model unit 216 were to learn only sentences generated by the linguistic unit 214, the large-scale language model unit 216 might not be able to appropriately respond to a user instruction input via the input device 24. Therefore, in this embodiment, the large-scale language model unit 216 also learns external input information acquired by the external input unit 215.

[0103] When the large-scale language model unit 216 learns the sentences generated by the linguistic unit 214 and the external input information acquired by the external input unit 215, the large-scale language model unit 216 can narrow down at least one of the possible causes of the event represented by the sentences generated by the linguistic unit 214 and the possible consequences of the event, compared to when the large-scale language model unit 216 learns only the sentences generated by the linguistic unit 214. Therefore, the information processing device 20 can appropriately respond to a user instruction input via the input device 24.

[0104] A specific example of the technical effect brought about by the information processing device 20 will be described. It is assumed that, among "Process 2: Foundation work; Period: May 1 to May 31, 2023; Work: Ground leveling, pile driving, reinforcement, formwork, pouring, piping" included in the process plan for the construction site described above, the specific pile driving method has been changed from the previous plan to "down-the-hole hammer construction." It is assumed that the surrounding condition acquisition unit 2132 cannot identify the heavy equipment used in the down-the-hole hammer construction.

[0105] Based on the above assumptions, the external input unit 215 may acquire, as external input information, a portion of the process plan change document, which reads, "Process 2: Foundation work; Work name: Pile driving; Construction method name: Down-the-hole hammer method; Heavy equipment used: 0.25 backhoe, down-the-hole hammer, oiler, swivel, 100 horsepower compressor, hose."

[0106] The large-scale language model unit 216 may use the large-scale language model to learn a portion of a process plan change document such as "Process 2: Foundation work; Work name: Pile driving; Construction method: Down-the-hole hammer method; Heavy equipment used: 0.25 backhoe, down-the-hole hammer, oiler, swivel, 100 horsepower compressor, hose."

[0107] For example, if the target identified by the target identification unit 212 is "Worker R," the target state acquired by the state acquisition unit 2131 is "Date and time: 10:00 on May 10, 2023," "Location: front gate of construction site A," and "Action: Enter by car," and the surrounding conditions acquired by the surrounding conditions acquisition unit 2132 are "The vehicle is a trailer" and "There is unknown heavy equipment in the loading platform," the languageization unit 214 may generate the sentence "At 10:00 on May 10, 2023, Worker R entered through the front gate of construction site A with unknown heavy equipment loaded on a trailer."

[0108] For example, if the target identified by the target identification unit 212 is "Worker P," the target state acquired by the state acquisition unit 2131 is "Date and time: 11:00, May 10, 2023," "Location: near the center of construction site A," and "Action: operating heavy machinery," and the surrounding conditions acquired by the surrounding condition acquisition unit 2132 are "surrounding area: earth and rocks," "heavy machinery is unknown," and "there are piles near the heavy machinery," the languageization unit 214 may generate the sentence "At 11:00, May 10, 2023, near the center of construction site A, worker P is operating unknown heavy machinery in the midst of earth and rocks in the surrounding area. There are piles near the heavy machinery."

[0109] The large-scale language model unit 216 may use a large-scale language model to learn sentences such as "At 10:00 on May 10, 2023, worker R entered the front gate of construction site A with unknown heavy equipment loaded on a trailer" and "At 11:00 on May 10, 2023, near the center of construction site A, worker P is operating unknown heavy equipment in the midst of earth and rocks in the surrounding area. There are piles near the heavy equipment."

[0110] The large-scale language model unit 216, which has learned a part of the construction plan change document as external input information, may estimate, as part of its learning, that "unknown heavy machinery" is, for example, "heavy machinery related to the down-the-hole hammer construction method." Furthermore, the large-scale language model unit 216 may estimate, as part of its learning, that each sentence generated by the verbalization unit 214 is "pile driving work using the down-the-hole hammer construction method."

[0111] In this way, by learning external input information, the large-scale language model unit 216 can appropriately learn the sentence generated by the languageization unit 214 even if the sentence contains an unclear description (e.g., "unknown heavy machinery").

[0112] Furthermore, if the large-scale language model unit 216 cannot infer the "down-the-hole hammer construction method" from the above sentence describing the event on May 10th, generated by the languageization unit 214, the large-scale language model unit 216 may perform the following operation.

[0113] For example, if the target identified by the target identification unit 212 is "Worker S," the target state acquired by the state acquisition unit 2131 is "Date and time: 10:00, May 15, 2023," "Location: Near the center of construction site A," and "Action: Reinforcement," and the surrounding conditions acquired by the surrounding condition acquisition unit 2132 are "Surrounded by foundations," "Surrounding rebar," and "Hacker in hand," the languageization unit 214 may generate the sentence "10:00, May 15, 2023, near the center of construction site A, surrounded by foundations, worker S holding a hacker in his hand and installing rebar."

[0114] For example, if the target identified by the target identification unit 212 is "Worker T," the target state acquired by the state acquisition unit 2131 is "Date and time: 11:00, May 15, 2023," "Location: east-central part of construction site A," and "Action: Cutting," and the surrounding conditions acquired by the surrounding condition acquisition unit 2132 are "surrounded by rebar" and "Tool: rebar cutter," the languageization unit 214 may generate the sentence "At 11:00, May 15, 2023, in the east-central part of construction site A, worker T is installing rebar using a rebar cutter."

[0115] The large-scale language model unit 216 may use the large-scale language model to learn sentences such as "At 10:00 on May 15, 2023, near the center of construction site A, surrounded by foundations, worker S is installing rebar with a hacker in his hand" and "At 11:00 on May 15, 2023, near the east side of the center of construction site A, worker T is installing rebar with a rebar cutter."

[0116] As part of its learning, the large-scale language model unit 216 may estimate that each sentence generated by the verbalization unit 214 is "reinforcement work in the foundation work process." The large-scale language model unit 216 may refer to the process plan and the process plan change document to estimate that there is a high possibility that the event on May 10th (i.e., the sentence related to May 10th generated by the verbalization unit 214) was "pile driving work" using the "down-the-hole hammer method."

[0117] In the information processing device 20, even if a change occurs in part of the process plan, there is no need to modify or re-learn at least one of the linguistic unit 214 and the large-scale language model unit 216. In addition, the large-scale language model unit 216 can recognize that a change has occurred in part of the process plan, for example, by learning a process plan change document. Therefore, even if the events described in the sentences generated by the linguistic unit 214 for the changed part of the process plan differ from the contents of the original process plan, if the events described in the sentences generated by the linguistic unit 214 are consistent with, for example, the contents of the process plan change document, the large-scale language model unit 216 can determine that there is no abnormality.

[0118] Fourth Embodiment A fourth embodiment relating to an information processing device, an information processing method, and a recording medium will be described with reference to FIG. 6 . Hereinafter, the fourth embodiment relating to an information processing device, an information processing method, and a recording medium will be described using an information processing device 20a. In the fourth embodiment, part of the configuration of the information processing device 20a differs from the information processing device 20 according to the third embodiment described above. Other parts may be the same as those of the first to third embodiments described above. Therefore, for the fourth embodiment, descriptions that overlap with those of the first to third embodiments will be omitted as appropriate.

[0119] 6, the arithmetic device 21 may have, as logically realized functional blocks or as physically realized processing circuits, a first acquisition unit 211, a target identification unit 212, a second acquisition unit 213, a verbalization unit 214, an external input unit 215, a large-scale language model unit 216, a directive statement construction unit 301, a directive statement improvement unit 302, a result determination unit 303, and an action execution unit 304. Note that the arithmetic device 21 does not necessarily have to have the verbalization unit 214.

[0120] Furthermore, at least one of the first acquisition unit 211, target identification unit 212, second acquisition unit 213, languageization unit 214, external input unit 215, large-scale language model unit 216, instruction sentence construction unit 301, instruction sentence improvement unit 302, result determination unit 303 and action execution unit 304 may be realized in a form that mixes logical functional blocks and physical processing circuits (i.e., hardware). If at least a portion of the first acquisition unit 211, the target identification unit 212, the second acquisition unit 213, the linguistics unit 214, the external input unit 215, the large-scale language model unit 216, the instruction sentence construction unit 301, the instruction sentence improvement unit 302, the result determination unit 303, and the action execution unit 304 are functional blocks, at least a portion of the first acquisition unit 211, the target identification unit 212, the second acquisition unit 213, the linguistics unit 214, the external input unit 215, the large-scale language model unit 216, the instruction sentence construction unit 301, the instruction sentence improvement unit 302, the result determination unit 303, and the action execution unit 304 may be realized by the calculation device 21 executing a predetermined computer program.

[0121] (Operation of Information Processing Device) The instruction sentence construction unit 301 constructs (in other words, generates) an instruction sentence from one object, an object state linked to the one object, and a peripheral situation linked to the one object. For example, the instruction sentence construction unit 301 may determine at least one of the object and the item to be analyzed according to a predetermined condition. The instruction sentence construction unit 301 may construct an instruction sentence according to a predetermined instruction sentence template. As with the above-described verbalization unit 214, the instruction sentence construction unit 301 may construct an instruction sentence (i.e., a sentence) that conforms to the grammar of one natural language as an instruction sentence.

[0122] For example, the instruction sentence construction unit 301 may construct, as an instruction sentence, a question sentence inquiring about a predetermined event. For example, if the predetermined event can take two states, that is, the event has occurred and the event has not occurred, the instruction sentence construction unit 301 may construct, as an instruction sentence, a question sentence inquiring whether the predetermined event has occurred. For example, if the predetermined event can take three or more states, the instruction sentence construction unit 301 may construct, as an instruction sentence, a question sentence such as "Is X A, B, or C?"

[0123] The instruction sentence improvement unit 301 improves the instruction sentence constructed by the instruction sentence construction unit 301 in accordance with the characteristics of the large-scale language model unit 216 so that an appropriate answer sentence can be obtained from the large-scale language model unit 216. When the instruction sentence improvement unit 301 improves the instruction sentence constructed by the instruction sentence construction unit 301, the improvement content may be stored, for example, in the storage device 22 (see "Improvement record 221" in FIG. 6 ). The improvement content may be, for example, an instruction sentence constructed by the instruction sentence construction unit or an instruction sentence input by a user. Alternatively, for example, multiple instruction sentences with high correlations may be extracted from multiple instruction sentences input to the large-scale language model over a predetermined period, and the differences between the instruction sentences may be stored as the improvement content. Alternatively, both the instruction sentence and the differences between the instruction sentences may be stored. For example, suppose that instruction sentence a1 is input to the large-scale language model, and then instruction sentence a2 is input. If the correlation between instruction sentence a1 and instruction sentence a2 is high, that is, if instruction sentences with the same content are input, it can be determined that there is a high possibility that the improvement to instruction sentence a1 has been made to instruction sentence a2. Therefore, the difference between the instruction sentence a1 and the instruction sentence a2 may be extracted and the improvement may be recorded.

[0124] The instruction sentence improvement unit 302 may improve the instruction sentence constructed by the instruction sentence construction unit 301 by referring to the improvement record 221 (i.e., past improvement records). For example, the instruction sentence improvement unit 302 may refer to the improvement record 221 using at least one of a verb and an object included in the instruction sentence constructed by the instruction sentence construction unit 301 as a clue. If the frequency of improvement for at least one of the verb and the object used as a clue is higher than a predetermined value, the instruction sentence improvement unit 302 may improve the instruction sentence. Note that if the frequency of improvement for at least one of the verb and the object used as a clue is lower than the predetermined value, the instruction sentence improvement unit 302 may not improve the instruction sentence. Note that if the frequency of improvement is "equal" to the predetermined value, either case may be included.

[0125] The instruction sentence improvement unit 302 may refer to external input information acquired by the external input unit 215 to improve the instruction sentence constructed by the instruction sentence construction unit 301 .

[0126] The instruction sentence improvement unit 302 may input the improved instruction sentence to the large-scale language model unit 216. Note that if the instruction sentence improvement unit 302 does not improve the instruction sentence constructed by the instruction sentence construction unit 301, the instruction sentence improvement unit 302 may input the instruction sentence constructed by the instruction sentence construction unit 301 to the large-scale language model unit 216.

[0127] As described above, the large-scale language model unit 216 uses the large-scale language model to learn the sentences generated by the verbalization unit 214 (i.e., sentences indicating the target state related to the target) and the external input information acquired by the external input unit 215. When a directive sentence (i.e., a directive sentence constructed by the directive sentence construction unit 301 or a directive sentence improved by the directive sentence improvement unit 302) is input from the directive sentence improvement unit 302, the large-scale language model unit 216 uses the large-scale language model to output a response sentence to the input directive sentence.

[0128] The result determination unit 303 acquires the answer sentence output from the large-scale language model unit 216. The result determination unit 303 determines whether the acquired answer sentence is a result (in other words, an answer sentence) that matches the request to the information processing device 20a. For example, the result determination unit 303 may determine whether the syntax of the answer sentence output from the large-scale language model unit 216 matches a template related to a pre-set answer sentence. In this case, the "template related to a pre-set answer sentence" corresponds to an example of a "request to the information processing device 20a."

[0129] If it is determined that the answer sentence output from the large-scale language model unit 216 does not match the request to the information processing device 20a, the result determination unit 303 sends difference information indicating the difference between the request to the information processing device 20a and the answer sentence (i.e., the result) output from the large-scale language model unit 216 to the instruction sentence improvement unit 302.

[0130] For example, if the syntax of the answer sentence output from the large-scale language model unit 216 does not match a template related to a pre-set answer sentence, the result determination unit 303 may send the above-mentioned difference information to the instruction sentence improvement unit 302, including information indicating that the answer sentence does not match the template related to the answer sentence and information indicating one or more templates related to the answer sentence.

[0131] The instruction sentence improvement unit 302, which has received the difference information from the result determination unit 303, improves the instruction sentence constructed by the instruction sentence construction unit 301 so as to resolve the difference indicated by the difference information. For example, if the difference is that the answer sentence does not match the template related to the answer sentence, the instruction sentence improvement unit 302 may improve the instruction sentence constructed by the instruction sentence construction unit 301 by adding a sentence such as "Please write your answer using the following syntax" and one or more templates related to the answer sentence.

[0132] Thereafter, the instruction sentence improvement unit 302 inputs the improved instruction sentence to the large-scale language model unit 216. The instruction sentence improvement unit 302 transmits (or notifies) the improved instruction sentence to the result determination unit 303. The instruction sentence improvement unit 302 associates the difference indicated by the difference information with the difference between the improved instruction sentence and the instruction sentence before improvement (in other words, information indicating what improvement was made in relation to what difference), and stores this as an improvement record in the storage device 22.

[0133] If it is determined that the answer sentence output from the large-scale language model unit 216 matches the request to the information processing device 20a, the result determination unit 303 determines whether or not a predetermined action needs to be taken, based on the words included in the output answer sentence. If it is determined that a predetermined action needs to be taken, the result determination unit 303 instructs the action execution unit 304 to execute the predetermined action. The action execution unit 304 executes the predetermined action instructed by the result determination unit 303.

[0134] Here, the predetermined action may vary depending on the case in which the information processing device 20 a is applied. The predetermined action may include, for example, at least one of a notification or warning to a person as a target identified by the target identification unit 212, a notification or warning to a predetermined person, and a notification or request to a system external to the information processing device 20 a.

[0135] (Application Example of Information Processing Device) For example, the operation of the information processing device 20a when the information processing device 20a is applied to process management at a construction site will be specifically described.

[0136] As a premise, because there have been several incidents of construction materials and the like being stolen from construction sites at night, the information processing device 20a is applied to nighttime crime prevention. It is assumed that multiple cameras are installed to capture images of multiple locations on the construction site. At least one of the multiple cameras may be installed to capture images near the entrance and exit of the construction site. The first acquisition unit 211 acquires video images captured by each of the multiple cameras.

[0137] The external input unit 215 acquires, for example, construction technical information and construction plan documents as external input information. The external input information may include proper names unique to construction sites. The large-scale language model unit 216 learns the external input information using the method described in the third embodiment or some other method.

[0138] It is assumed that a template relating to the instruction sentence "[Person's name] performed [action] at [construction site name] at [time]. Is there anything suspicious?" is registered in advance in the instruction sentence construction unit 301. The name or title of the person as the target identified by the target identification unit 212 may be assigned to [Person's name] in the template. The "time," "location," and "action" included in the target state acquired by the target state acquisition unit 2131 may be assigned to [Time], [Construction site name], and [Action] in the template, respectively.

[0139] The result determination unit 303 has a dictionary that indicates words that trigger the execution of a predetermined action by the action execution unit 304. The dictionary may include positive words that evoke crime, such as "suspicious" or "potentially suspicious," and negative words that do not evoke crime, such as "nothing suspicious" or "nothing suspicious is found."

[0140] If the answer sentence output from the large-scale language model unit 216 contains the above-mentioned positive words, the result determination unit 303 instructs the action execution unit 304 to execute the predetermined action of "notifying a security company." If the answer sentence output from the large-scale language model unit 216 contains the above-mentioned negative words, the result determination unit 303 instructs the action execution unit 304 to execute the predetermined action of "recording video and notifying the site supervisor by email."

[0141] For example, if the target identified by the target identification unit 212 is "Worker S," the target state acquired by the state acquisition unit 2131 is "Date and time: 19:00, May 15, 2023," "Location: Front gate of construction site A," and "Action: Entered on foot," and the surrounding conditions acquired by the surrounding conditions acquisition unit 2132 is "It is dusk around here," the instruction sentence construction unit 302 may construct an instruction sentence such as "At 19:00, May 15, 2023, at the front gate of construction site A, it is dusk around here, and worker S entered on foot. Is there anything suspicious about this?"

[0142] It is assumed that the large-scale language model unit 216 has learned the behavior of each worker at construction site A on May 15, 2023. When the above instruction sentence is input to the large-scale language model unit 216, the large-scale language model unit 216 may refer to the learned behavior of each worker and recognize, for example, that worker S is the worker who performed reinforcement work. As a result, the large-scale language model unit 216 may infer that worker S is not a suspicious person. Then, the large-scale language model unit 216 may output a response sentence such as, "He entered outside of business hours, but there is nothing suspicious at this time."

[0143] The result determination unit 303 determines whether the answer sentence output from the large-scale language model unit 216 contains a word contained in the dictionary. In the above case, the word "There is nothing suspicious" contained in the answer sentence corresponds to a negative word contained in the dictionary. Therefore, the result determination unit 303 instructs the action execution unit 304 to execute "recording video and notifying the site supervisor by email." As a result, the action execution unit 304 executes "recording video and notifying the site supervisor by email."

[0144] Subsequently, if the target identified by the target identification unit 212 is "Worker S," the target state acquired by the state acquisition unit 2131 is "Date and time: 19:10, May 15, 2023," "Location: front gate of construction site A," and "Action: Exit on foot," and the surrounding conditions acquired by the surrounding condition acquisition unit 2132 are "dusk around" and "bag in hand," the instruction sentence construction unit 302 may construct an instruction sentence such as "At 19:10, May 15, 2023, with dusk around, worker S exited the front gate of construction site A on foot. He had a bag in his hand. Is there anything suspicious?"

[0145] When the above instruction sentence is input to the large-scale language model unit 216, the large-scale language model unit 216 may recognize, in relation to the instruction sentence, for example, "At 19:00 on May 15, 2023, at the front gate of construction site A, it was dusk and worker S entered on foot. Is there anything suspicious?", that worker S stayed at construction site A for 10 minutes and left construction site A with his bag in his hand.

[0146] Here, the large-scale language model unit 216 may have learned the past behavior of each worker at the construction site A. The large-scale language model unit 216 may have recalled that workers sometimes come to retrieve items they left behind at night. The large-scale language model unit 216 may also have learned the details of incidents in which construction materials and the like have been stolen from construction sites. The large-scale language model unit 216 may have recalled that construction materials and heavy machinery are often stolen by criminals from construction sites. For this reason, the large-scale language model unit 216 may infer that the situation in which worker S has a bag in his or her hand does not constitute theft. As a result, the large-scale language model unit 216 may output a response sentence such as, "He or she entered and left the site outside of business hours, but nothing particularly suspicious appears."

[0147] The result determination unit 303 determines whether the answer sentence output from the large-scale language model unit 216 contains a word contained in the dictionary. In the above case, the word "nothing suspicious appears" contained in the answer sentence corresponds to a negative word contained in the dictionary. Therefore, the result determination unit 303 instructs the action execution unit 304 to "record video and notify the site supervisor by email." As a result, the action execution unit 304 executes "record video and notify the site supervisor by email."

[0148] For example, if the target identified by the target identification unit 212 is "unknown," the target state acquired by the state acquisition unit 2131 is "date and time: 20:00, May 15, 2023," "location: front gate of construction site A," and "action: enter by truck," and the surrounding conditions acquired by the surrounding conditions acquisition unit 2132 is "dusk around," the instruction sentence construction unit 302 may construct an instruction sentence such as "At 20:00, May 15, 2023, at the front gate of construction site A, it is dusk around, and an unknown person has entered by truck. Is there anything suspicious about this?"

[0149] When the above instruction sentence is input to the large-scale language model unit 216, the large-scale language model unit 216 may recognize that the accuracy of person recognition is insufficient due to the twilight conditions. The large-scale language model unit 216 may infer that an unknown person has entered the construction site after hours, and therefore there is a risk of suspicious activity. As a result, the large-scale language model unit 216 may output a response sentence such as, "It's twilight, so it's hard to see well, but it appears that an unknown person has entered after hours, and there is a risk of suspicious activity."

[0150] The result determination unit 303 determines whether the answer sentence output from the large-scale language model unit 216 contains a word contained in the dictionary. In the above case, the word "There is a risk of something suspicious" contained in the answer sentence corresponds to a positive word contained in the dictionary. Therefore, the result determination unit 303 instructs the action execution unit 304 to execute "report to a security company." As a result, the action execution unit 304 executes "report to a security company."

[0151] Subsequently, if the target identified by the target identification unit 212 is "unknown," the target status acquired by the status acquisition unit 2131 is "date and time: 20:30, May 15, 2023," "location: front gate of construction site A," and "action: leaving by truck," and the surrounding conditions acquired by the surrounding conditions acquisition unit 2132 are "dusk around" and "construction materials in the truck bed," the instruction sentence construction unit 302 may construct the following instruction sentence: "At 20:30, May 15, 2023, at the front gate of construction site A, it was dusk around, and an unknown person was leaving in a truck. Construction materials were loaded in the truck bed. Is there anything suspicious?"

[0152] The large-scale language model unit 216 may have learned the details of incidents in which construction materials and the like have been stolen from construction sites. The large-scale language model unit 216 may recall that construction materials and heavy machinery are often stolen by criminals from construction sites. For this reason, the large-scale language model unit 216 may estimate that the presence of construction materials in the bed of a truck poses a high risk of theft. As a result, the large-scale language model unit 216 may output a response sentence such as, "It appears that an unknown person has taken construction materials outside of business hours, so there is a risk of theft."

[0153] The result determination unit 303 determines whether the answer sentence output from the large-scale language model unit 216 contains a word contained in the dictionary. In the above case, the word "There is a risk of theft" contained in the answer sentence corresponds to a positive word contained in the dictionary. Therefore, the result determination unit 303 instructs the action execution unit 304 to execute "report to the security company." As a result, the action execution unit 304 executes "report to the security company."

[0154] As described above, the large-scale language model unit 216 learns the behavior of each worker and the context related to theft at construction sites, and can therefore estimate whether there is any suspicious activity based on what it has learned.

[0155] In addition, if the large-scale language model unit 216 outputs a response sentence "It appears that an unknown person has taken out construction materials outside of business hours, and there is a risk of theft" in response to an instruction sentence such as "At 20:30 on May 15, 2023, at the front gate of construction site A, it was dusk and an unknown person was leaving in a truck. The truck bed was loaded with construction materials. Is there anything suspicious?", and the dictionary of the result determination unit 303 contains "It is suspicious" and "There is nothing suspicious", but does not contain "There is a risk of theft", the result determination unit 303 may determine that the response sentence output from the large-scale language model unit 216 is not a result that conforms to the request to the information processing device 20a.

[0156] The result determination unit 303 may transmit difference information indicating that the answer sentence does not contain the words "suspicious" and "nothing suspicious" to the instruction sentence improvement unit 302. The instruction sentence improvement unit 302 may improve an instruction sentence such as "At 8:30 PM on May 15, 2023, at the front gate of construction site A, it was dusk around the area, and an unknown person was leaving in a truck. The truck bed was loaded with construction materials. Isn't there anything suspicious?" to, for example, "At 8:30 PM on May 15, 2023, at the front gate of construction site A, it was dusk around the area, and an unknown person was leaving in a truck. The truck bed was loaded with construction materials. Isn't there anything suspicious? Please answer using the following words: 'suspicious', 'nothing suspicious'."

[0157] (Technical Effect) For example, the information processing device 20a applied to construction site process management may learn information necessary for process management, such as construction technical information and process plans, as well as the behavior of each worker at the construction site. As described above, when a crime prevention instruction, such as "Is there anything suspicious?", is input to the large-scale language model unit 216, the third embodiment 16 may output a response to the crime prevention instruction by referring to the learned information necessary for process management. In other words, the information processing device 20a can change its intended use without modifying or retraining the large-scale language model unit 216. In other words, the information processing device 20a can accommodate purposes different from its original intended use without modifying or retraining the large-scale language model unit 216.

[0158] In particular, in the information processing device 20a, the instruction sentence construction unit 301 automatically constructs an instruction sentence based on, for example, the output of the target identification unit 212 and the second acquisition unit 213 (in other words, based on an event that occurred in the real world). In other words, in the information processing device 20a, the user (i.e., a human being) does not need to make a situational judgment and give an instruction. Automatic construction of an instruction sentence is impossible with conventional mathematical models.

[0159] Furthermore, by cooperation between the instruction sentence improvement unit 302 and the result determination unit 303, it is possible to automatically improve an instruction sentence so that a reply sentence that conforms to the request to the information processing device 20a is output from the large-scale language model unit 216. Furthermore, by cooperation between the result determination unit 303 and the action execution unit 304, it is possible to automatically take an appropriate action for the reply sentence output from the large-scale language model unit 216.

[0160] <Supplementary Notes> The following supplementary notes are further disclosed regarding the above-described embodiment.

[0161] (Supplementary Note 1) An information processing device comprising: a first acquisition means for acquiring a moving image; an object identification means for identifying an object from the moving image; a second acquisition means for acquiring an object state that is the state of the object; a verbalization means for generating a sentence indicating the object state related to the object from the object and the object state; and a learning means for training a large-scale language model with the sentence generated by the verbalization means.

[0162] (Supplementary Note 2) The information processing device described in Supplementary Note 1, wherein the second acquisition means includes a state acquisition means that acquires the target state from the video, and a surrounding situation acquisition means that acquires the surrounding situation of the target from the video, and the verbalization means generates a sentence indicating the target state related to the target based on the target, the target state, and the surrounding situation.

[0163] (Supplementary Note 3) The information processing device according to Supplementary Note 1 or 2, wherein the sentence indicating the target state related to the target is a sentence conforming to the grammar of one natural language.

[0164] (Supplementary Note 4) The information processing device according to any one of Supplementary Notes 1 to 3, further comprising: an instruction sentence construction means for generating an instruction sentence for at least one of prediction and inference regarding the object from the object and the object state.

[0165] (Supplementary Note 5) An information processing device according to Supplementary Note 4, comprising: a determination means for determining whether an output result output from the large-scale language model when the instruction sentence is input to the large-scale language model is a result that conforms to a request; and an instruction sentence improvement unit that improves the instruction sentence when it is determined that the output result does not conform to the request.

[0166] (Supplementary Note 6) An information processing device according to Supplementary Note 4 or 5, comprising: a determination means for determining whether or not to perform a predetermined action based on an output result output from a large-scale language model when the instruction sentence is input to the large-scale language model; and an execution means for performing the predetermined action when it is determined that the predetermined action should be performed.

[0167] (Supplementary Note 7) The information processing device according to any one of Supplementary Notes 1 to 6, wherein the target identification means identifies a person as the target through biometric authentication using the moving image.

[0168] (Supplementary Note 8) The information processing device according to any one of Supplementary Notes 1 to 7, wherein the object identification means identifies the object by performing image recognition processing on the moving image.

[0169] (Supplementary Note 9) An information processing method comprising: acquiring a video; identifying an object from the video; acquiring an object state that is a state of the object; generating a sentence indicating the object state related to the object from the object and the object state; and training a large-scale language model to learn the generated sentence.

[0170] (Supplementary Note 10) A recording medium having recorded thereon a computer program for causing a computer to execute an information processing method, which includes acquiring a video image, identifying an object from the video image, acquiring an object state that is a state of the object, generating a sentence indicating the object state related to the object from the object and the object state, and training a large-scale language model with the generated sentence.

[0171] This disclosure is not limited to the above-described embodiments, but may be modified as appropriate within the scope of the claims and the gist or idea of ​​the invention as can be read from the entire specification, and information processing devices, information processing methods, and recording media that involve such modifications are also included in the technical scope of this disclosure.

[0172] 10, 10a, 20, 20a Information processing device 11, 211 First acquisition unit 12, 212 Target identification unit 13, 213 Second acquisition unit 14, 214 Verbalization unit 131, 2131 State acquisition unit 132, 2132 Surrounding situation acquisition unit 215 External input unit 216 Large-scale language model unit 217 Answer display unit 301 Instruction sentence construction unit 302 Instruction sentence improvement unit 303 Result determination unit 304 Action execution unit

Claims

1. a first acquisition means for acquiring a moving image; an object identification means for identifying an object from the moving image; a second acquisition means for acquiring a target state, the target state being a state of the target; a verbalization means for generating a sentence indicating the target state relating to the target from the target and the target state; a learning means for learning the sentences generated by the language generation means into a large-scale language model; An information processing device comprising:

2. The second acquisition means a state acquisition means for acquiring the target state from the moving image; a surrounding situation acquisition means for acquiring a surrounding situation of the object from the moving image; and The verbalization means generates a sentence indicating the target state related to the target based on the target, the target state, and the surrounding circumstances. The information processing device according to claim 1 .

3. The sentence indicating the object state regarding the object is a sentence conforming to the grammar of one natural language. The information processing device according to claim 1 .

4. an instruction sentence construction means for generating an instruction sentence for at least one of prediction and inference regarding the object from the object and the object state; an output means for outputting the result of inputting the instruction sentence into the large-scale language model; The information processing device according to claim 1 , comprising:

5. a determination means for determining whether an output result output from the large-scale language model when the directive sentence is input to the large-scale language model is a result that meets a request; an instruction sentence improving unit that improves the instruction sentence when it is determined that the output result does not meet the requirement; The information processing device according to claim 4 , comprising:

6. a determination means for determining whether or not to perform a predetermined action based on an output result output from a large-scale language model in response to the instruction sentence being input to the large-scale language model; an execution means for executing the predetermined action when it is determined that the predetermined action should be executed; The information processing device according to claim 4 , comprising:

7. The target identification means identifies a person as the target by biometric authentication using the moving image. The information processing device according to claim 1 .

8. The object identification means identifies the object by performing image recognition processing on the moving image. The information processing device according to claim 1 .

9. Acquire video images, Identifying an object from the video; acquiring a target state that is a state of the target; generating a sentence indicating the target state relating to the object from the object and the target state; The generated sentences are trained into a large-scale language model. Information processing methods.

10. On the computer, Acquire video images, Identifying an object from the video; acquiring a target state that is a state of the target; generating a sentence indicating the target state relating to the object from the object and the target state; The generated sentences are trained into a large-scale language model. A recording medium on which a computer program for executing an information processing method is recorded.