Information processing device, information processing method, and recording medium

Abstract

An information processing device according to the present invention comprises: an acquisition means that acquires observation information, which is at least one of sensor data, an image, a video, and audio and which has been acquired through observation of the real world; a first generation means that generates first event data, which pertains to a single event indicated by the observation information and which has a structure data format; a second generation means that extracts a feature vector from the observation information so as to generate second event data, which pertains to the single event and which has a vector data format; and a storage means that stores the first event data and the second event data.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing method, and a recording medium.

[0002] As an example of this type of device, a device has been proposed that generates a prompt for inputting into a large-scale language model by adding reference information to an input question sentence (see Patent Document 1).

[0003] Patent No. 7313757

[0004] An object of this disclosure is to provide an information processing device, an information processing method, and a recording medium that aim to improve the technology related to the prior art documents mentioned above.

[0005] One aspect of an information processing device includes an acquisition means for acquiring observation information, which is at least one of sensor data, images, video, and audio, acquired by observing the real world; a first generation means for generating first event data in the form of structured data relating to an event indicated by the observation information; a second generation means for generating second event data in the form of vector data relating to the event by extracting a feature vector from the observation information; and a storage means for storing the first event data and the second event data.

[0006] One aspect of the information processing method acquires observation information, which is at least one of sensor data, images, video, and audio, obtained by observing the real world, generates first event data in the form of structured data related to an event indicated by the observation information, extracts a feature vector from the observation information, and generates second event data in the form of vector data related to the event, and stores the first event data and the second event data.

[0007] One aspect of the recording medium has recorded thereon a computer program for causing a computer to execute an information processing method that acquires observation information, which is at least one of sensor data, images, video, and audio obtained by observing the real world, generates first event data in the form of structured data related to an event indicated by the observation information, generates second event data in the form of vector data related to the event by extracting a feature vector from the observation information, and stores the first event data and the second event data.

[0008] FIG. 1 is a block diagram showing an example of the configuration of an information processing device according to an embodiment. FIG. 2 is a flowchart showing the operation of the information processing device according to an embodiment. FIG. 3 is a block diagram showing another example of the configuration of an information processing device according to an embodiment. FIG. 4 is a block diagram showing an example of the configuration of a arithmetic device according to an embodiment. FIG. 5 is a block diagram showing another example of the configuration of a arithmetic device according to an embodiment. FIG. 6 is a block diagram showing another example of the configuration of a arithmetic device according to an embodiment. FIG. 7 is a block diagram showing another example of the configuration of a arithmetic device according to an embodiment. FIG. 8 is a block diagram showing another example of the configuration of a arithmetic device according to an embodiment.

[0009] First Embodiment A first embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to Fig. 1 and Fig. 2. In the following, the first embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 10.

[0010] 1 , an information processing device 10 includes an acquisition unit 11, a first generation unit 12, a second generation unit 13, and a storage unit 14. The acquisition unit 11 acquires observation information. The observation information is at least one of sensor data, images, videos (i.e., videos), and audio acquired by observing the real world.

[0011] The sensor data may be, for example, sensor data related to at least one of a temperature sensor and a human presence sensor. The sensor data may be, for example, data acquired by a predetermined reading device by reading at least one of a barcode, a two-dimensional code, and an IC (Integrated Circuit) chip. Here, the human presence sensor can detect the presence of a person in a space corresponding to the detection range of the human presence sensor. Therefore, the sensor data related to the human presence sensor can be said to be data obtained by observing the presence of a person in the space. At least one of a barcode, a two-dimensional code, and an IC chip may be used, for example, for managing the entry and exit of people. Furthermore, at least one of a barcode, a two-dimensional code, and an IC chip may be used for logistics management. Therefore, the data acquired by the predetermined reading device can be said to be data obtained by observing the movement of at least one of a person and an object.

[0012] The image may be, for example, an image generated by detecting light in the visible wavelength region with a camera, or an image generated by detecting light in the infrared wavelength region with a camera. Similarly, the video may be, for example, an image generated by detecting light in the visible wavelength region with a camera, or an image generated by detecting light in the infrared wavelength region with a camera. The camera may be a camera whose position is fixed, a camera attached to a moving object, or a camera carried by a person. Note that the camera attached to a moving object may be at least one of a camera mounted on a vehicle (e.g., an automobile, a railcar, an airplane, a helicopter, etc.), a camera equipped on an unmanned moving object (e.g., a robot, a drone, etc.), and a camera worn by at least one of a person and an animal (a so-called wearable camera).

[0013] The term "audio" is not limited to human voices, but may also include, for example, animal cries, vehicle running sounds, machine operating sounds, etc. In other words, "audio" according to this embodiment may refer to environmental sounds. The microphone that detects the audio may be a microphone that is fixed in position, a microphone attached to a moving object, or a microphone carried by a person. The microphone attached to a moving object may be at least one of a microphone mounted on a vehicle (e.g., an automobile, a railcar, an airplane, a helicopter, etc.), a microphone equipped on an unmanned moving object (e.g., a robot, a drone, etc.), and a microphone worn by at least one of a person and an animal.

[0014] The first generation unit 11 generates first event data in the form of structured data relating to one event indicated by the observation information by structuring the observation information by applying it to a mathematical model. "Having the format of structured data" may mean that the data is shaped into a predetermined structure. Examples of structured data formats include the Comma Separated Values ​​(CSV) format, the Extensible Markup Language (XML) format, and the JavaScript Object Notation (JSON) format. Examples of mathematical models include algorithms written using approximate expressions, conditional branching, etc., and models created by machine learning.

[0015] The first event data is not limited to data structured by fitting observation information to a mathematical model, but may also be observation information (e.g., observation values) in the form of structured data. In this case, the observation information may be sensor data. For example, the first event data may include nine types of sensor data (e.g., acceleration in the X-axis direction, acceleration in the Y-axis direction, acceleration in the Z-axis direction, angular velocity around the X-axis, angular velocity around the Y-axis, angular velocity around the Z-axis, and three components of a three-dimensional vector indicating north (i.e., dx, dy, dz)) acquired at predetermined time intervals by a triaxial acceleration sensor, a triaxial angular velocity sensor, and a triaxial geomagnetic sensor included in a terminal device such as a smartphone. Furthermore, the three-axis angles calculated by estimating the terminal device's attitude in three-dimensional space using the nine types of sensor data and a mathematical model, such as an attitude estimation model using a Kalman filter, may be generated as the first event data.

[0016] The second generation unit 13 generates second event data in the form of vector data relating to one event by extracting a feature vector from the observation information. Here, the second event data may be, for example, data representing one event as a numerical array (i.e., a vector). In this case, the numerical array representing one event may be the feature vector itself extracted from the observation information. In other words, extracting a feature vector from the observation information may be equivalent to generating second event data.

[0017] The storage unit 14 stores first event data related to an event generated by the first generation unit 12 and second event data related to the event generated by the second generation unit 13. The storage unit 14 may store the first event data and the second event data in a linked manner. For example, the first event data and the second event data related to the event may be linked to identification information for identifying the event. In this case, it can be said that the first event data and the second event data related to the event are linked via the identification information related to the event. In addition to the first event data and the second event data, the storage unit 14 may also store observation information (e.g., at least one of sensor data, images, video, and audio).

[0018] The operation of the information processing device 10 will be further described with reference to the flowchart of FIG. 2. In FIG. 2, the acquisition unit 11 of the information processing device 10 acquires observation information (step S101). The first generation unit 12 of the information processing device 10 generates first event data related to one event by structuring the observation information by applying it to a mathematical model (step S102). The second generation unit 13 of the information processing device 10 generates second event data related to one event by extracting a feature vector from the observation information (step S103). Note that the processing of step S102 and the processing of step S103 may be performed in parallel or one after the other. Thereafter, the storage unit 14 stores the first event data and the second event data (step S104).

[0019] In this way, the information processing device 10 acquires observation information, which is at least one of sensor data, images, video, and audio, obtained by observing the real world, and structures the observation information by fitting it to a mathematical model to generate first event data in the form of structured data related to an event indicated by the observation information.It also extracts feature vectors from the observation information to generate second event data in the form of vector data related to the event, and performs an information processing method in which the first event data and the second event data are stored.

[0020] The information processing device 10 described above may be realized by a computer reading a computer program recorded on a recording medium. In this case, the recording medium may have recorded thereon a computer program for causing the computer to execute an information processing method that acquires observation information, which is at least one of sensor data, images, video, and audio obtained by observing the real world, applies the observation information to a mathematical model to structure it, thereby generating first event data in the form of structured data related to an event indicated by the observation information, extracts feature vectors from the observation information, thereby generating second event data in the form of vector data related to the event, and stores the first event data and the second event data.

[0021] (Technical Effect) For example, by applying observation information generated by observing the real world using a sensor or the like to a mathematical model as explanatory variables and then performing threshold processing or the like on the objective variable calculated by the mathematical model, it is possible to recognize events in the real world. However, the events that can be recognized using this method are limited to those defined by the objective variable.

[0022] The real world is complex and constantly changing. For this reason, it is not possible to predetermine all events that may occur in the real world. Therefore, the above-mentioned method may result in oversights (in other words, false negatives). On the other hand, if the predetermined content is relaxed to prevent oversights, there is a possibility that misrecognitions or false detections (in other words, false positives) may occur.

[0023] For example, assume that the observation information is an image captured by a surveillance camera installed in a park. For example, assume that the event defined by the objective variable is "the presence of a dog." In this case, the above method can recognize the event "there is a dog in the park" from the observation information. For example, if a bear appears in the park, the above method cannot recognize the bear from the observation information.

[0024] In contrast, in the information processing device 10, first event data in the form of structured data and second event data in the form of vector data are stored in the storage unit 14. By using the first event data in the form of structured data, it is possible to quickly and accurately extract events defined by target variables from the storage unit 14. By using the second event data in the form of vector data, it is possible to extract from the storage unit 14 not only events that match the desired event, but also events similar to the desired event.

[0025] For example, suppose the observation information is an image captured by a surveillance camera installed in a park. For example, dogs and bears are similar in that they both walk on all fours. For example, depending on the distance between the camera and the subject, the bear in the image may resemble a dog. Therefore, for example, an event indicated by an image of a bear as observation information may be stored in the memory unit 14 as an event of "presence of a dog." Therefore, if a bear appears in the park, for example, by searching for the event of "presence of a bear" using vector data (i.e., second event data), an event indicating a bear stored as the event of "presence of a dog" may be extracted from the memory unit 14.

[0026] That is, according to the information processing device 10, the first event data in the form of structure data and the second event data in the form of vector data are stored in the storage unit 14, so that in addition to the events defined by the objective variables (in other words, the events that are anticipated in advance), unanticipated events can also be extracted from the storage unit 14. Therefore, according to the information processing device 10, in addition to the events that are anticipated in advance, it is also possible to deal with unexpected events that may occur in the complex and constantly changing real world.

[0027] Second Embodiment A second embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to Figures 3 and 4. Hereinafter, the second embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 100. Note that, for the second embodiment, descriptions that overlap with the first embodiment will be omitted as appropriate.

[0028] 3, the information processing device 100 includes a calculation device 110, a storage device 120, a communication device 130, an input device 140, and an output device 150. The information processing device 100 does not necessarily include at least one of the input device 140 and the output device 150. The storage device 120 is a component corresponding to the "storage unit 14" in the first embodiment described above.

[0029] The computing device 110 may include a processor 110a. The computing device 110 may include other processors in addition to the processor 110a. That is, the computing device 110 may include one or more processors. The processor 110a may be a multi-core processor. When the computing device 110 includes a single processor 110a that is a multi-core processor, the computing device 110 can be said to logically include multiple processors.

[0030] The processor 110a may be, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a tensor processing unit (TPU), and a quantum processor.

[0031] The storage device 120 may include a memory 120a. The storage device 120 may include other memories in addition to the memory 120a. That is, the storage device 120 may include one or more memories. The memory 120a may be, for example, at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and an optical disk array. Therefore, the storage device 120 may include the memory 120a as a non-transitory recording medium.

[0032] The storage device 120 can store desired data. A computer program 121 to be executed by the arithmetic device 110 may be stored in the memory 120a of the storage device 120. The storage device 120 may temporarily store data that is temporarily used by the arithmetic device 110 when the arithmetic device 110 is executing the computer program 121.

[0033] The computer program 121 may be recorded on a computer-readable, non-transitory recording medium. In this case, the computer program 121 may be stored in the memory 120a by reading the recording medium using a recording medium reading device (not shown) included in the information processing device 100. The recording medium may be at least one of an optical disk, a magnetic medium, a magneto-optical disk, a semiconductor memory, and any other medium capable of storing a program. The computer program 121 may be acquired (in other words, downloaded) from a device (not shown) external to the information processing device 100 via the communication device 130. The acquired computer program 121 may be stored in the memory 120a.

[0034] The communication device 130 may be capable of communicating with devices external to the information processing device 100. The communication device 130 may perform wired communication or wireless communication.

[0035] The input device 140 is a device capable of accepting information input to the information processing device 100 from outside. The input device 140 may include an operation device (e.g., a keyboard, a mouse, a touch panel, etc.) that can be operated by a user of the information processing device 100. The input device 140 may include a recording medium reading device that can read information recorded on a recording medium that is detachable from the information processing device 100, such as a USB (Universal Serial Bus) memory. Note that when information is input to the information processing device 100 via the communication device 130 (in other words, when the information processing device 100 acquires information via the communication device 130), the communication device 130 may function as an input device.

[0036] The output device 150 is a device capable of outputting information to the outside of the information processing device 100. The output device 150 may output visual information such as text or images, auditory information such as sound, or tactile information such as vibration, as the information. The output device 150 may include, for example, at least one of a display, a speaker, a printer, and a vibration motor. The output device 150 may be capable of outputting information to a recording medium that is detachable from the information processing device 100, such as a USB memory. Note that when the information processing device 100 outputs information via the communication device 130, the communication device 130 may function as the output device.

[0037] The processor 110a of the arithmetic device 110 may execute the processing to be performed by the information processing device 100 together with the memory 120a of the storage device 120 in which the computer program 121 is stored (in other words, together with the memory 120a and the computer program 121 stored in the memory 120a). For example, the processor 110a may execute the computer program 121 to realize, within the arithmetic device 110, a logical functional block for executing the processing to be performed by the information processing device 100.

[0038] 4, the arithmetic device 110 may have, as logically realized functional blocks or physically realized processing circuits, an acquisition unit 111, a structuring unit 112, a verbalization unit 113, and a feature vector extraction unit 114. Note that at least one of the acquisition unit 111, the structuring unit 112, the verbalization unit 113, and the feature vector extraction unit 114 may be realized in a form in which logical functional blocks and physical processing circuits (i.e., hardware) are mixed.

[0039] When the acquisition unit 111, the structuring unit 112, the linguistic unit 113, and the feature vector extraction unit 114 are realized as functional blocks, the acquisition unit 111, the structuring unit 112, the linguistic unit 113, and the feature vector extraction unit 114 may be realized by a single processor. Alternatively, the acquisition unit 111, the structuring unit 112, the linguistic unit 113, and the feature vector extraction unit 114 may be realized by different processors. Alternatively, some of the acquisition unit 111, the structuring unit 112, the linguistic unit 113, and the feature vector extraction unit 114 may be realized by a single processor, and the remaining parts of the acquisition unit 111, the structuring unit 112, the linguistic unit 113, and the feature vector extraction unit 114 may be realized by one or more processors different from the single processor.

[0040] The "acquisition unit 111," "structuration unit 112," and "feature vector extraction unit 114" are components corresponding to the "acquisition unit 11," "first generation unit 12," and "second generation unit 13" in the first embodiment described above, respectively.

[0041] The acquisition unit 111 may acquire, as observation information, sensor data output from the sensor 21 by the sensor 21 observing the real world. The acquisition unit 111 may acquire, as observation information, at least one of an image and a video generated by the camera 22 by the camera 22 observing the real world. The acquisition unit 111 may acquire, as observation information, sound detected by the microphone 23 by the microphone 23 observing the real world.

[0042] The structuring unit 112 generates structured data relating to one event indicated by the observation information by structuring the observation information acquired by the acquiring unit 111 through fitting it to a mathematical model. The structured data generated by the structuring unit 112 corresponds to the "first event data" in the first embodiment described above.

[0043] The verbalization unit 113 generates a sentence in natural language relating to an event indicated by the observation information by performing natural language processing on the observation information acquired by the acquisition unit 111. Note that the verbalization unit 113 may generate a sentence in natural language using, for example, at least one of a term dictionary and a syntax template.

[0044] The feature vector extraction unit 114 generates vector data relating to one event indicated by the observation information (specifically, indicated by the natural language sentence) by extracting a feature vector from the natural language sentence generated by the verbalization unit 113. The vector data generated by the feature vector extraction unit 114 corresponds to the "second event data" in the first embodiment described above. A neural network such as a Sentence Transformer may be used to extract the feature vector.

[0045] The structure data generated by the structuring unit 112 , the natural language sentences generated by the languageizing unit 113 , and the vector data generated by the feature vector extraction unit 114 are stored in the storage device 120 .

[0046] The structure data, natural language sentences, and vector data may be stored in the storage device 120 in the data structure shown below, for example.

[0047] collection = { ids = [“id1”, “id2”, “id3”, …], metadatas = [{“chapter”: “3”, “verse”: “16”}, {“chapter”: “3”, “verse”: “5”}, {“chapter”: “29”, “verse”: “11”}, …], documents = [“doc1”, “doc2”, “doc3”, …], embeddings = [[1.1, 2.3, 3.2], [4.5, 6.9, 4.4], [1.1, 2.3, 3.2], …],} Here, “id1”, “id2”, and “id3”, for example, included in “ids”, are the IDs of each event. Examples of structured data for each event include {"chapter": "3", "verse": "16"}, {"chapter": "3", "verse": "5"}, and {"chapter": "29", "verse": "11"} in "metadatas." Examples of natural language text data for each event include "doc1," "doc2," and "doc3" in "documents." Examples of vector data for each event include [1.1, 2.3, 3.2], [4.5, 6.9, 4.4], and [1.1, 2.3, 3.2] in "embeddings." In the above data structure, for example, the structure data, sentence data, and vector data relating to the event “id1” are {“chapter”: “3”, “verse”: “16”}, “doc1”, and [1.1, 2.3, 3.2], respectively.

[0048] In addition, the storage device 120 may store observation information (e.g., at least one of sensor data, images, video, and audio) in addition to the structure data generated by the structuring unit 112, the natural language sentences generated by the languageizing unit 113, and the vector data generated by the feature vector extraction unit 114.

[0049] (Technical effect) As with the first embodiment described above, the information processing device 100 according to the second embodiment can respond to not only anticipated events but also unexpected events that may occur in the complex and constantly changing real world.

[0050] Third Embodiment A third embodiment relating to an information processing device, an information processing method, and a recording medium will be described with reference to Figures 3 and 5. Hereinafter, the third embodiment relating to an information processing device, an information processing method, and a recording medium will be described using an information processing device 100. Note that, with regard to the third embodiment, descriptions that overlap with the first and second embodiments will be omitted as appropriate.

[0051] 5, the arithmetic device 110 may include an acquisition unit 111, a structuring unit 112, and a multimodal feature vector extraction unit 114a as logically realized functional blocks or as physically realized processing circuits. That is, the information processing device 100 according to the third embodiment may include a multimodal feature vector extraction unit 114a instead of the verbalization unit 113 and the feature vector extraction unit 113 in the second embodiment. Note that at least one of the acquisition unit 111, the structuring unit 112, and the multimodal feature vector extraction unit 114a may be realized in a form in which logical functional blocks and physical processing circuits (i.e., hardware) are mixed.

[0052] When the acquisition unit 111, the structuring unit 112, and the multimodal feature vector extraction unit 114a are implemented as functional blocks, the acquisition unit 111, the structuring unit 112, and the multimodal feature vector extraction unit 114a may be implemented by a single processor. Alternatively, the acquisition unit 111, the structuring unit 112, and the multimodal feature vector extraction unit 114a may be implemented by different processors. Alternatively, parts of the acquisition unit 111, the structuring unit 112, and the multimodal feature vector extraction unit 114a may be implemented by a single processor, and the remaining parts of the acquisition unit 111, the structuring unit 112, and the multimodal feature vector extraction unit 114a may be implemented by one or more processors different from the single processor. The "multimodal feature vector extraction unit 114a" is a component corresponding to the "second generation unit 13" in the first embodiment described above.

[0053] The multimodal feature vector extraction unit 114a generates vector data relating to one event indicated by the observation information by extracting a multimodal feature vector from the observation information acquired by the acquisition unit 111. Note that a neural network such as a VAE (Variational Autoencoder) may be used to extract the multimodal feature vector.

[0054] A "multimodal feature vector" means, for example, a feature vector obtained by acquiring an event in the modal of an image and extracting the image from the image, a feature vector obtained by acquiring an event in the modal of an audio and extracting the audio from the audio, and a feature vector obtained by describing an event in the modal of a natural language sentence and extracting the audio from the audio, which are all located at the same or close to each other in vector space.

[0055] The structured data generated by the structuring unit 112 and the vector data generated by the multimodal feature vector extraction unit 114a are stored in the storage device 120. Note that the storage device 120 may also store observation information (e.g., at least one of sensor data, images, video, and audio) in addition to the structured data generated by the structuring unit 112 and the vector data generated by the multimodal feature vector extraction unit 114a.

[0056] (Technical Effects) As with the first embodiment, the information processing device 100 according to the third embodiment can handle not only anticipated events but also unexpected events that may occur in the complex and constantly changing real world. In particular, in the third embodiment, the multimodal feature vector extraction unit 114a extracts multimodal feature vectors from observed information. Therefore, compared to the second embodiment in which feature vectors are extracted from natural language sentences describing an event, the multimodal feature vector extracted by the multimodal feature vector extraction unit 114a can more accurately represent the characteristics of the event. As a result, when unexpected events are searched for using vector data, more appropriate events can be extracted from the storage device 120.

[0057] Fourth Embodiment A fourth embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to Fig. 3 and Fig. 6. Hereinafter, the fourth embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 100. Note that, with regard to the fourth embodiment, descriptions that overlap with the first to third embodiments will be omitted as appropriate.

[0058] In the fourth embodiment, a configuration will be described in which at least one of structure data and vector data relating to a desired event is extracted from structure data and vector data relating to each of a plurality of events stored in the storage device 120 of the information processing device 100.

[0059] 6, the arithmetic device 110 may include an event extraction unit 115 as a logically realized functional block or as a physically realized processing circuit. The event extraction unit 115 may include determination units 1151 and 1152 and a feature vector extraction unit 1153. Note that the event extraction unit 115 may be realized in a form in which a logical functional block and a physical processing circuit (i.e., hardware) are mixed.

[0060] Here, either the configuration described in the second embodiment or the configuration described in the third embodiment can be applied to the configuration for generating the structure data and vector data to be stored in the storage device 120. That is, the arithmetic device 110 may have an acquisition unit 111, a structuring unit 112, a verbalization unit 113, a feature vector extraction unit 114, and an event extraction unit 115. The arithmetic device 110 may have an acquisition unit 111, a structuring unit 112, a multimodal feature vector extraction unit 114a, and an event extraction unit 115.

[0061] The determination unit 1151 of the event extraction unit 115 may determine whether each of the plurality of structured data stored in the storage device 120 conforms to a rule that indicates the desired event described in the structured data. Note that the data described in the structured data may be created manually, or may be created from statistical analysis of past structured data related to real-world events.

[0062] The feature vector extraction unit 1153 of the event extraction unit 115 may extract feature vectors related to the event examples from the event examples that indicate the desired events and are written in natural language. Note that, for example, a neural network such as a Sentence Transformer may be used to extract the feature vectors.

[0063] The determination unit 1152 of the event extraction unit 115 may calculate the similarity between each of the plurality of vector data stored in the storage device 120 and the feature vector extracted by the feature vector extraction unit 1153. Note that the similarity may be calculated, for example, based on the distance between the vector data stored in the storage device 120 and the feature vector related to the event example, or may be calculated as cosine similarity. Based on the calculated similarity, the determination unit 1152 may determine whether each of the plurality of vector data stored in the storage device 120 is similar to the feature vector related to the event example.

[0064] For example, the event extraction unit 115 may extract one or more pieces of structure data determined by the determination unit 1151 to conform to the rule. For example, the event extraction unit 115 may extract one or more pieces of vector data determined by the determination unit 1152 to be similar to a feature vector. For example, the event extraction unit 115 may extract structure data and vector data related to one or more events corresponding to one or more pieces of vector data determined by the determination unit 1152 to be similar to the feature vector, among one or more events corresponding to one or more pieces of structure data determined by the determination unit 1151 to conform to the rule. For example, the event extraction unit 115 may extract structure data and vector data related to one or more events corresponding to one or more pieces of structure data determined by the determination unit 1152 to be similar to the feature vector, among one or more events corresponding to one or more pieces of vector data determined by the determination unit 1152 to be similar to the feature vector.

[0065] Note that the rules described in structured data may be input to the information processing device 100 by, for example, an IT engineer. On the other hand, the event examples described in natural language may be input to the information processing device 100 by, for example, an operator of the information processing device 100. Alternatively, for example, when the operator of the information processing device 100 inputs the event examples described in natural language to the information processing device 100 via the input device 140, the rules described in structured data may be automatically generated from the input event examples (i.e., natural language).

[0066] (Technical effect) As with the first embodiment described above, the information processing device 100 according to the fourth embodiment can respond to not only anticipated events but also unexpected events that may occur in the complex and constantly changing real world.

[0067] As described above, the event extraction unit 115 may extract structure data and vector data relating to one or more events corresponding to one or more vector data determined to be similar to a feature vector by the determination unit 1152, from one or more events corresponding to one or more structure data determined to be similar to a feature vector by the determination unit 1152. Furthermore, the event extraction unit 115 may extract structure data and vector data relating to one or more events corresponding to one or more structure data determined to be similar to a rule by the determination unit 1151, from one or more events corresponding to one or more vector data determined to be similar to a feature vector by the determination unit 1152.

[0068] With this configuration, for example, it is possible to narrow down events by limiting the time or place that exists as an explanatory variable, and then extract events that include items similar to items that do not exist as explanatory variables. Alternatively, it is possible to narrow down events that include items similar to items that do not exist as explanatory variables, and then extract events that correspond to the time or place that exists as an explanatory variable. In other words, it is possible to appropriately extract various real-world events.

[0069] Fifth Embodiment A fifth embodiment relating to an information processing device, an information processing method, and a recording medium will be described with reference to Figures 3 and 6. Below, the fourth embodiment relating to an information processing device, an information processing method, and a recording medium will be described using an information processing device 100. Note that, for the fifth embodiment, descriptions that overlap with the first to fourth embodiments will be omitted as appropriate.

[0070] In the above-described fourth embodiment, an event example described in natural language is input to the information processing device 10. In contrast to this, in the fifth embodiment, for example, at least one of sensor data, image, video, and audio indicating a desired case is input to the information processing device 10 as an event example.

[0071] The determination section 1151 of the event extraction section 115 may determine whether each of the plurality of structured data stored in the storage device 120 conforms to a rule that indicates a desired event and is described in the structured data.

[0072] The feature vector extraction unit 1153 of the event extraction unit 115 may extract a feature vector related to an event example from an event example that indicates a desired event and is at least one of sensor data, an image, a video, and an audio. Note that a neural network such as a VAE (Variational Autoencoder) may be used to extract the feature vector.

[0073] The determination unit 1152 of the event extraction unit 115 may calculate the similarity between each of the plurality of vector data stored in the storage device 120 and the feature vector extracted by the feature vector extraction unit 1153. Based on the calculated similarity, the determination unit 1152 may determine whether each of the plurality of vector data stored in the storage device 120 is similar to the feature vector related to the event example.

[0074] (Technical effect) As with the fourth embodiment described above, the information processing device 100 according to the fifth embodiment can respond to not only anticipated events but also unexpected events that may occur in the complex and constantly changing real world.

[0075] Sixth Embodiment A sixth embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to Fig. 3 and Fig. 7. Hereinafter, the sixth embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 100. Note that, with regard to the sixth embodiment, descriptions that overlap with the first to fifth embodiments will be omitted as appropriate.

[0076] In the sixth embodiment, a configuration will be described in which a large-scale language model 117 (in other words, a generative AI) learns events (corresponding to the above-mentioned desired events) corresponding to at least one of the structure data and vector data extracted by the above-mentioned event extraction unit 115. Note that the aspect described in the fourth embodiment or the aspect described in the fifth embodiment may be applied to the extraction of the structure data and vector data.

[0077] 7, the computing device 110 may include an event extraction unit 115, an acquisition unit 116, and a large-scale language model 117 as logically realized functional blocks or as physically realized processing circuits. Note that at least one of the event extraction unit 115, the acquisition unit 116, and the large-scale language model 117 may be realized in a form in which logical functional blocks and physical processing circuits (i.e., hardware) are mixed.

[0078] When the event extraction unit 115, the acquisition unit 116, and the large-scale language model 117 are realized as functional blocks, the event extraction unit 115, the acquisition unit 116, and the large-scale language model 117 may be realized by a single processor. Alternatively, the event extraction unit 115, the acquisition unit 116, and the large-scale language model 117 may be realized by different processors. Alternatively, parts of the event extraction unit 115, the acquisition unit 116, and the large-scale language model 117 may be realized by a single processor, and the remaining parts of the event extraction unit 115, the acquisition unit 116, and the large-scale language model 117 may be realized by one or more processors different from the single processor.

[0079] The large-scale language model 117 may be a language model generated by inputting a large amount of sentences (e.g., a huge amount of text on the web, past records similar to the subject of analysis, etc.) into a model having a model structure capable of handling a large number of variables and parameters, such as a Transformer, and then machine learning the model using unsupervised learning such as self-attention.

[0080] The acquisition unit 116 acquires natural language sentences that express events that correspond to at least one of the structure data and the vector data extracted by the event extraction unit 115. Here, as in the second embodiment described above, when natural language sentences are stored in the storage device 120, the acquisition unit 116 may acquire, from the natural language sentences stored in the storage device 120, natural language sentences that correspond to events that correspond to at least one of the structure data and the vector data extracted by the event extraction unit 115. Note that a case in which no natural language sentences are stored in the storage device 120 will be described in a seventh embodiment described later.

[0081] The acquisition unit 116 inputs the acquired natural language sentence into the large-scale language model 117. The large-scale language model 117 learns the input natural language sentence. Here, "learning" may mean, for example, a process of interpreting the content of the sentence based on the meaning of each word contained in the sentence, the relationship between words, etc. By learning the input natural language sentence into the large-scale language model 117, the large-scale language model 117 can analyze the meaning and context of the sentence. As a result, the large-scale language model can analyze, with a high degree of accuracy, the real world, where causes and effects exist infinitely and are intricately intertwined.

[0082] In addition to the natural language sentences acquired by the acquisition unit 116, the large-scale language model 117 receives instruction statements written in natural language input by the operator of the information processing device 100. The instruction statements may be, for example, sentences indicating what the operator wants the large-scale language model 117 to analyze.

[0083] The large-scale language model 117 outputs an analysis result based on the natural language sentence and the instruction sentence acquired by the acquisition unit 116. For example, the calculation device 110 may control the output device 150 to display the analysis result output from the large-scale language model 117.

[0084] For example, it is assumed that the information processing device 100 is operated by a park management company. For example, it is assumed that the observation information is an image captured by a surveillance camera installed in the park. It is assumed that the storage device 120 stores structure data, vector data, and natural language sentences relating to an event shown in the image captured by the surveillance camera. For example, it is assumed that the park management company receives a report that "a pervert has been seen."

[0085] In such a situation, an operator of the information processing device 100 (e.g., an employee of a park management company) may input rules and example events into the information processing device 100 via the input device 140, thereby causing the event extraction unit 115 to extract one or more events indicating human behavior at a specified date and time. The acquisition unit 116 may acquire a natural language sentence representing one or more events extracted by the event extraction unit 115. The operator may further input an instruction to the information processing device 100 via the input device 140, such as, "Please extract people suspected of being perverts from a series of actions for each individual." The large-scale language model 117 may output an analysis result based on the natural language sentence acquired by the acquisition unit 116 and the instruction. The large-scale language model 117 may output, as the analysis result, a sentence such as, "On XX month, XX day, at XX:00, XX minute, there was a person suspected of being a pervert at XXX in the park."

[0086] (Technical Effect) As with the first embodiment, the information processing device 100 according to the sixth embodiment can handle not only anticipated events but also unexpected events that may occur in the complex and constantly changing real world. In particular, the information processing device 100 according to the sixth embodiment can cause the large-scale language model 117 to perform analysis based on at least a portion of the multiple events stored in the storage device 120. In other words, the task of checking the multiple events stored in the storage device 120 can be automated.

[0087] Seventh Embodiment A seventh embodiment relating to an information processing device, an information processing method, and a recording medium will be described with reference to Figures 3, 7, and 8. Below, the seventh embodiment relating to an information processing device, an information processing method, and a recording medium will be described using an information processing device 100. Note that, with regard to the seventh embodiment, descriptions that overlap with the first to sixth embodiments will be omitted as appropriate.

[0088] 8, the arithmetic device 110 may include an acquisition unit 111, a structuring unit 112, and a multimodal feature vector extraction unit 114a as logically realized functional blocks or as physically realized processing circuits. That is, the information processing device 100 according to the seventh embodiment may include the acquisition unit 111, the structuring unit 112, and the multimodal feature vector extraction unit 114a, similar to the information processing device 100 according to the third embodiment described above.

[0089] In the seventh embodiment, the acquisition unit 111 may store the observation information in the storage device 120. The observation information may be, for example, sensor data output from the sensor 21, at least one of an image and a video generated by the camera 22, or audio detected by the microphone 23. In other words, the observation information is unprocessed data. For this reason, the observation information may be referred to as raw data. As shown in FIG. 8 , the storage device 120 does not store a natural language sentence related to an event indicated by the observation information.

[0090] 7 acquires a natural language sentence representing an event corresponding to at least one of the structure data and the vector data extracted by the event extraction unit 115. For example, the acquisition unit 116 may acquire observation information related to an event corresponding to at least one of the structure data and the vector data extracted by the event extraction unit 115 from among multiple events stored in the storage device 120. The acquisition unit 116 may further perform natural language processing on the acquired observation information to generate a natural language sentence related to one or more events indicated by the observation information. As a result, the acquisition unit 116 may acquire a natural language sentence representing an event corresponding to at least one of the structure data and the vector data extracted by the event extraction unit 115.

[0091] (Technical Effect) According to the information processing device 100 of the seventh embodiment, similar to the sixth embodiment described above, it is possible to deal with not only anticipated events but also unexpected events that may occur in the complex and constantly changing real world. According to the information processing device 100 of the seventh embodiment, similar to the sixth embodiment described above, it is possible to automate the task of checking multiple events stored in the storage device 120.

[0092] Eighth Embodiment An eighth embodiment relating to an information processing device, an information processing method, and a recording medium will be described with reference to Fig. 3 and Fig. 9. Below, the eighth embodiment relating to an information processing device, an information processing method, and a recording medium will be described using an information processing device 100. Note that with regard to the seventh embodiment, descriptions that overlap with the first to seventh embodiments will be omitted as appropriate.

[0093] 9, the arithmetic device 110 may include an event extraction unit 115, an acquisition unit 116, and a large-scale multimodal model 117a as logically realized functional blocks or as physically realized processing circuits. That is, the information processing device 100 according to the eighth embodiment may include a large-scale multimodal model 117a instead of the large-scale language model 117 in the sixth and seventh embodiments. Note that at least one of the event extraction unit 115, the acquisition unit 116, and the large-scale multimodal model 117a may be realized in a form in which logical functional blocks and physical processing circuits (i.e., hardware) are mixed.

[0094] The large-scale multimodal model 117a may be a model generated by inputting a large amount of data (e.g., text, images, video, audio, etc.) into a model having a model structure capable of handling a large number of variables and parameters, and then machine learning the model using unsupervised learning.

[0095] When the structure data is extracted by the event extraction unit 115, the acquisition unit 116 may acquire vector data related to the event corresponding to the extracted structure data from the storage device 120. Note that when the storage device 120 stores at least one of observation information (i.e., raw data) and natural language sentences related to the event indicated by the observation information, the acquisition unit 116 may acquire at least one of the vector data, observation information, and natural language sentences related to the event corresponding to the extracted structure data.

[0096] When vector data is extracted by the event extraction unit 115, the acquisition unit 116 may acquire structure data related to the event corresponding to the extracted vector data from the storage device 120. Note that when the storage device 120 stores at least one of observation information (i.e., raw data) and natural language sentences related to the event indicated by the observation information, the acquisition unit 116 may acquire at least one of the structure data, observation information, and natural language sentences related to the event corresponding to the extracted vector data.

[0097] The acquisition unit 116 inputs at least one of the acquired structure data and vector data (or at least one of the structure data, vector data, observation information, and natural language sentences) into the large-scale multimodal model 117a. The large-scale multimodal model 117a learns the input data. By learning the input data, the large-scale multimodal model 117a can analyze, with a high degree of accuracy, the real world, where an infinite number of causes and effects exist and are intricately intertwined.

[0098] In addition to the data acquired by the acquisition unit 116 (for example, at least one of structure data, vector data, observation information, and natural language sentences), the large-scale multimodal model 117a receives an instruction statement written in natural language input by an operator of the information processing device 100. Note that, instead of or in addition to the instruction statement, for example, at least one of an image, video, and audio indicating what the operator wants the large-scale multimodal model 117a to analyze may be input to the large-scale multimodal model 117a.

[0099] The large-scale multimodal model 117a outputs an analysis result based on the data and the instruction statement acquired by the acquisition unit 116. For example, the calculation device 110 may control the output device 150 to display the analysis result output from the large-scale multimodal model 117a.

[0100] (Technical Effect) According to the information processing device 100 of the eighth embodiment, in the same way as in the first embodiment described above, it is possible to deal with not only anticipated events but also unexpected events that may occur in the complex and constantly changing real world. According to the information processing device 100 of the eighth embodiment, it is possible to automate the task of checking multiple events stored in the storage device 120, in the same way as in the sixth and seventh embodiments described above.

[0101] While natural language sentences must be input to the large-scale language model 117, data in various formats, such as structured data and vector data, can be input to the large-scale multimodal model 117a in addition to natural language sentences. For this reason, the analytical accuracy of the large-scale multimodal model 117a can be expected to be higher than that of the large-scale language model 117.

[0102] Ninth Embodiment A ninth embodiment relating to an information processing device, an information processing method, and a recording medium will be described with reference to Fig. 3 and Fig. 10. Hereinafter, the ninth embodiment relating to an information processing device, an information processing method, and a recording medium will be described using an information processing device 100. Note that, with regard to the ninth embodiment, descriptions that overlap with the first to eighth embodiments will be omitted as appropriate.

[0103] 10 , the calculation device 110 may have, as logically realized functional blocks or physically realized processing circuits, an event extraction unit 115, an acquisition unit 116, a large-scale language model 117, a statistical analysis unit 118, and a text generation unit 119. Note that at least one of the event extraction unit 115, the acquisition unit 116, the large-scale language model 117, the statistical analysis unit 118, and the text generation unit 119 may be realized in a form in which logical functional blocks and physical processing circuits (i.e., hardware) are mixed.

[0104] When the event extraction unit 115, the acquisition unit 116, the large-scale language model 117, the statistical analysis unit 118, and the text generation unit 119 are realized as functional blocks, the event extraction unit 115, the acquisition unit 116, the large-scale language model 117, the statistical analysis unit 118, and the text generation unit 119 may be realized by a single processor. Alternatively, the event extraction unit 115, the acquisition unit 116, the large-scale language model 117, the statistical analysis unit 118, and the text generation unit 119 may be realized by different processors. Alternatively, some of the event extraction unit 115, the acquisition unit 116, the large-scale language model 117, the statistical analysis unit 118, and the text generation unit 119 may be realized by a single processor, and the remaining parts of the event extraction unit 115, the acquisition unit 116, the large-scale language model 117, the statistical analysis unit 118, and the text generation unit 119 may be realized by one or more processors different from the single processor.

[0105] When structure data is extracted by the event extraction unit 115, the acquisition unit 116 may acquire the extracted structure data. When vector data is extracted by the event extraction unit 115, the acquisition unit 116 may acquire structure data related to the event corresponding to the extracted vector data from the storage device 120.

[0106] The statistical analysis unit 118 performs statistical analysis on the multiple pieces of structured data acquired by the acquisition unit 116. The statistical analysis unit 118 may input the results of the statistical analysis to the large-scale language model 117 and the text generation unit 119. The results of the statistical analysis may be expressed, for example, in natural language so that the large-scale language model 117 can process them. Note that the results of the statistical analysis may not be input to the large-scale language model 117. In this case, the results of the statistical analysis may be input only to the text generation unit 119. Alternatively, the results of the statistical analysis may not be input to the text generation unit 119. In this case, the results of the statistical analysis may be input only to the large-scale language model 117.

[0107] The text generation unit 119 may generate, for example, a natural language text that indicates a trend indicated by the result of the statistical analysis processing by performing natural language processing on the result of the statistical analysis processing by the statistical analysis unit 118. The text generation unit 119 inputs the generated natural language text into the large-scale language model 117.

[0108] The large-scale language model 117 receives as input at least one of the results of the statistical analysis processing by the statistical analysis unit 118 and a natural language sentence generated by the sentence generation unit 119, as well as instruction sentences written in natural language input by the operator of the information processing device 100.

[0109] The large-scale language model 117 outputs an analysis result based on the result of the statistical analysis process by the statistical analysis unit 118, and at least one of the natural language sentences generated by the sentence generation unit 119 and the instruction sentence. For example, the calculation device 110 may control the output device 150 to display the analysis result output from the large-scale language model 117.

[0110] For example, it is assumed that the information processing device 100 is operated by a park management company. For example, it is assumed that the observation information is an image captured by a surveillance camera installed in the park. It is assumed that the storage device 120 stores structure data and vector data relating to an event shown in the image captured by the surveillance camera. For example, it is assumed that the park management company receives a report that "a pervert has been seen."

[0111] In such a situation, an operator of the information processing device 100 (e.g., an employee of a park management company) may input rules and example events into the information processing device 100 via the input device 140, thereby causing the event extraction unit 115 to extract one or more events indicating human behavior at a specified date and time. The acquisition unit 116 may acquire multiple pieces of structured data related to the one or more events extracted by the event extraction unit 115. The statistical analysis unit 118 may perform statistical analysis processing on the multiple pieces of structured data acquired by the acquisition unit 116. Here, the statistical analysis unit 118 may output, as a result of the statistical analysis processing, numerical data indicating, for example, the frequency of human presence for each time period and each location in the park. The text generation unit 119 may generate a natural language text indicating a trend indicated by the result of the statistical analysis processing.

[0112] The operator may further input an instruction to the information processing device 100 via the input device 140, such as "Please extract people suspected of being perverts from the series of actions of each individual." The large-scale language model 117 outputs an analysis result based on the instruction and at least one of the results of the statistical analysis process by the statistical analysis unit 118 and the natural language sentence generated by the sentence generation unit 119. The large-scale language model 117 may output, as the analysis result, a sentence such as "At XX month, XX day, XX hour, XX minute, there was a person suspected of being a pervert at XXX in the park."

[0113] The calculation device 110 may have a large-scale multimodal model 117a instead of the large-scale language model 117. In this case, the calculation device 110 does not need to have the text generation unit 119. Furthermore, the results of the statistical analysis process by the statistical analysis unit 118 may be expressed in any format, not limited to natural language.

[0114] (Technical Effect) According to the information processing device 100 of the ninth embodiment, similar to the first embodiment described above, it is possible to deal with not only anticipated events but also unexpected events that may occur in the complex and constantly changing real world. According to the information processing device 100 of the ninth embodiment, similar to the sixth to eighth embodiments described above, it is possible to automate the task of checking multiple events stored in the storage device 120.

[0115] For example, if a huge number of events are stored in the storage device 120, the number of events extracted by the event extraction unit 115 may also be relatively large. In this case, in the aspects described in the sixth to eighth embodiments, a relatively large amount of data (e.g., at least one of natural language sentences, structure data, vector data, and observation information) is input to the large-scale language model 117 or the large-scale multimodal model 117a. As a result, the processing load on the large-scale language model 117 and the large-scale multimodal model 117a may increase.

[0116] In contrast, in the information processing device 10 according to the ninth embodiment, the statistical analysis unit 118 performs statistical analysis processing, thereby reducing the amount of data input to the large-scale language model 117 (or the large-scale multimodal model 117a). Therefore, according to the ninth embodiment, the processing load on the large-scale language model 117 (and the large-scale multimodal model 117a) can be reduced. In addition, the large-scale language model 117 (and the large-scale multimodal model 117a) can analyze the real world within the token number limit of the large-scale language model 117 (and the large-scale multimodal model 117a). Furthermore, since the results of the statistical analysis processing by the statistical analysis unit 118 are input to the large-scale language model 117, the analysis accuracy of the large-scale language model 117 can be improved compared to, for example, a case where natural language sentences related to events extracted by the event extraction unit 115 are input to the large-scale language model 117.

[0117] Tenth Embodiment A tenth embodiment relating to an information processing device, an information processing method, and a recording medium will be described below. The tenth embodiment relating to an information processing device, an information processing method, and a recording medium will be described below using an information processing device 100. Note that, for the tenth embodiment, descriptions that overlap with the first to ninth embodiments will be omitted as appropriate.

[0118] In the tenth embodiment, the aspect described in the second embodiment or the aspect described in the third embodiment may be applied to the generation of the structure data and vector data stored in the storage device 120. The aspect described in the fourth embodiment or the aspect described in the fifth embodiment may be applied to the extraction of at least one of the structure data and the vector data by the event extraction unit 115. The aspect described in the sixth embodiment, the aspect described in the seventh embodiment, or the aspect described in the ninth embodiment may be applied to the input to the large-scale language model 117. The aspect described in the eighth embodiment or the aspect described in the ninth embodiment may be applied to the input to the large-scale multimodal model 117a.

[0119] In the tenth embodiment, information for identifying a person may be generated when information about a person is included in the observation information (for example, at least one of sensor data, image, video, and audio) acquired by the acquisition unit 111 of the calculation device 110. The information for identifying a person may be stored in the storage device 120 as additional data for at least one of the structure data and the vector data.

[0120] For example, if the sensor data as the observation information includes information indicated by a barcode, two-dimensional code, or IC chip attached to an admission ticket (e.g., an employee ID card), information for identifying a person may be generated based on that information. For example, if an image or video as the observation information includes a person, biometric authentication may be performed using the image or video. Then, information for identifying a person may be generated based on the results of the biometric authentication. Alternatively, information for identifying a person may be generated based on the appearance of the person included in the image or video. For example, if the audio as the observation information includes a human voice, biometric authentication (e.g., voiceprint authentication) may be performed using the voice. Then, information for identifying a person may be generated based on the results of the biometric authentication. Note that examples of biometric authentication include at least one of face authentication, iris authentication, fingerprint authentication, palmprint authentication, voiceprint authentication, and ear acoustic authentication.

[0121] The information for identifying a person is not limited to information that can identify an individual (e.g., name, identification number, etc.), but may also be information that makes it difficult to identify an individual. Examples of information that makes it difficult to identify an individual include at least one of age group, company name, department name, occupation, job position, and type (guest, etc.). When biometric authentication is performed using observation information, the information processing device 100 may perform the biometric authentication, or a device different from the information processing device 100 (e.g., cloud) may perform the biometric authentication.

[0122] In the tenth embodiment, information for identifying a person may be added to at least one of the structure data and the vector data extracted by the event extraction unit 115. Therefore, at least one of the large-scale language model 117 and the large-scale multimodal model 117a, to which information about the event extracted by the event extraction unit 115 (for example, at least one of a natural language sentence, structure data, vector data, and observation information, or the result of statistical analysis processing by the statistical analysis unit 118) is input, may analyze the behavior of each person.

[0123] For example, the observation information may be images captured by multiple surveillance cameras installed at multiple locations in an office building. The storage device 120 stores structure data and vector data related to events indicated by the images captured by each surveillance camera. An operator of the information processing device 100 (e.g., an employee of a management company for the office building) may input rules and example events into the information processing device 100 via the input device 140, thereby causing the event extraction unit 115 to extract events indicating the behavior of a specific person.

[0124] For example, the event extraction unit 115 may extract events such as "a cleaner entered an office building," "a cleaner collected garbage at a garbage collection site," and "a cleaner left an office building" as events indicating the behavior of a specific person. The operator may input an instruction, for example, "Please check the behavior history for any suspicious behavior," to the information processing device 100 via the input device 140. In this case, the large-scale language model 117 or the large-scale multimodal model 117a may output a sentence, for example, "No suspicious behavior was found," as an analysis result based on the events extracted by the event extraction unit 115 and the instruction input by the operator.

[0125] For example, the event extraction unit 115 may extract events such as "a copier serviceman entered an office building," "a copier serviceman performed maintenance on a copier in an office area," "a copier serviceman collected garbage at a garbage collection site," and "a copier serviceman left an office building" as events indicating the behavior of a specific person. The operator may input an instruction, for example, "Please check the behavior history for any suspicious behavior," to the information processing device 100 via the input device 140. In this case, the large-scale language model 117 or the large-scale multimodal model 117a may output a sentence, for example, "It is suspicious that a copier serviceman entered a garbage collection site," as an analysis result based on the events extracted by the event extraction unit 115 and the instruction input by the operator.

[0126] (Technical Effect) As with the first embodiment, the information processing device 100 according to the tenth embodiment can handle not only anticipated events but also unexpected events that may occur in the complex and constantly changing real world. In particular, the information processing device 100 according to the tenth embodiment can detect suspicious behavior from the behavior history of a specific person.

[0127] <Supplementary Notes> A part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0128] (Supplementary Note 1) An information processing device comprising: an acquisition means for acquiring observation information, which is at least one of sensor data, images, videos, and audio acquired by observing the real world; a first generation means for generating first event data in the form of structured data relating to an event indicated by the observation information by structuring the observation information by fitting it to a mathematical model; a second generation means for generating second event data in the form of vector data relating to the event by extracting a feature vector from the observation information; and a storage means for storing the first event data and the second event data.

[0129] (Supplementary Note 2) The information processing device according to Supplementary Note 1, wherein the second generating means generates the second event data by extracting the feature vector from a sentence in a natural language that expresses the one event.

[0130] (Supplementary Note 3) The information processing device according to Supplementary Note 1, wherein the second generating means generates the second event data by extracting a multimodal feature vector from the observation information.

[0131] (Supplementary Note 4) The information processing device according to any one of Supplementary Notes 1 to 3, wherein the storage means stores third event data that is a sentence in a natural language that expresses the one event.

[0132] (Supplementary Note 5) The information processing device according to any one of Supplementary Notes 1 to 4, wherein the storage means stores the observation information.

[0133] (Supplementary Note 6) The information processing device according to any one of Supplementary Notes 1 to 5, wherein the storage means stores a plurality of first event data corresponding to a plurality of events respectively, and a plurality of second event data corresponding to the plurality of events respectively, and the information processing device further comprises extraction means for extracting one or more first event data corresponding to a desired event from the plurality of first event data using rules described in structure data, and extracting one or more second event data similar to the desired event from the plurality of second event data using vector data related to the desired event.

[0134] (Supplementary Note 7) The information processing device according to Supplementary Note 6, wherein the extraction means extracts vector data relating to the desired event from a sentence in a natural language that expresses the desired event.

[0135] (Supplementary Note 8) The information processing device according to Supplementary Note 6, wherein the extraction means extracts vector data relating to the desired event from at least one of sensor data, images, videos, and audio corresponding to the desired event.

[0136] (Supplementary Note 9) The information processing device according to Supplementary Note 6, further comprising an input means for inputting a natural language sentence expressing one or more events corresponding to the extracted one or more first event data and the extracted one or more second event data, and an instruction sentence expressed in natural language, into a large-scale language model.

[0137] (Supplementary Note 10) The information processing device according to Supplementary Note 6, further comprising an input means for inputting at least one of sensor data, image, video, and audio, and an instruction sentence, corresponding to the extracted one or more first event data and the extracted one or more second event data, into a large-scale language multimodal model.

[0138] (Supplementary Note 11) The information processing device according to Supplementary Note 6, further comprising an input means for inputting into a large-scale language model the results of a statistical analysis performed on structured data relating to one or more events corresponding to the extracted one or more first event data and the extracted one or more second event data, and at least one of a natural language sentence representing the results of the statistical analysis and an instruction sentence.

[0139] (Supplementary Note 12) The information processing device according to Supplementary Note 6, further comprising an input means for inputting into a large-scale multimodal model the results of a statistical analysis performed on structured data relating to one or more events corresponding to the extracted one or more first event data and the extracted one or more second event data, and at least one of a natural language sentence expressing the results of the statistical analysis, and an instruction sentence.

[0140] (Supplementary Note 13) An information processing method comprising: acquiring observation information, which is at least one of sensor data, images, video, and audio, obtained by observing the real world; generating first event data in the form of structured data relating to an event indicated by the observation information by structuring the observation information by fitting it to a mathematical model; generating second event data in the form of vector data relating to the event by extracting a feature vector from the observation information; and storing the first event data and the second event data.

[0141] (Supplementary Note 14) A recording medium having recorded thereon a computer program for causing a computer to execute an information processing method, the method comprising: acquiring observation information, which is at least one of sensor data, images, video, and audio obtained by observing the real world; generating first event data in the form of structured data relating to an event indicated by the observation information by structuring the observation information by fitting it to a mathematical model; generating second event data in the form of vector data relating to the event by extracting a feature vector from the observation information; and storing the first event data and the second event data.

[0142] Furthermore, some or all of the configurations described in Supplementary Notes 2 to 12, which are dependent on Supplementary Note 1, may also be dependent on Supplementary Notes 13 and 14 in the same dependent relationship as Supplementary Notes 2 to 12. Furthermore, not limited to Supplementary Notes 1, 13, and 14, some or all of the configurations described as Supplements may be made dependent on various hardware, software, various recording means for recording software, or systems, within the scope of each of the above-mentioned embodiments.

[0143] This disclosure is not limited to the above-described embodiments, but may be modified as appropriate within the scope of the claims and the gist or idea of ​​the invention as can be read from the entire specification, and information processing devices, information processing methods, and recording media that involve such modifications are also included in the technical scope of this disclosure.

[0144] REFERENCE SIGNS LIST 10, 100 Information processing device 11, 111 Acquisition unit 12 First generation unit 13 Second generation unit 14 Storage unit 110 Arithmetic unit 120 Storage device

Claims

1. An information processing device comprising: an acquisition means for acquiring observation information, which is at least one of sensor data, images, video, and audio, acquired by observing the real world; a first generation means for generating first event data in the form of structured data, relating to an event indicated by the observation information; a second generation means for generating second event data in the form of vector data, relating to the event, by extracting a feature vector from the observation information; and a storage means for storing the first event data and the second event data.

2. The information processing device according to claim 1, wherein the first generation means generates the first event data in the form of structured data relating to the one event indicated by the observation information by structuring the observation information by fitting it to a mathematical model.

3. The information processing device according to claim 1, wherein the second generating means generates the second event data by extracting the feature vector from a natural language sentence expressing the one event.

4. The information processing device according to claim 1, wherein the second generating means generates the second event data by extracting a multimodal feature vector from the observation information.

5. The information processing device according to claim 1, wherein said storage means stores third event data which is a sentence in a natural language expressing said one event.

6. The information processing device according to claim 1, wherein the storage means stores the observation information.

7. The information processing device according to claim 2, wherein said storage means stores said observation information.

8. The information processing device according to claim 3, wherein the storage means stores the observation information.

9. An information processing device as described in claim 1, wherein the storage means stores a plurality of first event data corresponding to a plurality of events respectively and a plurality of second event data corresponding to the plurality of events respectively, and the information processing device is provided with extraction means for extracting one or more first event data corresponding to a desired event from the plurality of first event data using rules described in structure data, and extracting one or more second event data similar to the desired event from the plurality of second event data using vector data related to the desired event.

10. The information processing device according to claim 9, wherein said extraction means extracts vector data relating to the desired phenomenon from a sentence in natural language expressing the desired phenomenon.

11. The information processing device according to claim 9, wherein the extraction means extracts vector data relating to the desired event from at least one of sensor data, images, video, and audio corresponding to the desired event.

12. An information processing device according to claim 9, further comprising an input means for inputting a natural language sentence expressing one or more events corresponding to the extracted one or more first event data and the extracted one or more second event data, and an instruction sentence expressed in natural language, into a large-scale language model.

13. An information processing device according to claim 9, further comprising an input means for inputting at least one of sensor data, images, video and audio, and instruction sentences corresponding to the extracted one or more first event data and the extracted one or more second event data, into a large-scale language multimodal model.

14. An information processing device as described in claim 9, comprising an input means for inputting into a large-scale language model the results of a statistical analysis performed on structured data relating to one or more events corresponding to the extracted one or more first event data and the extracted one or more second event data, and at least one of a natural language sentence expressing the results of the statistical analysis and an instruction sentence.

15. An information processing device as described in claim 9, comprising an input means for inputting into a large-scale multimodal model the results of a statistical analysis performed on structured data relating to one or more events corresponding to the extracted one or more first event data and the extracted one or more second event data, and at least one of a natural language sentence expressing the results of the statistical analysis and an instruction sentence.

16. An information processing device as described in claim 2, wherein the storage means stores a plurality of first event data corresponding to a plurality of events respectively, and a plurality of second event data corresponding to the plurality of events respectively, and the information processing device is provided with extraction means for extracting one or more first event data corresponding to a desired event from the plurality of first event data using rules described in structure data, and extracting one or more second event data similar to the desired event from the plurality of second event data using vector data related to the desired event.

17. An information processing device as described in claim 3, wherein the storage means stores a plurality of first event data corresponding to a plurality of events respectively and a plurality of second event data corresponding to the plurality of events respectively, and the information processing device is provided with extraction means for extracting one or more first event data corresponding to a desired event from the plurality of first event data using rules described in structure data, and extracting one or more second event data similar to the desired event from the plurality of second event data using vector data related to the desired event.

18. An information processing device as described in claim 4, wherein the storage means stores a plurality of first event data corresponding to a plurality of events, and a plurality of second event data corresponding to the plurality of events, and the information processing device is provided with extraction means for extracting one or more first event data corresponding to a desired event from the plurality of first event data using rules described in structure data, and extracting one or more second event data similar to the desired event from the plurality of second event data using vector data related to the desired event.

19. An information processing method comprising: acquiring observation information, which is at least one of sensor data, images, video, and audio obtained by observing the real world; generating first event data in the form of structured data relating to an event indicated by the observation information; extracting a feature vector from the observation information to generate second event data in the form of vector data relating to the event; and storing the first event data and the second event data.

20. A recording medium having recorded thereon a computer program for causing a computer to execute an information processing method, which includes: acquiring observation information, which is at least one of sensor data, images, video, and audio obtained by observing the real world; generating first event data in the form of structured data relating to an event indicated by the observation information; generating second event data in the form of vector data relating to the event by extracting a feature vector from the observation information; and storing the first event data and the second event data.

Images