Information processing system, information processing method, and non-transitory computer-readable medium

Abstract

There is provided an information processing system including circuitry configured to receive first time-series data having a first modality, receive second time-series data having a second modality, convert the first time-series data into a first data vector, convert the second time-series data into a second data vector, and synchronize the first data vector with the second data vector and generate a combined data vector based on the synchronization, wherein the first data vector is converted directly from the first time-series data and the second data vector is converted directly from the second time-series data.

Description

INFORMATION PROCESSING SYSTEM, INFORMATION PROCESSING METHOD, AND NON-TRANSITORY COMPUTER-READABLE MEDIUM

[0001] The present disclosure relates to an information processing system, an information processing method, and a non-transitory computer-readable medium.

[0002] Known has been a technology of detecting, from picture data or sound data, a behavior label regarding the behavior of a human, calculating similarity between a vector generated by vectorization from the behavior label and a vector indicating a procedure sentence describing a behavior, and determining, based on the similarity, whether the behavior of the human is unusual (for example, refer to PTL 1).

[0003] [PTL 1] Japanese Laid-open Patent Publication No. 2023-012795

[0004] However, according to such a conventional technology, picture data or sound data is converted into a behavior label of a combination of finite labels, followed by vectorization. Thus, different pieces of picture data are likely to converge on the same behavior label, leading to a deterioration in the accuracy of determination.

[0005] Thus, it is an object of the present disclosure to provide an information processing system, an information processing method, and an information processing program that enable behavior determination of a user using a language vector corresponding to a behavior procedure and a behavior data vector resulting from direct vectorization of behavior data regarding the user.

[0006] According to an aspect of the present disclosure, there is provided an information processing system including circuitry configured to receive first time-series data having a first modality, receive second time-series data having a second modality, convert the first time-series data into a first data vector, convert the second time-series data into a second data vector, and synchronize the first data vector with the second data vector and generate a combined data vector based on the synchronization, wherein the first data vector is converted directly from the first time-series data and the second data vector is converted directly from the second time-series data. According to another aspect of the present disclosure, there is provided an information processing method including receiving first time-series data having a first modality, receiving second time-series data having a second modality, converting the first time-series data into a first data vector, converting the second time-series data into a second data vector, and synchronizing the first data vector with the second data vector and generate a combined data vector based on the synchronization. According to another aspect of the present disclosure, there is provided a non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute an information processing method, the method including receiving first time-series data having a first modality, receiving second time-series data having a second modality, converting the first time-series data into a first data vector, converting the second time-series data into a second data vector, and synchronizing the first data vector with the second data vector and generate a combined data vector based on the synchronization, wherein the first data vector is converted directly from the first time-series data and the second data vector is converted directly from the second time-series data.

[0007] Fig. 1 schematically illustrates an information processing system according to an embodiment of the present disclosure.Fig. 2 illustrates a configuration of the information processing system according to an embodiment of the present disclosure.Fig. 3 illustrates a first configuration of a control unit.Fig. 4 is a flowchart illustrating a vectorization process for a procedure manual.Fig. 5 is a flowchart illustrating a learning process for a behavior data vectorization unit.Fig. 6 schematically illustrates a multi-modal vector coupling method.Fig. 7 schematically illustrates multi-modal user behavior determination.Fig. 8 schematically illustrates document searching using similarity calculation between behavior data and natural language.Fig. 9 schematically illustrates sentence generation using behavior data and a language generation model.Fig. 10 illustrates a display UI displayed on a correct / incorrect determination application.Fig. 11 illustrates a second configuration of the control unit.Fig. 12 illustrates a third configuration of the control unit.Fig. 13 schematically illustrates a behavior recognition AI ecosystem.Fig. 14 schematically illustrates a behavior recognition AI ecosystem.Fig. 15 is a block diagram illustrating a hardware configuration of a computer that achieves the function of an information processing system.

[0008] Embodiments of the present disclosure will be described in detail below based on the drawings. Note that the same constituents are denoted with the same reference signs in the following embodiments and overlapping descriptions will be omitted.

[0009] The present disclosure will be described in the following order. 1. Embodiments 1-1. Overview of Information Processing according to Embodiments 1-2. Overview of Information Processing System according to Embodiments 1-3. Configuration of Information Processing System according to Embodiments 1-4. First Configuration of Control Unit 1-5. Multi-Modal Vector Coupling Method 1-6. System for Multi-Modal User Behavior Determination 1-7. Document Searching using Similarity Calculation between Behavior Data and Natural Language 1-8. Sentence Generation using Behavior Data and Language Generation Model 1-9. UI Displayed on Application 1-10. Second Configuration of Control Unit 1-11. Third Configuration of Control Unit 1-12. Behavior Recognition AI Ecosystem 2. Other Embodiments 3. Information Processing System according to Present Disclosure and Effects thereof 4. Hardware Configuration

[0010] (1. Embodiments) (1-1. Overview of Information Processing according to Embodiments) First, an embodiment of the present disclosure will be described.

[0011] In order to determine a complicated human behavior properly in accordance with the situation, a determination system capable of taking into account context, such as ambient surroundings or situations and the order of behaviors, is desirable.

[0012] However, for human behavior recognition taking context into account, behavior recognition including background circumstances, such as the place of a behavior and reasons for the behavior, is desirable, instead of a system that detects a behavior, such as just "running". For such behavior recognition, may be necessary to process multiple pieces of time-series data, for example, from video, a microphone, and a sensor attached to a human, simultaneously, to determine whether the order of behaviors or the situation is proper.

[0013] Thus, according to an embodiment of the present disclosure, a plurality of pieces of sequential data regarding the behavior of a human input from a plurality of sensors, such as a camera, a microphone, and a motion sensor, is converted into a format, which can be handled equivalently to a sentence (natural language), using a machine learning model while being arranged on a time-series basis, so that a sentence including context and the behavior data regarding the human can be compared.

[0014] That is, according to an embodiment of the present disclosure, for composite information processing including ambient surroundings and the order of behaviors, a human behavior recognition system arranges, on a time-series basis, sequential data given from a plurality of sensors in response to the surroundings and handles the sequential data equivalently to natural language to perform behavior recognition (behavior determination) with comparison between a human behavior and natural language.

[0015] In addition, according to an embodiment of the present disclosure, since behavior data is handled equivalently to natural language, behavior data is input into a machine learning model capable of generating natural language, so that a sentence describing a behavior is generated.

[0016] Thus, according to an embodiment of the present disclosure, without strict definition of the order of behaviors of a human and a correct pattern for behavior recognition, whether a human behaves properly can be determined only with a description of a correct behavior with natural language. In addition, according to an embodiment of the present disclosure, human behavior recognition / determination can be performed with sequential data from a motion sensor as an input different from picture / sound sequential data, and an increase can be made in the number of types of sensors.

[0017] (1-2. Overview of Information Processing System according to Embodiments) First, an overview of an information processing system 100 according to an embodiment of the present disclosure will be given with Fig. 1. Fig. 1 schematically illustrates the information processing system 100 according to an embodiment of the present disclosure.

[0018] As illustrated in Fig. 1, a behavior data vectorization unit 131 includes a sensor signal reception unit 1311 that receives, from a motion sensor or the like, time-series data regarding the behavior of a human, a sound signal reception unit 1312 that receives, from a microphone or the like, time-series data regarding the behavior of the human, and a camera signal reception unit 1313 that receives, from a camera or the like, time-series data regarding the behavior of the human.

[0019] For example, time-series behavior data regarding a user is acquired from a sensor on the body of the user or a sensor installed in a place away from the user. Here, types of sensors that can be used include a motion sensor capable of acquiring posture data regarding a human, a camera for acquiring a picture, a microphone for acquiring a sound, and other sensors.

[0020] As illustrated in Fig. 1, each vectorization unit 1314 included in the behavior data vectorization unit 131 converts the time-series data received by a sensor or the like into a certain-dimensional embedding vector. Here, the embedding vector is a certain-dimensional vector generated, through a machine learning model, from the time-series data received by the sensor or the like. For example, such a vectorization unit 1314 converts, into a certain-dimensional vector, a sequence acquired from a sensor.

[0021] For example, using an encoder for a transformer or long short term memory (LSTM) based on a neural network, the vectorization unit 1314 encodes motion data (signal) received by the motion sensor and then performs conversion into an embedding vector.

[0022] For example, using an encoder for a transformer, a model based on a transformer (e.g., Whisper available for transcription), or LSTM, another vectorization unit 1314 encodes sound data (signal) received by the microphone and then performs conversion into an embedding vector.

[0023] For example, using an encoder for a model based on a transformer or a model based on residual neural networks (ResNet) or convolutional neural network (CNN), the other vectorization unit 1314 encodes image data (signal) received by the camera and then performs conversion into an embedding vector. The model based on a transformer is, for example, a vision transformer (ViT) available for image classification or object detection or contrastive language-image pre training (CLIP) as a pre-training model including a text encoder based on a transformer and enabling image classification. The model based on CNN is, for example, a 3D CNN.

[0024] As illustrated in Fig. 1, a sequence temporal synchronization unit 1315 included in the behavior data vectorization unit 131 temporally synchronizes, with respect to a timestamp or sensor frequency, the respective embedding vectors generated by the vectorization units 1314 and adjusts the widths of the vectors as necessary. For example, the sequence temporal synchronization unit 1315 synchronizes, per predetermined time interval, respective vectors (e.g., a motion data vector, a sound data vector, and an image data vector) converted from pieces of modal time-series data (e.g., motion data, sound data, and image data) such that all the vectors are in temporal synchronization between the modes.

[0025] For example, in a case where the vectors created based on pieces of sensor data are different in frequency / interval, the sequence temporal synchronization unit 1315 performs temporal synchronization based on the frequency at which a sensor acquires data or the timestamp. Here, in a case where the frequency due to vectorization of sensor data is smaller than that of the other sensor data, the sequence temporal synchronization unit 1315 supplies, as processing, a zero vector or an interpolation vector generated by reference to another vector to an insufficient part.

[0026] As illustrated in Fig. 1, a multi-modal vector coupling unit 1316 included in the behavior data vectorization unit 131 couples a plurality of vector sequences in temporal synchronization for a single vector sequence. For example, the multi-modal vector coupling unit 1316 couples the vectors synchronized per predetermined time interval between the modes by the sequence temporal synchronization unit 1315 (e.g., a motion data vector, a sound data vector, and an image data vector) to acquire a single behavior data vector per minute.

[0027] As illustrated in Fig. 1, a behavior procedure manual reading unit 1321 reads text data from a behavior procedure manual. For example, the behavior procedure manual reading unit 1321 reads text data from behavior procedure manual data, for example, using optical character recognition (OCR).

[0028] As illustrated in Fig. 1, a vectorization unit 1322 converts, into a language vector, the text data from the behavior procedure manual. For example, using an encoder for a transformer, a derived model based on a transformer (e.g., bidirectional encoder representations from transformers (BERT) as a deep learning model for natural language processing), or LSTM, the vectorization unit 1322 encodes the text data and performs conversion into a natural vector.

[0029] As above, according to an embodiment of the present disclosure, multi-modal behavior or ambient-surrounding data sensed from a human behavior is converted so as to be handled equivalently to natural language. Then, a plurality of pieces of data different in format is coupled together while being temporally synchronized, and a behavior data vector is finally output. The behavior data vectorization unit 131 is trained such that the output behavior data vector is equivalent to a vector output from a language model handling natural language.

[0030] (1-3. Configuration of Information Processing System according to Embodiments) Next, the configuration of the information processing system 100 according to an embodiment of the present disclosure will be described with Fig. 2. Fig. 2 illustrates a configuration of the information processing system 100 according to an embodiment of the present disclosure.

[0031] As illustrated in Fig. 2, the information processing system 100 includes a communication unit 110, a storage unit 120, a control unit 130, an input unit 140, and an output unit 150. Note that the information processing system 100 may include an input unit that receives various types of operations from a user or the like who operates the information processing system 100 (e.g., a touch panel) and an output unit that outputs various types of information in a display and audio manner (e.g., a liquid crystal display and a speaker). Note that the information processing system 100 serves as a system including an information processing device and a camera, and the system may be single or may include a plurality of such devices.

[0032] The communication unit 110 is achieved, for example, by a network interface card (NIC). The communication unit 110 is connected to a network N (e.g., the Internet) by wire or wirelessly (e.g., near field communication (NFC) or Bluetooth(Registered Trademark)) and performs transmission and reception of information with an external device through the network N.

[0033] The storage unit 120 is achieved, for example, by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disc. As illustrated in Fig. 2, the storage unit 120 includes a procedure manual storage unit 121 and a learning storage unit 122.

[0034] The procedure manual storage unit 121 stores data regarding a behavior procedure manual or the like. Specifically, the procedure manual storage unit 121 stores text data indicating a behavior procedure.

[0035] The learning storage unit 122 stores behavior data and a machine learning model. For example, the behavior data includes at least any of picture data, motion data, and sound data. Specifically, the motion data includes at least any of user's line-of-sight data, user's expression data, and user's posture data. The machine learning model corresponds to a model based on at least one of a convolutional neural network (CNN), a recurrent neural network (RNN), or a transformer. A model based on the RNN is, for example, a long short term memory (LSTM) network.

[0036] For example, a program stored inside the information processing system 100 (e.g., an information processing program according to an aspect of the present disclosure) is executed by a central processing unit (CPU), a micro processing unit (MPU), or the like with a random access memory (RAM) or the like as a working area, so that the control unit 130 is achieved. The control unit 130 is a controller and may be achieved, for example, by an integrated circuit, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Unless otherwise particularly described, hereinafter, the control unit 130 is achieved by a processing circuit. Provided that no contradiction is given, the control unit 130 may be regarded as the processing circuit as appropriate. A specific configuration of the control unit 130 will be described with Fig. 3 and beyond.

[0037] The input unit 140 receives various types of data as inputs. For example, the input unit 140 includes a camera, a microphone, or a sensor. Specifically, the sensor includes various sensors, such as a motion sensor, a line-of-sight sensor, an expression sensor, and a posture sensor. The input unit 140 receives text data or the like as an input through an input device, such as a user interface (UI) or a keyboard, that a user can operate.

[0038] The output unit 150 outputs data to be output from the information processing system 100. The output unit 150 is, for example, a liquid crystal display that is built in the information processing system 100 or is connected to the information processing system 100. The output unit 150 is, for example, a speaker that is built in the information processing system 100 or is connected to the information processing system 100.

[0039] (1-4. First Configuration of Control Unit) Next, a specific configuration of the control unit 130 will be described with Fig. 3. Fig. 3 illustrates a first configuration of the control unit 130.

[0040] As illustrated in Fig. 3, the control unit 130 includes the behavior data vectorization unit 131, a language vectorization unit 132, a model training unit 133, a vector comparison unit 134, a result output unit 135, a language vector reading unit 136, and a language output unit 137, and achieves or performs the function and operation of the following information processing. As described above, the behavior data vectorization unit 131 includes the sensor signal reception unit 1311, the sound signal reception unit 1312, the camera signal reception unit 1313, the vectorization units 1314, the sequence temporal synchronization unit 1315, and the multi-modal vector coupling unit 1316. The language vectorization unit 132 includes the behavior procedure manual reading unit 1321 and the vectorization unit 1322. The vector comparison unit 134 includes a similarity calculation unit 1341. Note that the internal configuration of the control unit 130 is not limited to the configuration illustrated in Fig. 3 and thus may be any configuration, provided that the following information processing can be performed.

[0041] The behavior data vectorization unit 131 acquires a behavior vector corresponding to behavior data regarding a user along a time series. For example, based on a plurality of individual behavior data vectors corresponding to multi-modal behavior data along a time series, the behavior data vectorization unit 131 acquires a behavior data vector along the time series.

[0042] The behavior data vectorization unit 131 synchronizes a plurality of individual behavior data vectors corresponding to multi-modal behavior data along a time series, along the time series and then acquires a behavior data vector, based on the synchronized individual behavior data vectors. The behavior data vectorization unit 131 synchronizes a plurality of individual behavior data vectors corresponding to multi-modal behavior data along a time series, along the time series and then couples the synchronized individual behavior data vectors to acquire a behavior data vector.

[0043] The behavior data vectorization unit 131 synchronizes a plurality of individual behavior data vectors corresponding to multi-modal behavior data along a time series, along the time series with respect to a time stamp or frequency and acquires a behavior data vector, based on the synchronized individual behavior data vectors. The behavior data vectorization unit 131 acquires a behavior data vector converted from behavior data regarding a user along a time series using a language model such that the behavior data vector is allowed to be handled equivalently to a language vector.

[0044] The behavior data vectorization unit 131 synchronizes a plurality of individual behavior data vectors corresponding to multi-modal behavior data along a time series, along the time series. In a case where a point in time is present at which a vector as a synchronous counterpart is absent between the individual behavior data vectors, a zero vector or a complementary vector generated based on the corresponding individual behavior data vector at another point in time is supplied as the synchronous counterpart to acquire a behavior data vector.

[0045] As above, according to an embodiment of the present disclosure, signals input from various sensors are converted into a vector comparable to a vector output from a language model capable of text generation or text reading, enabling comparison / evaluation between a human behavior and a procedure manual.

[0046] The language vectorization unit 132 acquires a language vector corresponding to text data indicating a behavior procedure. For example, using OCR or the like, the language vectorization unit 132 reads text data indicating a behavior procedure from a behavior procedure manual and then encodes the text data using an encoder for a transformer, BERT, or LSTM and performs conversion into a language vector.

[0047] The model training unit 133 causes the machine learning model to learn the correspondence relationship between a language vector corresponding to text data indicating a behavior procedure and a behavior data vector corresponding to behavior data suitable to the behavior procedure. For example, the model training unit 133 causes the machine learning model to learn the correspondence relationship between a language vector corresponding to text data indicating a behavior procedure and a behavior data vector converted from behavior data suitable to the behavior procedure using a language model such that the behavior data vector is allowed to be handled equivalently to the language vector.

[0048] The vector comparison unit 134 determines similarity between the behavior of the user and the text data, based on the language vector and the behavior data vector. For example, the vector comparison unit 134 determines similarity between the behavior of the user and the text data indicating the behavior procedure, using the machine learning model, based on the language vector and the behavior data vector. For example, the vector comparison unit 134 calculates similarity between the language vector and the behavior data vector (e.g., cosine similarity). Then, based on the similarity, the degree of accuracy of behavior of the user is evaluated (determined).

[0049] As above, according to an embodiment of the present disclosure, for example, provided that a procedure manual including documented operations is given, a sentence in the procedure manual and behavior data regarding an operator are compared, so that a complicated behavior can be evaluated.

[0050] The result output unit 135 outputs a behavior determination result according to the similarity (e.g., an evaluation result). For example, the result output unit 135 displays picture data at the timing of high similarity or picture data at the timing of low similarity.

[0051] The result output unit 135 displays motion data at the timing of high similarity or motion data at the timing of low similarity. The result output unit 135 outputs, in an audio manner, sound data at the timing of high similarity or sound data at the timing of low similarity. For example, the result output unit 135 outputs the sensor data received by the sensor signal reception unit 1311, the sound data received by the sound signal reception unit 1312, or the image data received by the camera signal reception unit 1313 through the output unit 150 as a UI.

[0052] The language vector reading unit 136 acquires, from the behavior data vector converted from the behavior data regarding the user such that the behavior data vector is allowed to be handled equivalently to the language vector, text data indicating the behavior of the user. For example, the language vector reading unit 136 decodes, using a decoder for the transformer, the behavior data vector allowed to be handled equivalently to the language vector and performs conversion into the text data indicating the behavior of the user.

[0053] The language output unit 137 outputs the text data indicating the behavior of the user. For example, the language output unit 137 displays the text data indicating the behavior of the user through the output unit 150 as a UI. For example, the language output unit 137 outputs, in an audio manner, the text data indicating the behavior of the user through the output unit 150 as a UI.

[0054] Next, the processing of each constituent included in the information processing system 100 described above will be described in detail along the flows in Figs. 4 and 5. Fig. 4 is a flowchart illustrating a vectorization process for a procedure manual.

[0055] The behavior procedure manual reading unit 1321 reads, from a procedure manual, text data regarding a sentence describing a behavior (Step S11). Subsequently, the behavior procedure manual reading unit 1321 inputs the text data regarding the sentence into the vectorization unit 1322 (Step S12).

[0056] Subsequently, the vectorization unit 1322 vectorizes the text data regarding the sentence and then stores the vectorized sentence into the procedure manual storage unit (DB) 121 (Step S13).

[0057] According to an embodiment of the present disclosure, the behavior data vectorization unit 131 performs a series of processing in which sensor data is received and a single vector sequence is output, and the behavior data vectorization unit 131 is trained such that the vector output from the behavior data vectorization unit 131 is allowed to be handled equivalently to the vector generated from natural language by another language model due to such a procedure as illustrated in Fig. 4. The behavior data vectorization unit 131 may use a machine learning model based on a CNN, a RNN, a transformer, or the like. For example, the behavior data vectorization unit 131 may use a LSTM network as a model based on a RNN. As the language model, a model based on a transformer may be used. For example, as such a model based on a transformer, a model like BERT may be used.

[0058] Next, a flow of learning processing of the behavior data vectorization unit 131 will be described with Fig. 5. Fig. 5 is a flowchart illustrating a learning process for the behavior data vectorization unit 131.

[0059] The model training unit 133 verifies whether or not a not-yet-processed data type is present (Step S21). Subsequently, in a case where the verification indicates that a not-yet-processed data type is present (Step S21: Yes), the model training unit 133 extracts the human behavior data from the learning storage unit (DB) 122 and inputs the human behavior data into the vectorization units 1314 (Step S22).

[0060] Subsequently, the model training unit 133 verifies the type of processed behavior data and then the processing proceeds to Step S21 (Step S23). On the other hand, in a case where the verification indicates that no not-yet-processed data type is present (Step S21: No), the model training unit 133 inputs a plurality of pieces of vectorized behavior data into the multi-modal vector coupling unit 1316 (Step S24).

[0061] Subsequently, the model training unit 133 compares and evaluates the behavior data vector obtained from the behavior data and the language vector obtained from the procedure manual storage unit (DB) 121 to update the behavior data vectorization unit 131 (Step S25). Subsequently, the model training unit 133 outputs the updated behavior data vectorization unit 131 (Step S26).

[0062] (1-5. Multi-Modal Vector Coupling Method) Next, a case where the information processing system 100 performs multi-modal vector coupling will be given. Fig. 6 schematically illustrates a multi-modal vector coupling method.

[0063] In the example, the sequence temporal synchronization unit 1315 temporally synchronizes vectors different in frequency and then the multi-modal vector coupling unit 1316 couples the temporally synchronized vectors vertically for a single vector. According to an embodiment of the present disclosure, for example, respective signals input from different sensors are coupled together while temporally synchronized, so that a composite vector expression is obtained.

[0064] For example, a vectorization unit 1314 encodes the motion data received by the sensor signal reception unit 1311 using an encoder for a transformer or LSTM and then performs conversion into a vector. Another vectorization unit 1314 encodes the sound data received by the sound signal reception unit 1312 using an encoder for a transformer or LSTM and then performs conversion into a vector. The other vectorization unit 1314 encodes the image data received by the camera signal reception unit 1313 using an encoder for CLIP, a ViT, or a 3DCNN and then performs conversion into a vector.

[0065] Then, the sequence temporal synchronization unit 1315 synchronizes these three types of vectors per predetermined time interval. In a case where any of the vectors is absent per time for synchronization, a zero vector is added to the corresponding vector. Thus, all the three types of vectors are in synchronization per predetermined time interval between the modes. The multi-modal vector coupling unit 1316 couples the vectors synchronized per predetermined time interval between the modes to acquire a single behavior data vector per predetermined time interval.

[0066] (1-6. System for Multi-Modal User Behavior Determination) Next, an example in which the information processing system 100 performs behavior determination due to comparison between behavior data and natural language will be given. Fig. 7 schematically illustrates multi-modal user behavior determination.

[0067] In the example, a vector generated from behavior data and a vector generated from natural language are allowed to be handled equivalently. Thus, such human behavior determination due to similarity between the vectors as illustrated in Fig. 7 can be performed, so that the accuracy of operation of a user in training can be determined with natural language and sensors attached around the user.

[0068] For example, according to an embodiment of the present disclosure, similarity is calculated between the behavior data vector acquired by the behavior data vectorization unit 131 including the sensor signal reception unit 1311, the sound signal reception unit 1312, the camera signal reception unit 1313, the vectorization units 1314, the sequence temporal synchronization unit 1315, and the multi-modal vector coupling unit 1316, of which the configuration is similar to the configuration illustrated in Fig. 1, and the language vector acquired by the behavior procedure manual reading unit 1321 and the vectorization unit 1322, of which the configuration is similar to the configuration illustrated in Fig. 1. Then, according to an embodiment of the present disclosure, based on the similarity, a determination result of the degree of accuracy of behavior of the user can be acquired as a behavior determination result.

[0069] According to an embodiment of the present disclosure, as specific applications, such human behavior determination can be used for skill evaluation in medical treatment at a medical site, training, or sport practice, and can be further used for navigation due to comparison between a sensed behavior in cooking and sentences in the recipe and for real-time advice to a user based on the guidance for operating a product.

[0070] According to an embodiment of the present disclosure, furthermore, even in a case where the behavior of a person is proper but an error is present in a sentence, the error in the sentence can be detected due to similarity between vectors. For example, according to an embodiment of the present disclosure, an error in a behavior procedure manual can be detected based on an accurate behavior of a skilled person.

[0071] According to an embodiment of the present disclosure, respective signals input from various sensors are converted into a vector that is allowed to be handled equivalently to a vector output from a language model capable of text generation or text reading, enabling evaluation due to comparison between a human behavior and a procedure manual.

[0072] For example, according to an embodiment of the present disclosure, provided that a procedure manual including documented operations is given, sentences in the procedure manual and behavior data regarding an operator are compared, so that a complicated behavior can be evaluated. According to an embodiment of the present disclosure, due to such comparison and evaluation, whether the operator operates along the procedure manual is determined in real time, enabling application to procedure verification in the guidance for cooking or the guidance for operating a new product.

[0073] (1-7. Document Searching using Similarity Calculation between Behavior Data and Natural Language) Next, an example in which the information processing system 100 performs document searching using similarity calculation between behavior data and natural language will be given. Fig. 8 schematically illustrates document searching using similarity calculation between behavior data and natural language.

[0074] In the example, using similarity calculation similar to that in Fig. 7, the procedure manual storage unit (DB) 121 can be searched for a document corresponding to behavior data. For example, the similarity calculation unit 1341 calculates similarity between the behavior data vector acquired by the behavior data vectorization unit 131 of which the configuration is similar to the configuration illustrated in Fig. 7 and the respective language vectors converted from text data indicating a plurality of behavior procedures read from a plurality of behavior procedure manuals by the language vectorization unit 132 including the behavior procedure manual reading unit 1321 and the vectorization unit 1322, of which the configuration is similar to the configuration illustrated in Fig. 7 (e.g., cosine similarity). Thus, according to an embodiment of the present disclosure, a behavior procedure manual highest in similarity can be found (specified).

[0075] As above, according to an embodiment of the present disclosure, natural language describing the order and detail of behaviors and human behavior data input from a plurality of sensors are equivalently handled. Thus, there is no need to implement a program for evaluating the degree of accuracy for each behavior as in the past for achievement of a human behavior correct / incorrect determination system. Therefore, behavior evaluation taking order into account can be carried out due to only calculation of similarity between natural language and behavior data.

[0076] (1-8. Sentence Generation using Behavior Data and Language Generation Model) Next, an example in which the information processing system 100 performs sentence generation using behavior data and a language generation model will be given. Fig. 9 schematically illustrates sentence generation using behavior data and a language generation model.

[0077] In the example, a vector generated from behavior data is input into a language model capable of generating natural language, so that a sentence describing a human behavior can be generated. For example, according to an embodiment of the present disclosure, an input signal from a sensor is vectorized and then the resultant vector is input into a language model capable of text output, so that the behavior of the user can be described with text.

[0078] For example, using a transformer decoder based on a neural network, the language vector reading unit 136 decodes the behavior data vector acquired by the behavior data vectorization unit 131 including the sensor signal reception unit 1311, the sound signal reception unit 1312, the camera signal reception unit 1313, the vectorization units 1314, the sequence temporal synchronization unit 1315, and the multi-modal vector coupling unit 1316, of which the configuration is similar to the configuration illustrated in Fig. 7 and then performs conversion into text data of a sentence describing the behavior of the user. Then, the language output unit 137 displays the text data of the sentence describing the behavior of the user through the output unit 150 as a UI.

[0079] Specifically, according to an embodiment of the present disclosure, based on camera pictures and human posture data in a sports broadcast game, the play-by-play commentary of the game can be generated as language. According to an embodiment of the present disclosure, from human skeleton data or video data obtained from sports live pictures, a live play-by-play commentary can be automatically generated.

[0080] (1-9. UI Displayed on Application) Next, a UI displayed on an application using an embodiment of the present disclosure will be described. Fig. 10 illustrates a display UI displayed on a correct / incorrect determination application.

[0081] In the example, a display UI is displayed on a behavior correct / incorrect determination application using similarity determination between a sentence and a behavior and sentence generation from the behavior based on video and motion data.

[0082] (1-10. Second Configuration of Control Unit) Next, an example in which the information processing system 100 evaluates blood collection of a nurse will be given. Fig. 11 illustrates a second configuration of the control unit 130.

[0083] With the second configuration, for example, sensor data regarding the behavior of a user as an input is compared with sentences describing the procedure of blood collection to evaluate the blood collection of a nurse as the behavior of a user. Note that, in a case where the posture of a syringe (e.g., an insertion angle) is desirable in the procedure of evaluation of blood collection, as behavior data, the posture of the syringe may be additionally sensed and used as an input.

[0084] For example, the vector comparison unit 134 calculates cosine similarity (= 95%) between the behavior data vector (= A) acquired from motion data, sound data, image data, and object posture data regarding the syringe by the behavior data vectorization unit 131 of which the configuration is similar to the configuration illustrated in Fig. 3 and the language vector (= B) converted from text data read from a behavior procedure manual indicating "1. wipe an arm with an alcohol pad, 2. insert a hypodermic needle into the arm, 3. replace the blood collection tube with another tube" by the language vectorization unit 132 of which the configuration is similar to the configuration illustrated in Fig. 3. Thus, according to an embodiment of the present disclosure, based on the cosine similarity, the blood collection of the nurse can be evaluated.

[0085] (1-11. Third Configuration of Control Unit) Next, an example in which the information processing system 100 performs behavior recognition taking a situation into account will be given. Fig. 12 illustrates a third configuration of the control unit 130.

[0086] With the third configuration, from behavior data of the motion of a sports player, a sports live broadcast caption can be automatically generated. For example, according to an embodiment of the present disclosure, motion data, sound data, and image data regarding a player in a soccer game are acquired using a multi-modal behavior encoder. Then, according to an embodiment of the present disclosure, using a transformer encoder, the motion data, the sound data, and the image data are converted into vectors. Then, according to an embodiment of the present disclosure, the vectors are coupled in synchronization between the modes to acquire a behavior data vector. Then, according to an embodiment of the present disclosure, the behavior data vector can be handled equivalently to a language vector. Thus, the behavior data vector is input into a transformer decoder to output text data, so that a play-by-play commentary caption indicating a behavior can be automatically generated. That is, the transformer decoder serves as a decoder that generates text data. For example, the transformer decoder receives, as inputs, the language vector generated by a text encoder and the behavior data vector generated by the transformer encoder with the behavior data vector as an input to generate play-by-play commentary text data. Here, Prompt represents an instruction to a language generation model.

[0087] (1-12. Behavior Recognition AI Ecosystem) Next, a behavior recognition AI ecosystem using an embodiment of the present disclosure will be described. Figs. 13 and 14 schematically illustrate a behavior recognition AI ecosystem.

[0088] In the example, the behavior recognition AI ecosystem illustrated in Fig. 13 includes an encoder that extracts a feature from behavior data and a decoder that generates, from the feature, information, pictures, and the like to be presented to a user.

[0089] The encoder requires fine tuning in order to obtain a feature expressing knowledge about a particular domain well. As the decoder, different decoders are desirable depending on display data to be displayed.

[0090] For example, as illustrated in Fig. 14, according to an embodiment of the present disclosure, for generation of sentences for play-by-play commentary from soccer pictures, an encoder having a knowledge of soccer due to additional learning and a decoder that generates a description of input behavior data are desirable. For example, according to an embodiment of the present disclosure, behavior data in a soccer game is acquired using a behavior encoder having a knowledge of soccer due to additional learning. Then, according to an embodiment of the present disclosure, position trajectory analysis is conducted to the acquired behavior data and then a description is generated using a behavior decoder for play-by-play commentary to the behavior data. Then, according to an embodiment of the present disclosure, a caption can be generated to a live TV broadcast screen or a virtual game screen based on avatars on a metaverse.

[0091] For example, according to an embodiment of the present disclosure, for control of the behavior of a customer-service avatar to customers, an encoder having a knowledge of merchandise due to additional learning and a decoder that predicts the behavior of a clerk to input behavior data are desirable.

[0092] A processing flow of mapping a feature of behavior data in a language space taking knowledge about a particular domain into account and predicting a description of the feature and subsequent sentences can be used in various use cases, such as the above-described example.

[0093] For such a processing flow, an encoder and a decoder suitable to each use case require preparing. Because of different teaching data for additional learning per encoder or decoder, a network pre-trained parameter varies per encoder or decoder, but a common execution environment / learning environment can be provided.

[0094] According to an embodiment of the present disclosure, provision of an encoder / decoder execution environment to various service developers who effectively use behavior data and management of a store from which encoders and decoders developed by third parties can be purchased enable service developers to construct target services with a combination of a trained encoder and a trained decoder.

[0095] According to an embodiment of the present disclosure, a development environment is provided to encoder / decoder developers such that additional learning regarding a particular domain is allowed and trained encoders and decoders can be sold to third parties at a store, leading to benefit. An embodiment of the present disclosure can be used in combination with such an ecosystem.

[0096] (2. Other Embodiments) The respective pieces of processing according to the above-described embodiments may be carried out in various modes different from the above-described embodiments.

[0097] From among the pieces of processing described in the embodiments described above, a piece of processing described as to be automatically performed can be manually performed entirely or partially and a piece of processing described as to be manually performed can be automatically performed entirely or partially with a publicly known method. In addition, unless otherwise particularly noted, the processing procedures, specific names, and information including various types of data and parameters in the above descriptions and the drawings can be freely changed. For example, the various types of information illustrated in each drawing are not limited to the illustrated information.

[0098] The constituent elements of each device illustrated are each conceptual in function and thus are not necessarily physically provided as illustrated. That is, the respective specific modes for dispersion / integration of the devices are not limited to the illustrations, and thus each device can be entirely or partially dispersed or integrated functionally or physically in any unit, for example, in accordance with various types of loads or usage situations.

[0099] The above-described embodiments and modifications can be combined as appropriate, provided that no contradiction is given in processing contents.

[0100] The effects described herein are just examples and are not limiting, and thus other effects may be provided.

[0101] (3. Information Processing System according to Present Disclosure and Effects thereof) As described above, an information processing system according to an aspect of the present disclosure includes a behavior data vectorization unit 131, a language vectorization unit 132, a model training unit 133, a vector comparison unit 134, a result output unit 135, a language vector reading unit 136, and a language output unit 137. The information processing system acquires a language vector corresponding to text data indicating a behavior procedure, acquires a behavior data vector corresponding to behavior data regarding a user along a time series, and determines similarity between the behavior of the user and the text data, based on the language vector and the behavior data vector.

[0102] As above, in the information processing system according to an aspect of the present disclosure, respective vectors generated from behavior data and natural language are equivalently handled, enabling human behavior determination due to similarity between the vectors.

[0103] The information processing system acquires, based on a plurality of individual behavior data vectors corresponding to multi-modal behavior data along a time series, a behavior data vector along the time series.

[0104] Specifically, the information processing system synchronizes a plurality of individual behavior data vectors corresponding to multi-modal behavior data along a time series, along the time series and acquires a behavior data vector, based on the synchronized individual behavior data vectors. For example, the information processing system synchronizes a plurality of individual behavior data vectors corresponding to multi-modal behavior data along a time series, along the time series with respect to a time stamp or frequency and acquires a behavior data vector, based on the synchronized individual behavior data vectors.

[0105] For example, the information processing system synchronizes a plurality of individual behavior data vectors corresponding to multi-modal behavior data along a time series, along the time series and couples the synchronized individual behavior data vectors to acquire a behavior data vector. For example, the information processing system synchronizes a plurality of individual behavior data vectors corresponding to multi-modal behavior data along a time series, along the time series. In a case where a point in time is present at which a vector as a synchronous counterpart is absent between the individual behavior data vectors, a zero vector or a complementary vector generated based on the corresponding individual behavior data vector at another point in time is supplied as the synchronous counterpart to acquire a behavior data vector.

[0106] As above, the information processing system couples respective signals input from different signal senders, in temporal synchronization, so that a composite vector expression can be obtained.

[0107] The information processing system outputs a behavior determination result according to similarity.

[0108] As above, for example, in a case where the information processing system is applied to equipment operation training, when new equipment is provided, from a signal of "equipment operation + line of sight + utterance", the information processing system can provide a determination result as to whether a proper operation is being made.

[0109] For example, with text regarding a goal for exercise intensity / exercise and biological signal / motion data as inputs, the information processing system determines whether the exercise is appropriate for the goal to provide a determination result, such as "the exercise intensity may be slightly high", to a user who wants to exercise light.

[0110] For example, in a case where the information processing system is applied to cooking navigation, the information processing system recognizes "food materials + seasoning + quantity + procedure + time" to output a proper instruction. For example, in a case where the information processing system is applied to driving navigation, with "expression with light of sight + voice + acceleration + handling" regarding a driver as an input, the information processing system can support safe driving.

[0111] For example, in a case where the information processing system is applied to guidance for operating a new product, the information processing system verifies whether a proper operation is being made to give time-series advice if a wrong operation is being made.

[0112] Behavior data includes at least picture data, and the information processing system displays the picture data at the timing of high similarity or the picture data at the timing of low similarity. The behavior data further includes sound data, and the information processing system outputs, in an audio manner, the sound data at the timing of high similarity or the sound data at the timing of low similarity.

[0113] Behavior data includes at least motion data, and the information processing system displays the motion data at the timing of high similarity or the motion data at the timing of low similarity.

[0114] The motion data includes at least any of user's line-of-sight data, user's expression data, and user's posture data.

[0115] The information processing system causes a machine learning model to learn the correspondence relationship between a language vector corresponding to text data indicating a behavior procedure and a behavior data vector corresponding to behavior data suitable to the behavior procedure, and determines, using the machine learning model, similarity between the behavior of the user and the text data indicating the behavior procedure, based on the language vector and the behavior data vector.

[0116] For example, the information processing system causes a machine learning model to learn the correspondence relationship between a language vector corresponding to text data indicating a behavior procedure and a behavior data vector converted from behavior data suitable to the behavior procedure using a language model such that the behavior data vector is allowed to be handled equivalently to the language vector. The machine learning model corresponds to a model based on at least one of a CNN, a RNN, or a transformer. A model based on the RNN is, for example, a LSTM network.

[0117] As above, using machine learning, the information processing system can determine the behavior of a human more accurately.

[0118] The information processing system acquires a behavior data vector converted from behavior data regarding a user in a time series using a language model such that the behavior data vector is allowed to be handled equivalently to a language vector and acquires text data indicating the behavior of the user from the behavior data vector converted. The information processing system outputs the text data indicating the behavior of the user.

[0119] As above, the information processing system handles behavior data equivalently to natural language, so that a sentence describing a behavior can be generated.

[0120] (4. Hardware Configuration) An information apparatus, such as the information processing system 100 according to the embodiments described above, is achieved, for example, by a computer 1000 having such a configuration as illustrated in Fig. 15. The following description is given with the information processing system 100 according to the embodiments as an example. Fig. 15 is a hardware configuration diagram illustrating an example of the computer 1000 that achieves the function of the information processing system 100. The computer 1000 includes a CPU 1100, a RAM 1200, a read only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. The constituents in the computer 1000 are connected through a bus 1050.

[0121] The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400 to control each constituent. For example, the CPU 1100 loads a program stored in the ROM 1300 or the HDD 1400 onto the RAM 1200 and then performs processing corresponding to the type of the program.

[0122] The ROM 1300 stores a boot program, such as basic input output system (BIOS), that the CPU 1100 executes in response to start-up of the computer 1000, and programs dependent on the hardware of the computer 1000.

[0123] The HDD 1400 serves as a computer-readable recording medium that records, in a non-transitory manner, a program to be executed by the CPU 1100 and data to be used by the program. Specifically, the HDD 1400 serves as a recording medium that records a conversion program according to an aspect of the present disclosure as an example of program data 1450.

[0124] The communication interface 1500 serves as an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from a different apparatus or transmits data generated by the CPU 1100 to a different apparatus through the communication interface 1500.

[0125] The input / output interface 1600 serves as an interface for connecting an input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device, such as a keyboard or a mouse, through the input / output interface 1600. The CPU 1100 transmits data to an output device, such as a display, a speaker, or a printer, through the input / output interface 1600. The input / output interface 1600 may function as a media interface that reads, for example, a program recorded on a predetermined recording medium. Examples of the medium include optical recording media such as a digital versatile disc (DVD) and a phase change rewritable disk (PD), magneto-optical recording media such as a magneto-optical disk (MO), tape media, magnetic recording media, and semiconductor memories.

[0126] For example, in a case where the computer 1000 functions as the information processing system 100 according to an embodiment of the present disclosure, the CPU 1100 of the computer 1000 executes an information processing program loaded on the RAM 1200 to achieve, for example, the function of the control unit 130. The HDD 1400 stores the information processing program according to an aspect of the present disclosure and the data in the storage unit 120. Note that the CPU 1100 reads the program data 1450 from the HDD 1400 and then executes the corresponding program, but may acquire such a program from a different device through the external network 1550 as another example.

[0127] Note that the present disclosure can have the following configurations. (1) 　　An information processing system including: 　　circuitry configured to: 　　receive first time-series data having a first modality; 　　receive second time-series data having a second modality; 　　convert the first time-series data into a first data vector; 　　convert the second time-series data into a second data vector; and 　　synchronize the first data vector with the second data vector and generate a combined data vector based on the synchronization, 　　wherein the first data vector is converted directly from the first time-series data and the second data vector is converted directly from the second time-series data. (2) 　　The information processing system according to (1), wherein 　　the circuitry is further configured to synchronize the first data vector with the second data vector based on time stamps of the first time-series data and the second time-series data. (3) 　　The information processing system according to any one of (1) or (2), wherein 　　the circuitry is further configured to synchronize the first data vector with the second data vector based on frequencies at which the first time-series data and the second time-series data are acquired. (4) 　　The information processing system according to any one of (1) to (3), wherein 　　the conversion of the first time-series data and the second time-series data includes encoding the first time-series data and the second time-series data using an encoder for a model based on a residual neural network (ResNet) or a convolutional neural network (CNN). (5) 　　The information processing system according to any one of (1) to (4), wherein 　　the circuitry is further configured to adjust widths of the first data vector or the second data vector to synchronize the first data vector with the data second vector. (6) 　　The information processing system according to any one of (1) to (5), wherein 　　in a case a frequency of one of the first time-series data or the second time-series data is smaller than another of the first time-series data or the second time-series data, the circuitry is further configured to supply a zero vector or an interpolation vector to an insufficient part of the one of the first time-series data or the second time-series data. (7) 　　The information processing system according to any one of (1) to (6), wherein 　　the synchronization includes temporally synchronizing the first data vector and the second data vector. (8) 　　The information processing system according to any one of (1) to (7), wherein 　　after the temporal synchronization, in a case a frequency of one of the first time-series data or the second time-series data is smaller than another of the first time-series data or the second time-series data, the circuitry is further configured to supply a zero vector or an interpolation vector to an insufficient part of the one of the first time-series data or the second time-series data. (9) 　　The information processing system according to any one of (1) to (8), wherein 　　the circuitry is further configured to convert text data into a language vector. (10) 　　The information processing system according to any one of (1) to (9), wherein 　　the circuitry is further configured to compare the language vector and the combined data vector to calculate a similarity between the language vector and the combined data vector. (11) 　　The information processing system according to any one of (1) to (10), wherein 　　the calculated similarity is a cosine similarity. (12) 　　The information processing system according to any one of (1) to (11), wherein 　　the circuitry is further configured to control a displaying of a user interface including the calculated similarity between the language vector and the combined data vector and a video corresponding to the first time-series data and the second time-series data. (13) 　　The information processing system according to any one of (1) to (12), wherein 　　the first modality and the second modality include at least one of motion, sound, video, or object pressure. (14) 　　The information processing system according to any one of (1) to (13), wherein 　　the circuitry is further configured to decode the combined data vector into text data. (15) 　　The information processing system according to any one of (1) to (14), wherein 　　the first time-series data and the second time-series data are collected during a type of an event, and 　　the combined data vector is decoded into text data based on behavior data acquired using a behavior encoder having knowledge of the type of the event. (16) 　　The information processing system according to any one of (1) to (15), wherein 　　the circuitry is further configured to encode an instruction input into a language generation model to generate a language vector and generate commentary text data based on the combined data vector and the language vector. (17) 　　The information processing system according to any one of (1) to (16), wherein 　　the circuitry is further configured to control a displaying of a user interface including the generated commentary and a video of the event corresponding to the first time-series data and the second time-series data. (18) 　　The information processing system according to any one of (1) to (17), wherein 　　the first time-series data and the second time-series data are collected during an event, and the circuitry is further configured to control a displaying of the generated commentary with a broadcast of the event or a virtual representation of the event. (19) 　　An information processing method including: 　　receiving first time-series data having a first modality; 　　receiving second time-series data having a second modality; 　　converting the first time-series data into a first data vector; 　　converting the second time-series data into a second data vector; and 　　synchronizing the first data vector with the second data vector and generate a combined data vector based on the synchronization, 　　wherein the first data vector is converted directly from the first time-series data and the second data vector is converted directly from the second time-series data. (20) 　　A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute an information processing method, the method including: 　　receiving first time-series data having a first modality; 　　receiving second time-series data having a second modality; 　　converting the first time-series data into a first data vector; 　　converting the second time-series data into a second data vector; and 　　synchronizing the first data vector with the second data vector and generate a combined data vector based on the synchronization, 　　wherein the first data vector is converted directly from the first time-series data and the second data vector is converted directly from the second time-series data. (21) 　　An information processing system comprising a processing circuit configured to: 　　acquire a language vector corresponding to text data indicating a behavior procedure; 　　acquire a behavior data vector corresponding to behavior data regarding a user along a time series; and 　　determine similarity between a behavior of the user and the text data, based on the language vector and the behavior data vector. (22) 　　The information processing system according to (21), wherein 　　the processing circuit acquires the behavior data vector along the time series, based on a plurality of individual behavior data vectors corresponding to the behavior data that is multi-modal along the time series. (23) 　　The information processing system according to (22), wherein 　　the processing circuit synchronizes, along the time series, the plurality of individual behavior data vectors corresponding to the behavior data that is multi-modal along the time series and acquires the behavior data vector, based on the plurality of individual behavior data vectors in synchronization. (24) 　　The information processing system according to (23), wherein 　　the processing circuit synchronizes, along the time series, the plurality of individual behavior data vectors corresponding to the behavior data that is multi-modal along the time series and couples the plurality of individual behavior data vectors in synchronization to acquire the behavior data vector. (25) 　　The information processing system according to any one of (21) to (24), wherein 　　the processing circuit outputs a behavior determination result according to the similarity. (26) 　　The information processing system according to any one of (21) to (24), wherein 　　the behavior data includes at least picture data, and 　　the processing circuit displays the picture data at timing at which the similarity is high or the picture data at timing at which the similarity is low. (27) 　　The information processing system according to any one of (21) to (24), wherein 　　the behavior data includes at least motion data, and 　　the processing circuit displays the motion data at timing at which the similarity is high or the motion data at timing at which the similarity is low. (28) 　　The information processing system according to (27), wherein 　　the motion data includes at least any of line-of-sight data regarding the user, expression data regarding the user, and posture data. (29) 　　The information processing system according to any one of (26) to (28), wherein 　　the behavior data further includes sound data, and 　　the processing circuit outputs, in an audio manner, the sound data at timing at which the similarity is high or the sound data at timing at which the similarity is low. (30) 　　The information processing system according to any one of (26) to (29), wherein 　　the processing circuit 　　causes a machine learning model to learn a correspondence relationship between the language vector corresponding to the text data indicating the behavior procedure and the behavior data vector corresponding to the behavior data suitable to the behavior procedure, and 　　determines, using the machine learning model, the similarity between the behavior of the user and the text data indicating the behavior procedure, based on the language vector and the behavior data vector. (31) 　　The information processing system according to (30), wherein 　　the processing circuit causes the machine learning model to learn the correspondence relationship between the language vector corresponding to the text data indicating the behavior procedure and the behavior data vector converted from the behavior data suitable to the behavior procedure using a language model such that the behavior data vector is allowed to be handled equivalently to the language vector. (32) 　　The information processing system according to (30) or (31), wherein 　　the machine learning model is based on at least one of a convolutional neural network, a recurrent neural network, or a transformer. (33) 　　The information processing system according to (23), wherein 　　the processing circuit synchronizes, along the time series, the plurality of individual behavior data vectors corresponding to the behavior data that is multi-modal along the time series, with respect to a time stamp or frequency, and acquires the behavior data vector, based on the plurality of individual behavior data vectors in synchronization. (34) 　　The information processing system according to (21), wherein 　　the processing circuit 　　acquires the behavior data vector converted from the behavior data regarding the user along the time series using a language model such that the behavior data vector is allowed to be handled equivalently to the language vector, and 　　acquires text data indicating the behavior of the user from the behavior data vector converted. (35) 　　The information processing system according to (34), wherein 　　the processing circuit outputs the text data indicating the behavior of the user. (36) 　　The information processing system according to (24), wherein 　　the processing circuit synchronizes, along the time series, the plurality of individual behavior data vectors corresponding to the behavior data that is multi-modal along the time series, and supplies, in a case where a point in time is present at which a vector as a synchronous counterpart is absent between the plurality of individual behavior data vectors, a zero vector or a complementary vector generated based on the corresponding individual behavior data vector at another point in time as the synchronous counterpart to acquire the behavior data vector. (37) 　　An information processing method for causing a processing circuit to perform processing comprising: 　　acquiring a language vector corresponding to text data indicating a behavior procedure; 　　acquiring a behavior data vector corresponding to behavior data regarding a user along a time series; and determining similarity between a behavior of the user and the text data, based on the language vector and the behavior data vector. (38) 　　An information processing program for causing a computer to perform a method comprising: 　　acquiring a language vector corresponding to text data indicating a behavior procedure; 　　acquiring a behavior data vector corresponding to behavior data regarding a user along a time series; and 　　determining similarity between a behavior of the user and the text data, based on the language vector and the behavior data vector.

[0128] 100 Information processing system 110 Communication unit 120 Storage unit 121 Procedure manual storage unit 122 Learning storage unit 130 Control unit 131 Behavior data vectorization unit 132 Language vectorization unit 133 Model training unit 134 Vector comparison unit 135 Result output unit 136 Language vector reading unit 137 Language output unit

Claims

1. An information processing system comprising: 　　circuitry configured to: 　　receive first time-series data having a first modality; 　　receive second time-series data having a second modality; 　　convert the first time-series data into a first data vector; 　　convert the second time-series data into a second data vector; and 　　synchronize the first data vector with the second data vector and generate a combined data vector based on the synchronization, 　　wherein the first data vector is converted directly from the first time-series data and the second data vector is converted directly from the second time-series data.

2. The information processing system according to claim 1, wherein 　　the circuitry is further configured to synchronize the first data vector with the second data vector based on time stamps of the first time-series data and the second time-series data.

3. The information processing system according to claim 1, wherein 　　the circuitry is further configured to synchronize the first data vector with the second data vector based on frequencies at which the first time-series data and the second time-series data are acquired.

4. The information processing system according to claim 1, wherein 　　the conversion of the first time-series data and the second time-series data includes encoding the first time-series data and the second time-series data using an encoder for a model based on a residual neural network (ResNet) or a convolutional neural network (CNN).

5. The information processing system according to claim 1, wherein 　　the circuitry is further configured to adjust widths of the first data vector or the second data vector to synchronize the first data vector with the data second vector.

6. The information processing system according to claim 1, wherein 　　in a case a frequency of one of the first time-series data or the second time-series data is smaller than another of the first time-series data or the second time-series data, the circuitry is further configured to supply a zero vector or an interpolation vector to an insufficient part of the one of the first time-series data or the second time-series data.

7. The information processing system according to claim 1, wherein 　　the synchronization includes temporally synchronizing the first data vector and the second data vector.

8. The information processing system according to claim 7, wherein 　　after the temporal synchronization, in a case a frequency of one of the first time-series data or the second time-series data is smaller than another of the first time-series data or the second time-series data, the circuitry is further configured to supply a zero vector or an interpolation vector to an insufficient part of the one of the first time-series data or the second time-series data.

9. The information processing system according to claim 1, wherein 　　the circuitry is further configured to convert text data into a language vector.

10. The information processing system according to claim 9, wherein 　　the circuitry is further configured to compare the language vector and the combined data vector to calculate a similarity between the language vector and the combined data vector.

11. The information processing system according to claim 10, wherein 　　the calculated similarity is a cosine similarity.

12. The information processing system according to claim 10, wherein 　　the circuitry is further configured to control a displaying of a user interface including the calculated similarity between the language vector and the combined data vector and a video corresponding to the first time-series data and the second time-series data.

13. The information processing system according to claim 1, wherein 　　the first modality and the second modality include at least one of motion, sound, video, or object pressure.

14. The information processing system according to claim 1, wherein 　　the circuitry is further configured to decode the combined data vector into text data.

15. The information processing system according to claim 14, wherein 　　the first time-series data and the second time-series data are collected during a type of an event, and 　　the combined data vector is decoded into text data based on behavior data acquired using a behavior encoder having knowledge of the type of the event.

16. The information processing system according to claim 14, wherein 　　the circuitry is further configured to encode an instruction input into a language generation model to generate a language vector and generate commentary text data based on the combined data vector and the language vector.

17. The information processing system according to claim 16, wherein 　　the circuitry is further configured to control a displaying of a user interface including the generated commentary and a video of the event corresponding to the first time-series data and the second time-series data.

18. The information processing system according to claim 16,wherein 　　the first time-series data and the second time-series data are collected during an event, and the circuitry is further configured to control a displaying of the generated commentary with a broadcast of the event or a virtual representation of the event.

19. An information processing method comprising: 　　receiving first time-series data having a first modality; 　　receiving second time-series data having a second modality; 　　converting the first time-series data into a first data vector; 　　converting the second time-series data into a second data vector; and 　　synchronizing the first data vector with the second data vector and generate a combined data vector based on the synchronization, 　　wherein the first data vector is converted directly from the first time-series data and the second data vector is converted directly from the second time-series data.

20. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute an information processing method, the method comprising: 　　receiving first time-series data having a first modality; 　　receiving second time-series data having a second modality; 　　converting the first time-series data into a first data vector; 　　converting the second time-series data into a second data vector; and 　　synchronizing the first data vector with the second data vector and generate a combined data vector based on the synchronization, 　　wherein the first data vector is converted directly from the first time-series data and the second data vector is converted directly from the second time-series data.

Images