Information processing system, information processing method, and information processing program

Abstract

To provide an information processing system, an information processing method, and an information processing program capable of determining a user's action by using a language vector corresponding to an action procedure and an action data vector obtained by directly vectorizing user's action data.SOLUTION: An information processing system according to the present disclosure acquires a language vector corresponding to text data indicating an action procedure, acquires an action data vector corresponding to action data of a user along a time series, and determines a similarity between an action of the user and the text data on the basis of the language vector and the action data vector.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] A technology is known that detects action labels relating to human actions from video data or audio data, calculates the similarity between a vector generated from the action label and a vector representing a procedural statement that describes the action through vectorization, and determines whether the human action is abnormal based on the similarity (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-012795 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional technologies, video or audio data is converted into action labels, which are a finite combination of labels, and then vectorized. This means that different video data may converge to the same action label, leading to a decrease in judgment accuracy.

[0005] Therefore, this disclosure proposes an information processing system, an information processing method, and an information processing program that can determine a user's behavior using a language vector corresponding to a behavior procedure and a behavior data vector that is a direct vectorization of the user's behavior data. [Means for solving the problem]

[0006] In order to solve the above problem, an information processing system according to one embodiment of the present disclosure acquires a language vector corresponding to text data indicating behavioral procedures, acquires a behavioral data vector corresponding to a user's behavioral data over time, and determines the similarity between the user's behavior and the text data based on the language vector and the behavioral data vector. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating an overview of an information processing system according to an embodiment. [Figure 2] 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. [Figure 3] FIG. 2 is a diagram illustrating a first configuration example of a control unit. [Figure 4] 1 is a flowchart showing a process for vectorizing a procedure manual. [Figure 5] 10 is a flowchart showing a learning process of the behavior data vectorization unit. [Figure 6] FIG. 1 is a diagram illustrating an overview of a multimodal vector combination method. [Figure 7] FIG. 1 is a diagram illustrating an overview of multimodal user behavior determination. [Figure 8] 1 is a diagram showing an overview of document retrieval using calculation of similarity between behavioral data and natural language. [Figure 9] FIG. 1 is a diagram illustrating an overview of sentence generation using behavioral data and a language generation model. [Figure 10] FIG. 10 is a diagram showing an example of a display UI displayed in a correct / incorrect judgment application. [Figure 11] FIG. 10 is a diagram illustrating a second configuration example of a control unit. [Figure 12] FIG. 10 is a diagram illustrating a third exemplary configuration of a control unit. [Figure 13] This is a diagram showing an overview of the behavior recognition AI ecosystem. [Figure 14] This is a diagram showing an overview of the behavior recognition AI ecosystem. [Figure 15]FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing system. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The present disclosure will be described in the following order: 1. Embodiment 1-1. Overview of information processing according to the embodiment 1-2. Overview of information processing system according to embodiment 1-3. Configuration of information processing system according to embodiment 1-4. First configuration example of the control unit 1-5. Example of how to combine multimodal vectors 1-6. Example of a multimodal user behavior assessment system 1-7. Example of document search using similarity calculation between behavioral data and natural language 1-8. Example of sentence generation using behavioral data and language generation model 1-9. Example of UI displayed in the application 1-10. Second example of the control unit configuration 1-11.Third example of the control unit configuration 1-12. Example of an ecosystem for behavior recognition AI 2. Other embodiments 3. Information Processing System and Effects of the Information Processing System According to the Present Disclosure 4. Hardware Configuration

[0010] (1. Embodiment) (1-1. Overview of Information Processing According to the Embodiment) First, the technology of the present disclosure will be described.

[0011] In order to accurately judge complex human behavior in accordance with the situation, a judgment system that can take into account context such as the surrounding environment, situation, and order of actions is required.

[0012] However, human behavior recognition that takes context into account requires more than a system that simply detects actions such as "running." It also requires behavior recognition that takes into account the background circumstances, such as the location and reason for the action. To achieve this, it is necessary to simultaneously process complex time-series data from video, microphones, sensors attached to people, etc., and determine whether the order and circumstances of the actions are correct.

[0013] Therefore, in the technology disclosed herein, multiple series of data related to human behavior input from multiple sensors such as cameras, microphones, and motion sensors are aligned in chronological order and converted into a format that can be treated equivalently to text (natural language) in a machine learning model, making it possible to compare text including context with human behavior data.

[0014] That is, in the technology disclosed herein, a human behavior recognition system aligns sequential data provided by multiple sensors in time series for the surroundings and treats it equivalent to natural language, thereby comparing human behavior with natural language to recognize (determine) behavior, thereby addressing issues that require complex information processing, such as the surrounding environment and the order of behavior.

[0015] Furthermore, in the technology disclosed herein, behavioral data is treated equivalently to natural language, and the behavioral data is input into a machine learning model that can generate natural language, thereby generating sentences that explain the behavior.

[0016] As a result, the technology disclosed herein does not require strict definitions of human behavior sequences or correct patterns for behavior recognition, and can determine whether a person is behaving correctly simply by describing the correct behavior in natural language. Furthermore, the technology disclosed herein can recognize and determine human behavior using input sequence data other than video and audio, such as from motion sensors, and can increase the number of sensor types as needed.

[0017] (1-2. Overview of Information Processing System According to Embodiment) First, an overview of an information processing system 100 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an overview of an information processing system 100 according to an embodiment.

[0018] As shown in Figure 1, the sensor signal receiving unit 1311 included in the behavioral data vectorization unit 131 receives time series data related to human behavior from a motion sensor or the like, the audio signal receiving unit 1312 receives time series data related to human behavior from a microphone or the like, and the camera signal receiving unit 1313 receives time series data related to human behavior from a camera or the like.

[0019] For example, time-series behavioral data of a user is acquired from a sensor installed on the user's body or at a location remote from the user. The types of sensors used here include motion sensors that can acquire human posture data, cameras for capturing images, and microphones for capturing audio, as well as other sensors.

[0020] 1, the vectorization unit 1314 included in the behavior data vectorization unit 131 converts time-series data received by a sensor or the like into an embedding vector of a certain dimension. Here, the embedding vector is a vector of a certain dimension generated from the time-series data received by a sensor or the like through a machine learning model. For example, the vectorization unit 1314 converts a sequence acquired from a sensor into a vector of a certain dimension.

[0021] For example, the vectorization unit 1314 uses a transformer using a neural network or an encoder constituting a long short term memory (LSTM) to encode the motion data (signal) received by the motion sensor and convert it into an embedding vector.

[0022] For example, the vectorization unit 1314 uses a Transformer or a model based on a Transformer (e.g., Whisper, which can be used for transcription), or an encoder that constitutes an LSTM, to encode the voice data (signal) received by the microphone and convert it into an embedding vector.

[0023] For example, the vectorization unit 1314 encodes image data (signals) received by the camera into embedding vectors using an encoder constituting a Transformer-based model, ResNet (Residual Neural Networks), or CNN (Convolutional Neural Network)-based model. For example, the Transformer-based model is ViT (Vision Transformer), which can be used for image classification and object detection, or CLIP (Contrastive Language-Image Pre-training), which is a pre-training model equipped with a Transformer-based text encoder and can classify images. For example, the CNN-based model is 3D CNN.

[0024] 1, the sequence time synchronization unit 1315 included in the behavior data vectorization unit 131 synchronizes the embedding vectors generated by the vectorization unit 1314 in time based on a timestamp or a sensor frequency, and adjusts the width of the vectors as necessary. For example, the sequence time synchronization unit 1315 synchronizes vectors (e.g., motion data vectors, audio data vectors, and image data vectors) converted from time-series data of each modal (e.g., motion data, audio data, and image data) at predetermined time intervals, thereby aligning all vectors in time across modalities.

[0025] For example, if vectors created based on each sensor data have different frequencies and intervals, the time series synchronization unit 1315 synchronizes them in time based on the frequency and timestamp when the sensor acquires the data. Here, if the frequency when vectorized is smaller than that of other sensor data, the time series synchronization unit 1315 performs processing such as filling in the missing part with an empty vector or a vector generated by interpolating with reference to other vectors.

[0026] 1, the multimodal vector combining unit 1316 included in the behavior data vectorization unit 131 combines multiple time-synchronized vector sequences into one vector sequence. For example, the multimodal vector combining unit 1316 combines vectors (e.g., motion data vectors, audio data vectors, and image data vectors) that have been aligned across modalities at predetermined time intervals by the sequence time synchronization unit 1315, thereby obtaining one behavior data vector per minute.

[0027] 1, the behavior procedure manual reading unit 1321 reads text data from the behavior procedure manual. For example, the behavior procedure manual reading unit 1321 reads text data from the behavior procedure manual data using OCR (Optical Character Recognition) or the like.

[0028] 1, the vectorization unit 1322 converts the text data of the action procedure manual into language vectors. For example, the vectorization unit 1322 encodes the text data and converts it into language vectors using a Transformer or a derived model based on a Transformer (for example, BERT: Bidirectional Encoder Representations from Transformers, which is a deep learning model for natural language processing), or an encoder that constitutes an LSTM.

[0029] In this way, the technology disclosed herein converts multimodal behavior and surrounding environment data sensed from human behavior so that it can be treated equivalently to natural language, combines multiple data of different formats while synchronizing them over time, and finally outputs a behavior data vector.The behavior data vectorization unit 131 is trained so that this output behavior data vector becomes equivalent to a vector output from a language model that handles natural language.

[0030] (1-3. Configuration of Information Processing System According to Embodiment) Next, the configuration of the information processing system 100 according to the embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the information processing system 100 according to the embodiment.

[0031] 2, the information processing system 100 includes a communication unit 110, a storage unit 120, a control unit 130, an input unit 140, and an output unit 150. The information processing system 100 may also include an input unit (e.g., a touch panel) that accepts various operations from a user who operates the information processing system 100, and an output unit (e.g., a liquid crystal display and a speaker) that displays and outputs various information. The information processing system 100 is implemented as a system including an information processing device, a camera, and the like, and may be a single system or may include multiple of these devices.

[0032] The communication unit 110 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is connected to a network N (the Internet, near field communication (NFC), Bluetooth (registered trademark), etc.) by wire or wirelessly, and transmits and receives information to and from external devices, etc. via the network N.

[0033] The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 2 , the storage unit 120 has a procedure manual storage unit 121 and a learning storage unit 122.

[0034] The procedure manual storage unit 121 stores data related to an action procedure manual, etc. Specifically, the procedure manual storage unit 121 stores text data indicating an action procedure.

[0035] The learning memory unit 122 stores behavioral data and a machine learning model. For example, the behavioral data includes at least one of video data, motion data, and audio data. Specifically, the motion data includes at least one of user gaze data, user facial expression data, and posture data. The machine learning model is a model based on at least one of a convolutional neural network (CNN), a recurrent neural network (RNN), or a transformer. For example, the RNN-based model is a long short-term memory (LSTM) network.

[0036] The control unit 130 is realized, for example, by a central processing unit (CPU) or a micro processing unit (MPU) executing a program stored in the information processing system 100 (for example, an information processing program according to the present disclosure) using a random access memory (RAM) or the like as a working area. The control unit 130 is also a controller, and may be realized by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Unless otherwise specified, hereinafter, the control unit 130 is assumed to be realized by a processing circuit. To the extent consistent, the control unit 130 may be appropriately interpreted as a processing circuit. The specific configuration of the control unit 130 will be described in detail with reference to FIG. 3 and subsequent figures.

[0037] The input unit 140 accepts input of various data. For example, the input unit 140 is configured with a camera, a microphone, or a sensor. Specifically, the sensors include various sensors such as a motion sensor, a gaze sensor, a facial expression sensor, and a posture sensor. The input unit 140 also accepts input of text data and the like via an input device such as a user-operable UI (User Interface) or keyboard.

[0038] The output unit 150 outputs data output by the information processing system 100. For example, the output unit 150 is a liquid crystal display or the like that is built into the information processing system 100 or that is connected to the information processing system 100. The output unit 150 is also a speaker or the like that is built into the information processing system 100 or that is connected to the information processing system 100.

[0039] (1-4. First Configuration Example of Control Unit) Next, a specific configuration example of the control unit 130 will be shown with reference to Fig. 3. Fig. 3 is a diagram showing a first configuration example of the control unit 130.

[0040] As shown in FIG. 3 , the control unit 130 includes a behavioral data vectorization unit 131, a language vectorization unit 132, a model learning unit 133, a vector comparison unit 134, a result output unit 135, a language vector reading unit 136, and a language output unit 137, and realizes or executes the information processing functions and actions described below. As described above, the behavioral data vectorization unit 131 includes a sensor signal receiving unit 1311, a voice signal receiving unit 1312, a camera signal receiving unit 1313, a vectorization unit 1314, a sequence time synchronization unit 1315, and a multimodal vector combination unit 1316. The language vectorization unit 132 includes a behavioral procedure manual reading unit 1321 and a vectorization unit 1322. The vector comparison unit 134 includes a similarity calculation unit 1341. The internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 3 , and may be any other configuration that performs the information processing described below.

[0041] The behavior data vectorization unit 131 acquires behavior data vectors corresponding to the user's behavior data along a time series. For example, the behavior data vectorization unit 131 acquires behavior data vectors along a time series based on a plurality of individual behavior data vectors corresponding to the multimodal behavior data along a time series.

[0042] The behavior data vectorization unit 131 synchronizes, along a time series, a plurality of individual behavior data vectors corresponding to multimodal behavior data along a time series, and acquires a behavior data vector based on the synchronized individual behavior data vectors. The behavior data vectorization unit 131 acquires, as a behavior data vector, a vector obtained by synchronizing and combining, along a time series, a plurality of individual behavior data vectors corresponding to multimodal behavior data along a time series.

[0043] The behavior data vectorization unit 131 synchronizes multiple individual behavior data vectors corresponding to time-series multimodal behavior data along a time series based on timestamps or frequencies, and acquires behavior data vectors based on the synchronized individual behavior data vectors. The behavior data vectorization unit 131 acquires behavior data vectors obtained by converting the user's time-series behavior data using a language model so that it can be treated equivalently to a language vector.

[0044] The behavior data vectorization unit 131 synchronizes multiple individual behavior data vectors corresponding to multimodal behavior data along a time series along a time series, and if there is a time point at which there is no vector to be synchronized between the individual behavior data vectors, it obtains the behavior data vector by filling in the empty vector to be synchronized or a complementary vector generated based on the individual behavior data vector at another time point.

[0045] In this way, the technology disclosed herein enables the comparison and evaluation of human behavior and procedure manuals by converting signals input from various sensors into vectors that can be compared with vectors output by a language model capable of text generation and text comprehension.

[0046] The language vectorization unit 132 acquires language vectors corresponding to text data indicating action procedures. For example, the language vectorization unit 132 reads the text data indicating action procedures from an action procedure manual using OCR or the like, and encodes the text data using an encoder constituting a Transformer, BERT, or LSTM to convert it into a language vector.

[0047] The model learning unit 133 causes the machine learning model to learn the correspondence between the language vector corresponding to the text data indicating the behavioral steps and the behavioral data vector corresponding to the behavioral data that matches the behavioral steps. For example, the model learning unit 133 causes the machine learning model to learn the correspondence between the language vector corresponding to the text data indicating the behavioral steps and the behavioral data vector obtained by converting the behavioral data that matches the behavioral steps using the language model so that it can be treated equivalently to the language vector.

[0048] The vector comparison unit 134 determines the similarity between the user's behavior and the text data based on the language vector and the behavior data vector. For example, the vector comparison unit 134 determines the similarity between the user's behavior and the text data indicating the behavioral steps using a machine learning model based on the language vector and the behavior data vector. For example, the vector comparison unit 134 calculates the similarity (e.g., cosine similarity) between the language vector and the behavior data vector. Then, the accuracy of the user's behavior is evaluated (determined) based on the similarity.

[0049] In this way, the technology of the present disclosure can evaluate complex behavior by comparing the text of a procedure manual with the behavioral data of a worker, for example, if there is a procedure manual that documents the work.

[0050] The result output unit 135 outputs the behavior determination result related to the similarity (for example, an evaluation result, etc.). For example, the result output unit 135 displays video data at a timing when the similarity is high or video data at a timing when the similarity is low.

[0051] The result output unit 135 displays the motion data at a timing when the similarity is high or the motion data at a timing when the similarity is low. The result output unit 135 outputs the audio data at a timing when the similarity is high or the audio data at a timing when the similarity is low. For example, the result output unit 135 outputs the sensor data received by the sensor signal receiving unit 1311, the audio data received by the audio signal receiving unit 1312, or the image data received by the camera signal receiving unit 1313 via the output unit 150, which is a UI.

[0052] The language vector reading unit 136 acquires text data indicating the user's behavior from the behavior data vector obtained by converting the user's behavior data so that it can be treated equivalently to the language vector. For example, the language vector reading unit 136 uses a Transformer decoder to decode the behavior data vector that can be treated equivalently to the language vector and convert it into text data indicating the user's behavior.

[0053] The language output unit 137 outputs text data indicating the user's behavior. For example, the language output unit 137 displays the text data indicating the user's behavior via the output unit 150, which is a UI. For example, the language output unit 137 outputs the text data indicating the user's behavior as sound via the output unit 150, which is a UI.

[0054] Next, the processing of each unit constituting the information processing system 100 will be described in detail along the flow with reference to Figures 4 and 5. Figure 4 is a flowchart showing the vectorization process of a procedure manual.

[0055] The action procedure manual reading unit 1321 reads text data of a sentence describing an action from the procedure manual (step S11), and then inputs the text data of the sentence to the vectorization unit 1322 (step S12).

[0056] Next, the vectorization unit 1322 vectorizes the text data of the sentence, and stores the vectorized sentence in the procedure manual storage unit (DB) 121 (step S13).

[0057] In the technology disclosed herein, the behavior data vectorization unit 131 is a unit that performs a series of processes from receiving sensor data to outputting a single vector sequence, and the behavior data vectorization unit 131 is trained so that the vectors output by the behavior data vectorization unit 131 can be treated equivalently to vectors generated from natural language using another language model in a procedure such as that shown in FIG. 4. This behavior data vectorization unit 131 may use a machine learning model such as CNN, RNN, or Transformer. For example, the behavior data vectorization unit 131 may use an LSTM network, which is a model based on RNN. Furthermore, a Transformer-based model may be used as the language model. For example, the Transformer-based model may use a model such as BERT.

[0058] Next, the flow of the learning process of the behavior data vectorization unit 131 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the learning process of the behavior data vectorization unit 131.

[0059] The model learning unit 133 checks whether there is an unprocessed data type (step S21). If the model learning unit 133 checks that there is an unprocessed data type (step S21: Yes), it retrieves human behavior data from the learning memory unit (DB) 122 and inputs the data to the vectorization unit 1314 (step S22).

[0060] Next, the model learning unit 133 checks the type of the processed behavioral data and shifts the process to step S21 (step S23). On the other hand, if the model learning unit 133 checks that there is no unprocessed data type (step S21: No), it inputs the vectorized multiple behavioral data to the multimodal vector combination unit 1316 (step S24).

[0061] Next, the model learning unit 133 compares and evaluates the behavior data vector obtained from the behavior data with the language vector obtained from the procedure manual storage unit (DB) 121, and updates the behavior data vectorization unit 131 (step S25). Next, the model learning unit 133 outputs the updated behavior data vectorization unit 131 (step S26).

[0062] (1-5. Example of how to combine multimodal vectors) Next, an example will be shown in which the information processing system 100 combines multimodal vectors. Fig. 6 is a diagram showing an outline of a method for combining multimodal vectors.

[0063] In this example, vectors with different frequencies are synchronized in time by the sequence time synchronization unit 1315, and the time-synchronized vectors are combined into one vector in the vertical direction by the multimodal vector combination unit 1316. In the technology disclosed herein, signals input from different sensors, etc. are combined while being synchronized in time, thereby obtaining a vector representation with complex characteristics.

[0064] For example, the vectorization unit 1314 uses an encoder constituting a Transformer or LSTM to encode and convert into vectors the motion data received by the sensor signal receiving unit 1311. The vectorization unit 1314 also uses an encoder constituting a Transformer or LSTM to encode and convert into vectors the audio data received by the audio signal receiving unit 1312. The vectorization unit 1314 also uses an encoder constituting a CLIP, ViT, or 3DCNN to encode and convert into vectors the image data received by the camera signal receiving unit 1313.

[0065] The sequence time synchronization unit 1315 synchronizes these three types of vectors at predetermined time intervals, and if any of the vectors does not exist at each synchronized time, it adds an empty vector corresponding to that vector. This results in all three types of vectors being synchronized across modalities at predetermined time intervals. The multimodal vector combination unit 1316 combines the vectors that have been synchronized across modalities at predetermined time intervals to obtain a single behavioral data vector at each predetermined time interval.

[0066] (1-6. Example of a multimodal user behavior determination system) Next, an example will be described in which the information processing system 100 performs behavior determination by comparing behavior data with natural language. Fig. 7 is a diagram showing an overview of multimodal user behavior determination.

[0067] In such an example, vectors generated from behavioral data and natural language can be treated equivalently, making it possible to determine human behavior based on the similarity between vectors as shown in Figure 7, and thereby determining the correctness of user actions in training based on natural language and sensors attached to the user's surroundings.

[0068] For example, the technology disclosed herein calculates a similarity between a behavior data vector acquired by a behavior data vectorization unit 131 including a sensor signal receiving unit 1311, an audio signal receiving unit 1312, a camera signal receiving unit 1313, a vectorization unit 1314, a sequence time synchronization unit 1315, and a multimodal vector combining unit 1316 equivalent to the configuration shown in Fig. 1, and a language vector acquired from a behavior procedure reading unit 1321 and a vectorization unit 1322 equivalent to the configuration shown in Fig. 1. Then, the technology disclosed herein can obtain a judgment result of the accuracy of the user's behavior as a behavior judgment result based on the similarity.

[0069] Furthermore, the technology disclosed herein can be used to assess human behavior in specific applications such as evaluating skills in medical treatments, training, and sports practice, sensing behavior while cooking, and providing navigation by comparing recipe text with behavior, or providing real-time advice to users based on product operation guidance.

[0070] Conversely, with the technology disclosed herein, even if the person's actions are correct but the text contains an error, it is possible to detect the error in the text based on the similarity between vectors. For example, with the technology disclosed herein, it is possible to detect an error in an action procedure manual based on the accurate actions of an expert.

[0071] Furthermore, in the technology disclosed herein, signals input from various sensors are converted into vectors that can be treated equivalently to vectors output by a language model capable of text generation and text comprehension, making it possible to evaluate human behavior by comparing it with a procedure manual.

[0072] For example, with the technology disclosed herein, if there is a procedure manual documenting the work, complex actions can be evaluated by comparing the text of the procedure manual with the worker's behavioral data. Utilizing this, the technology disclosed herein can determine in real time whether the worker is performing the work according to the procedure manual, and can be applied to confirming procedures in operating guidance for cooking or new products.

[0073] (1-7. Example of document search using similarity calculation between behavioral data and natural language) Next, an example will be described in which the information processing system 100 performs document retrieval using similarity calculation between behavioral data and natural language. Fig. 8 is a diagram showing an overview of document retrieval using similarity calculation between behavioral data and natural language.

[0074] In this example, by using a similarity calculation as in Fig. 7, a document corresponding to the behavior data can be searched for in the procedure manual storage unit (DB) 121. For example, the similarity calculation unit 1341 calculates the similarity (e.g., cosine similarity) between a behavior data vector acquired by a behavior data vectorization unit 131 having the same configuration as that shown in Fig. 7 and each language vector converted from text data indicating multiple behavior procedures read from multiple behavior procedure manuals by a language vectorization unit 132 including a behavior procedure manual reading unit 1321 and a vectorization unit 1322 having the same configuration as that shown in Fig. 7. As a result, the technique disclosed herein can search for (identify) a behavior procedure manual with the highest similarity.

[0075] In this way, in the technology disclosed herein, by treating natural language describing the order and details of actions and human action data input from multiple sensors as equivalent, it is not necessary to implement a program to evaluate the accuracy of each action as in the past in order to realize a system for determining the correctness of human actions, and it is possible to evaluate actions including order simply by calculating the similarity between natural language and action data.

[0076] (1-8. Example of sentence generation using behavioral data and language generation model) Next, a description will be given of an example in which the information processing system 100 generates sentences using behavioral data and a language generation model. Fig. 9 is a diagram showing an overview of sentence generation using behavioral data and a language generation model.

[0077] In such an example, a sentence describing a human's behavior can be generated by inputting vectors generated from behavioral data into a language model capable of generating natural language. For example, in the technology disclosed herein, input signals from a sensor are vectorized and input into a language model capable of outputting text, thereby enabling a user's behavior to be described in text.

[0078] For example, the language vector reading unit 136 decodes the behavioral data vector acquired by the behavioral data vectorization unit 131, which includes a sensor signal receiving unit 1311, an audio signal receiving unit 1312, a camera signal receiving unit 1313, a vectorization unit 1314, a sequence time synchronization unit 1315, and a multimodal vector combining unit 1316, which are equivalent to the configuration shown in Fig. 7, using a Transformer decoder that uses a neural network, and converts it into text data of sentences that explain the user's behavior. Then, the language output unit 137 displays the text data of sentences that explain the user's behavior via the output unit 150, which is a UI.

[0079] Specifically, the technology disclosed herein makes it possible to generate live commentary of a sporting event as language based on camera footage and human posture data from the live sports broadcast. The technology disclosed herein makes it possible to automatically generate live commentary from human skeleton data and video data obtained from live sports footage.

[0080] (1-9. Example of UI displayed in application) Next, an example of a UI displayed in an application using this technology will be described. Fig. 10 is a diagram showing an example of a display UI displayed in a correct / incorrect judgment application.

[0081] An example of this is a display UI displayed in an application that determines the similarity between sentences and actions, and determines whether an action is correct or incorrect using sentence generation from actions using video and motion data.

[0082] (1-10. Second Configuration Example of Control Unit) Next, an example will be described in which the information processing system 100 evaluates the blood collection by a nurse. FIG. 11 is a diagram showing a second configuration example of the control unit 130.

[0083] In the second configuration example, sensor data etc. related to the user's behavior is input, and by comparing it with a document describing the blood collection procedure, the nurse's blood collection is evaluated as the user's behavior. Note that if the syringe posture (insertion angle etc.) is required in the blood collection evaluation procedure, the syringe posture may be additionally sensed as behavior data and used as input.

[0084] For example, the vector comparison unit 134 calculates the cosine similarity (=95%) between an action data vector (=A) acquired from motion data, voice data, image data, and object posture data of a syringe by an action data vectorization unit 131 having the same configuration as that shown in Fig. 3, and a language vector (=B) converted from text data read from an action procedure manual indicating "1. Wipe the arm with an alcohol pad, 2. Prick the arm with the syringe needle, 3. Replace the blood collection tube" by a language vectorization unit 132 having the same configuration as that shown in Fig. 3. In this way, the technology disclosed herein can evaluate the nurse's blood collection based on the cosine similarity.

[0085] (1-11. Third Configuration Example of Control Unit) Next, an example will be described in which the information processing system 100 performs situation-aware behavior recognition. FIG. 12 is a diagram showing a third configuration example of the control unit 130.

[0086] In a third configuration example, subtitles for live sports broadcasts can be automatically generated from behavioral data of athletes' movements. For example, the technology disclosed herein uses a multimodal behavior encoder to acquire motion data, audio data, and image data of players in a soccer game. The motion data, audio data, and image data are then converted into vectors using a Transformer encoder. These vectors are then synchronized across modalities to obtain a behavioral data vector. Because the behavioral data vector can be treated equivalently to a language vector, the technology disclosed herein can automatically generate commentary subtitles representing the actions by inputting the behavioral data vector to a Transformer decoder and outputting text data. In other words, the Transformer decoder generates text data. For example, the Transformer decoder generates text data for a live broadcast using the language vector generated by the text encoder and the behavioral data vector generated by the Transformer encoder using the behavioral data vector as input. Here, "Prompt" represents an instruction to a language generation model.

[0087] (1-12. Example of an ecosystem for behavior recognition AI) Next, we will explain an example of an ecosystem for behavior recognition AI using this technology. Figures 13 and 14 are diagrams showing an overview of the ecosystem for behavior recognition AI.

[0088] In this example, the behavior recognition AI ecosystem shown in Figure 13 consists of an encoder that extracts features from behavioral data and a decoder that generates information, images, etc. to be presented to the user from the features.

[0089] The encoder needs to be fine-tuned to obtain features that effectively represent knowledge about a specific domain, and different decoders are required depending on the display data to be displayed.

[0090] For example, as shown in FIG. 14, in the technology disclosed herein, to generate commentary from soccer video footage, an encoder that has additionally learned soccer knowledge and a decoder that generates explanatory text for the input behavioral data are required. For example, in the technology disclosed herein, behavioral data from a soccer game is acquired using a behavioral encoder that has additionally learned soccer knowledge. Then, in the technology disclosed herein, position trajectory analysis is performed on the acquired behavioral data, and explanatory text that serves as commentary for the behavioral data is generated using a behavioral decoder. Then, in the technology disclosed herein, subtitles can be generated for a TV broadcast screen or a simulated game screen using avatars in the metaverse.

[0091] For example, in the technology disclosed herein, if you want to control the behavior of a customer service response avatar toward customers, you need an encoder that has additionally learned product knowledge and a decoder that predicts the behavior of store clerks based on input behavioral data.

[0092] This processing flow, which involves mapping the features of behavioral data into a language space based on knowledge about a specific domain and predicting the description of the features and the following sentences, can be used in a variety of use cases, as in the example above.

[0093] To do this, it is necessary to prepare encoders and decoders that are suitable for each use case. Each encoder and decoder has different training data for additional learning, and therefore different trained network parameters, but the execution environment and learning environment can be the same.

[0094] The technology disclosed herein provides an encoder / decoder execution environment for various service developers who utilize behavioral data, and operates a store where they can purchase encoders / decoders developed by third parties, allowing service developers to build their desired services using a combination of trained encoders / decoders.

[0095] Furthermore, the technology disclosed herein can provide a development environment for developers of encoders and decoders, enable them to perform additional learning for specific domains, and enable them to sell trained encoders and decoders to third parties in a store, thereby enabling them to earn profits. The technology disclosed herein can be used in combination with such an ecosystem.

[0096] (2. Other Embodiments) The processing according to each of the above-described embodiments may be implemented in various different forms other than the above-described embodiments.

[0097] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. Furthermore, the information, including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings, can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0098] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0099] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0100] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0101] (3. Information Processing System and Effects of the Information Processing System According to the Present Disclosure) As described above, the information processing system according to the present disclosure includes the behavioral data vectorization unit 131, the language vectorization unit 132, the model learning unit 133, the vector comparison unit 134, the result output unit 135, the language vector reading unit 136, and the language output unit 137. The information processing system acquires language vectors corresponding to text data indicating behavioral steps, acquires behavioral data vectors corresponding to the user's behavioral data along a time series, and determines the similarity between the user's behavior and the text data based on the language vectors and the behavioral data vectors.

[0102] In this way, the information processing system according to the present disclosure can treat vectors generated from behavioral data and natural language equivalently, thereby enabling human behavior to be determined based on the similarity between vectors.

[0103] The information processing system also acquires behavior data vectors along a time series based on a plurality of individual behavior data vectors corresponding to multimodal behavior data along a time series.

[0104] Specifically, the information processing system synchronizes, along a time series, a plurality of individual behavior data vectors corresponding to multimodal behavior data along a time series, and acquires a behavior data vector based on the synchronized individual behavior data vectors. For example, the information processing system synchronizes, along a time series, a plurality of individual behavior data vectors corresponding to multimodal behavior data along a time series based on timestamps or frequencies, and acquires a behavior data vector based on the synchronized individual behavior data vectors.

[0105] For example, the information processing system acquires a vector obtained by synchronizing and combining multiple individual behavior data vectors corresponding to multimodal behavior data along a time series along with the time series as a behavior data vector. For example, the information processing system synchronizes multiple individual behavior data vectors corresponding to multimodal behavior data along a time series along with the time series, and when there is a time point at which no vector to be synchronized exists among the individual behavior data vectors, the information processing system acquires the behavior data vector by filling the time point at which there is no vector to be synchronized with, an empty vector or a complementary vector generated based on the individual behavior data vector at another time point.

[0106] In this way, the information processing system can obtain a vector representation with complex characteristics by combining signals input from different signal sources while synchronizing them in time.

[0107] The information processing system also outputs a behavior determination result related to the similarity.

[0108] In this way, the information processing system can be applied to equipment operation training, for example, and when a new piece of equipment is introduced, it can provide a judgment result as to whether the operation is being performed correctly based on the signals of ``equipment operation + gaze + speech.''

[0109] For example, an information processing system can input text about exercise intensity and exercise goals, along with biosignal and motion data, determine whether the exercise is appropriate for the goals, and provide a judgment result such as "I feel like the exercise intensity is a little strong" to a user who wants to exercise lightly.

[0110] For example, when applied to cooking navigation, the information processing system can recognize "ingredients + seasonings + amounts + steps + time" and output appropriate instructions. For example, when applied to driving navigation, the information processing system can support safe driving by inputting the driver's "gaze, facial expression + voice + accelerator + handling."

[0111] For example, information processing systems can be applied to operating guidance for new products to check whether operations are being performed correctly and provide real-time advice if they are incorrect.

[0112] The information processing system also includes at least video data in the behavioral data, and displays the video data at a timing with high similarity or the video data at a timing with low similarity. The information processing system also includes audio data in the behavioral data, and outputs the audio data at a timing with high similarity or the audio data at a timing with low similarity.

[0113] The information processing system includes at least motion data in the behavior data, and displays motion data with timings that have high similarity or motion data with timings that have low similarity.

[0114] The motion data includes at least one of user's gaze data, user's facial expression data, and posture data.

[0115] The information processing system also trains a machine learning model to learn the correspondence between language vectors corresponding to text data indicating behavioral procedures and behavioral data vectors corresponding to behavioral data that matches the behavioral procedures, and uses the machine learning model to determine the similarity between the user's behavior and the text data indicating the behavioral procedures based on the language vectors and the behavioral data vectors.

[0116] For example, the information processing system makes a machine learning model learn the correspondence between a language vector corresponding to text data indicating an action procedure and an action data vector obtained by converting action data that matches the action procedure using a language model so that the action data can be treated equivalently to the language vector. The machine learning model is a model based on at least one of CNN, RNN, and Transformer. For example, the RNN-based model is an LSTM network.

[0117] In this way, the information processing system can determine human behavior with greater accuracy by using machine learning.

[0118] The information processing system also acquires behavioral data vectors obtained by converting the user's behavioral data along a time series using a language model so that the data can be treated equivalently to language vectors, and acquires text data indicating the user's behavior from the converted behavioral data vectors. The information processing system outputs the text data indicating the user's behavior.

[0119] In this way, the information processing system can generate sentences that explain the behavior by treating the behavior data equivalent to natural language.

[0120] (4. Hardware Configuration) An information device such as the information processing system 100 according to each of the above-described embodiments is realized by, for example, a computer 1000 configured as shown in FIG. 15. The information processing system 100 according to the embodiment will be described below as an example. FIG. 15 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing system 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, an HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0121] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0122] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0123] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records a conversion program according to the present disclosure, which is an example of program data 1450.

[0124] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0125] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disk), magneto-optical recording media such as an MO (Magneto-Optical disk), tape media, magnetic recording media, and semiconductor memories.

[0126] For example, when the computer 1000 functions as the information processing system 100 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200, thereby realizing functions of the control unit 130 and the like. The information processing program according to the present disclosure and data in the storage unit 120 are stored in the HDD 1400. The CPU 1100 reads and executes program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0127] The present technology can also be configured as follows. (1) An information processing system including a processing circuit, The processing circuitry Acquire a language vector corresponding to text data indicating an action procedure; Obtaining a behavioral data vector corresponding to the user's behavioral data over time; determining a similarity between the user's behavior and the text data based on the language vector and the behavior data vector; An information processing system configured as follows. (2) The processing circuitry acquiring the behavior data vector along the time series based on a plurality of individual behavior data vectors corresponding to the multimodal behavior data along the time series; The information processing system according to (1) above. (3) The processing circuitry a plurality of individual behavior data vectors corresponding to the multimodal behavior data along the time series are synchronized along the time series, and the behavior data vector is acquired based on the synchronized individual behavior data vectors; The information processing system according to (2) above. (4) The processing circuitry a plurality of individual behavior data vectors corresponding to the multimodal behavior data along the time series are synchronized and combined along the time series to obtain the behavior data vector; The information processing system according to (3) above. (5) The processing circuitry outputting a behavior determination result relating to the similarity; The information processing system according to any one of (1) to (4). (6) The behavioral data is Contains at least video data, The processing circuitry displaying the video data at a timing when the degree of similarity is high or the video data at a timing when the degree of similarity is low; The information processing system according to any one of (1) to (4). (7) The behavioral data is including at least motion data, The processing circuitry displaying the motion data at a timing when the degree of similarity is high or the motion data at a timing when the degree of similarity is low; The information processing system according to any one of (1) to (4). (8) The motion data is The data includes at least one of gaze data of the user, facial expression data of the user, and posture data. The information processing system according to (7) above. (9) The behavioral data is Further, the audio data is included, The processing circuitry outputting the audio data at a timing when the degree of similarity is high or the audio data at a timing when the degree of similarity is low; The information processing system according to any one of (6) to (8). (10) The processing circuitry a machine learning model is made to learn a correspondence relationship between the language vector corresponding to the text data indicating the behavioral procedure and the behavioral data vector corresponding to the behavioral data that matches the behavioral procedure; determining the similarity between the user's behavior and the text data indicating the behavioral procedure using the machine learning model based on the language vector and the behavioral data vector; The information processing system according to any one of (6) to (9). (11) The processing circuitry making the machine learning model learn a correspondence relationship between the language vector corresponding to the text data indicating the behavioral procedure and the behavioral data vector obtained by converting the behavioral data conforming to the behavioral procedure using a language model so that the behavioral data can be treated equivalently to the language vector; The information processing system according to (10) above. (12) The machine learning model is The model is based on at least one of a Convolutional Neural Network, a Recurrent Neural Network, or a Transformer. The information processing system according to (10) or (11) above. (13) The processing circuitry a plurality of the individual behavior data vectors corresponding to the multimodal behavior data along the time series are synchronized along the time series based on timestamps or frequencies, and the behavior data vector is acquired based on the synchronized individual behavior data vectors; The information processing system according to (3) above. (14) The processing circuitry The behavioral data vector obtained by converting the behavioral data of the user along the time series using a language model so that it can be treated equivalently to the language vector is obtained; obtaining text data indicating the user's behavior from the converted behavior data vector; The information processing system according to (1) above. (15) The processing circuitry outputting text data indicating the user's behavior; The information processing system according to (14) above. (16) The processing circuitry a plurality of individual behavior data vectors corresponding to the multimodal behavior data along the time series are synchronized along the time series, and when there is a time point at which a vector to be synchronized does not exist among the individual behavior data vectors, the behavior data vector is acquired by filling in the empty vector to be synchronized or a complementary vector generated based on the individual behavior data vector at another time point; The information processing system according to (4) above. (17) An information processing method for causing a processing circuit to execute processing, comprising: The process comprises: Acquire a language vector corresponding to text data indicating an action procedure; Obtaining a behavioral data vector corresponding to the user's behavioral data over time; determining a similarity between the user's behavior and the text data based on the language vector and the behavior data vector; An information processing method consisting of steps. (18) An information processing program, On the computer, Acquire a language vector corresponding to text data indicating an action procedure; Obtaining a behavioral data vector corresponding to the user's behavioral data over time; determining a similarity between the user's behavior and the text data based on the language vector and the behavior data vector; An information processing program that causes the method to be executed. [Explanation of symbols]

[0128] 100 Information Processing Systems 110 Communications Department 120 Storage section 121 Procedure Manual Memory Unit 122 Learning Memory 130 control section 131 Behavioral Data Vectorization Unit 132 Language Vectorization Unit 133 Model Learning Department 134 Vector Comparison Unit 135 Result output section 136 Language Vector Reading Unit 137 Language Output Unit

Claims

1. An information processing system including a processing circuit, The processing circuitry Acquire a language vector corresponding to text data indicating an action procedure; Obtaining a behavioral data vector corresponding to the user's behavioral data over time; determining a similarity between the user's behavior and the text data based on the language vector and the behavior data vector; An information processing system configured as follows.

2. The processing circuitry acquiring the behavior data vector along the time series based on a plurality of individual behavior data vectors corresponding to the multimodal behavior data along the time series; The information processing system according to claim 1 .

3. The processing circuitry a plurality of individual behavior data vectors corresponding to the multimodal behavior data along the time series are synchronized along the time series, and the behavior data vector is acquired based on the synchronized individual behavior data vectors; The information processing system according to claim 2 .

4. The processing circuitry a plurality of individual behavior data vectors corresponding to the multimodal behavior data along the time series are synchronized and combined along the time series to obtain the behavior data vector; The information processing system according to claim 3 .

5. The processing circuitry outputting a behavior determination result relating to the similarity; The information processing system according to claim 1 .

6. The behavioral data is Contains at least video data, The processing circuitry displaying the video data at a timing when the degree of similarity is high or the video data at a timing when the degree of similarity is low; The information processing system according to claim 1 .

7. The behavioral data is including at least motion data, The processing circuitry displaying the motion data at a timing when the degree of similarity is high or the motion data at a timing when the degree of similarity is low; The information processing system according to claim 1 .

8. The motion data is The data includes at least one of gaze data of the user, facial expression data of the user, and posture data. The information processing system according to claim 7 .

9. The behavioral data is Further, the audio data is included, The processing circuitry outputting the audio data at a timing when the degree of similarity is high or the audio data at a timing when the degree of similarity is low; The information processing system according to claim 6.

10. The processing circuitry a machine learning model is made to learn a correspondence relationship between the language vector corresponding to the text data indicating the behavioral procedure and the behavioral data vector corresponding to the behavioral data that matches the behavioral procedure; determining the similarity between the user's behavior and the text data indicating the behavioral procedure using the machine learning model based on the language vector and the behavioral data vector; The information processing system according to claim 1 .

11. The processing circuitry making the machine learning model learn a correspondence relationship between the language vector corresponding to the text data indicating the behavioral procedure and the behavioral data vector obtained by converting the behavioral data conforming to the behavioral procedure using a language model so that the behavioral data can be treated equivalently to the language vector; The information processing system according to claim 10.

12. The machine learning model is A model based on at least one of a Convolutional Neural Network, a Recurrent Neural Network, or a Transformer. The information processing system according to claim 10.

13. The processing circuitry a plurality of the individual behavior data vectors corresponding to the multimodal behavior data along the time series are synchronized along the time series based on timestamps or frequencies, and the behavior data vector is acquired based on the synchronized individual behavior data vectors; The information processing system according to claim 3 .

14. The processing circuitry The behavioral data vector obtained by converting the behavioral data of the user along the time series using a language model so that it can be treated equivalently to the language vector is obtained; obtaining text data indicating the user's behavior from the converted behavior data vector; The information processing system according to claim 1 .

15. The processing circuitry outputting text data indicating the user's behavior; The information processing system according to claim 14.

16. The processing circuitry a plurality of individual behavior data vectors corresponding to the multimodal behavior data along the time series are synchronized along the time series, and when there is a time point at which a vector to be synchronized does not exist among the individual behavior data vectors, the behavior data vector is acquired by filling in the empty vector to be synchronized or a complementary vector generated based on the individual behavior data vector at another time point; The information processing system according to claim 4 .

17. An information processing method for causing a processing circuit to execute processing, comprising: The process comprises: Acquire a language vector corresponding to text data indicating an action procedure; Obtaining a behavioral data vector corresponding to the user's behavioral data over time; determining a similarity between the user's behavior and the text data based on the language vector and the behavior data vector; An information processing method consisting of steps.

18. An information processing program, On the computer, Acquire a language vector corresponding to text data indicating an action procedure; Obtaining a behavioral data vector corresponding to the user's behavioral data over time; determining a similarity between the user's behavior and the text data based on the language vector and the behavior data vector; An information processing program that causes the method to be executed.

Images