Training device, training method, machine learning model, movement determination device, movement determination method, and program

Abstract

The present disclosure provides a technology for efficiently generating a machine learning model for processing video data. An embodiment of the present disclosure relates to a training device comprising: a to-be-trained model processing unit that uses a machine learning model to be trained to generate a prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector of an object from training video data; a spatial teacher model processing unit that uses a spatial teacher model to generate a spatial teacher prediction class vector from image frames of the training video data; a temporal teacher model processing unit that uses a temporal teacher model to generate a temporal teacher prediction class vector from flow frames of the training video data; and a training unit that trains the machine learning model to be trained, on the basis of a first error between a movement label for training corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector.

Description

Training device, training method, machine learning model, action determination device, action determination method, and program

[0001] The present disclosure relates to a training device, a training method, a machine learning model, an action determination device, an action determination method, and a program.

[0002] With the recent advancement of deep learning technology, machine learning models are being used in a wide range of technical fields. For example, research and development of machine learning models is progressing rapidly in image processing and language processing. Under these circumstances, attempts are being made to research and develop machine learning models for video data, i.e., video data, using technologies developed in image processing and language processing. For example, the application of convolutional neural networks, which are effectively used in image processing, and transformer models, which are used in language processing, to machine learning models for video data is being considered.

[0003] For example, as a machine learning model for video data, development is underway on an action recognition model that recognizes human actions, object movements, etc. In a typical action recognition process, video data showing the movement of an object is input to the action recognition model, and the action recognition model outputs the object's action class (e.g., walking, moving, performing abnormal behavior, etc.).

[0004] Patent No. 6875058

[0005] In video data action recognition processing, an approach has been considered in which spatial features for recognizing an object in each image frame and temporal features for recognizing the object's movement across multiple image frames are extracted using a machine learning model, and the object's movement in the video data is determined based on the spatial features and temporal features. For example, such an approach may use a convolutional neural network for extracting spatial features and a convolutional neural network for extracting temporal features. Another approach has also been proposed, which uses a transformer model for processing time-series data, such as language processing, to process video data.

[0006] However, training convolutional neural networks and transformer models is known to require large training datasets, and the training process using these training datasets can require significant computational and time resources.

[0007] In view of the above problems, one objective of the present disclosure is to provide a technique for efficiently generating a machine learning model for processing video data.

[0008] One aspect of the present disclosure relates to a training device comprising: a training object model processing unit that generates an object prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector from training video data using a machine learning model of a training object; a spatial teacher model processing unit that generates a spatial teacher prediction class vector from an image frame of the training video data using a spatial teacher model; a temporal teacher model processing unit that generates a temporal teacher prediction class vector from a flow frame of the training video data using a temporal teacher model; and a training unit that trains the machine learning model of the training object based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector.

[0009] According to the present disclosure, it is possible to provide a technique for efficiently generating machine learning models for processing video data.

[0010] FIG. 1 is a schematic diagram showing a training process for a movement determination model according to an embodiment of the present disclosure. FIG. 2 is a block diagram showing the hardware configuration of a training device and a movement determination device according to an embodiment of the present disclosure. FIG. 3 is a block diagram showing the functional configuration of a training device according to an embodiment of the present disclosure. FIG. 4 is a schematic diagram showing the architecture of a movement determination model for a training subject according to an embodiment of the present disclosure. FIG. 5 is a flowchart showing a training process according to an embodiment of the present disclosure. FIG. 6 is a schematic diagram showing an inference process for a movement determination model according to an embodiment of the present disclosure. FIG. 7 is a block diagram showing the functional configuration of a movement determination device according to an embodiment of the present disclosure. FIG. 8 is a flowchart showing a movement determination process according to an embodiment of the present disclosure.

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0012] In the following embodiments, a training device for training a machine learning model for processing video data and an action decision device using the trained machine learning model are disclosed.

[0013] [Outline of the Present Disclosure] To summarize the present disclosure, as shown in FIG. 1 , a training device 100 trains a training target action determination model 20 using a training data database (DB) 30, a spatial teacher model 40, and a temporal teacher model 50. Here, the training target action determination model 20 is a machine learning model that, upon receiving video data, determines the action of an object, such as a person, captured in the video data. In an embodiment described below, the training device 100 uses the action determination model 20 to generate a predicted probability vector, a spatial teacher predicted class vector, and a temporal teacher predicted class vector of an object from training video data stored in the training data DB 30. Specifically, the training device 100 extracts a predetermined number of image frames from the training video data, inputs the extracted image frames to the training target action determination model 20, and obtains the predicted probability vector, the spatial teacher predicted class vector, and the temporal teacher predicted class vector from the training target action determination model 20.

[0014] Meanwhile, the training device 100 uses a spatial teacher model 40 and a temporal teacher model 50 prepared in advance as teacher models to train the training target motion decision model 20 according to the distillation method. Specifically, the training device 100 inputs image frames input to the training target motion decision model 20 into the spatial teacher model 40 and acquires a spatial teacher prediction class vector from the spatial teacher model 40. The training device 100 also extracts flow frames such as optical flow for the image frame set, inputs the extracted flow frames into the temporal teacher model 50, and acquires a temporal teacher prediction class vector from the temporal teacher model 50. For example, the spatial teacher prediction class vector may represent a motion class of an object predicted based on spatial features of the object in the input image frames, and the temporal teacher prediction class vector may represent a motion class of an object predicted based on temporal features of the object in the input flow frames.

[0015] After acquiring the spatial teacher prediction class vector and the temporal teacher prediction class vector in this manner, the training device 100 adjusts the parameters of the training target movement determination model 20 based on the error between the training movement label (correct label) corresponding to the image frame and the prediction probability vector output from the training target movement determination model 20, the error between the spatial teacher prediction class vector and the spatial token prediction probability vector output from the training target movement determination model 20, and the error between the temporal teacher prediction class vector and the temporal token prediction probability vector output from the training target movement determination model 20.

[0016] When a predetermined termination condition is satisfied, the training device 100 may provide the finally acquired action determination model 20 to the action determination device 200 as a trained action determination model 60. Upon acquiring the trained action determination model 60, the action determination device 200 determines an action class indicating the action of an object in the video data to be recognized, using the trained action determination model 60, as will be described later.

[0017] In this way, the training target motion decision model 20 is trained according to the distillation method using the pre-prepared spatial teacher model 40 and temporal teacher model 50. This may enable the training device 100 to train the training target motion decision model 20 with a smaller amount of training video data compared to a case in which the training target motion decision model 20 is trained using only a training dataset of training video data without using the spatial teacher model 40 and the temporal teacher model 50.

[0018] In the above-described embodiment, the motion determination model 20 that determines the motion class of an object captured in video data from the video data is described as the machine learning model to be trained, but the training process according to the present disclosure is not necessarily limited to this and can be similarly applied to machine learning models that perform any other task of processing video data.

[0019] Here, the training device 100 and the action determination device 200 are each realized by a computing device such as a server, and may have, for example, a hardware configuration as shown in Fig. 2. That is, the training device 100 and the action determination device 200 each have a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106, which are interconnected via a bus B.

[0020] The programs or instructions that realize the various functions and processes described below in the training device 100 and the movement determination device 200 may be stored on a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or flash memory. When the storage medium is inserted into the drive device 101, the programs or instructions are installed from the storage medium to the storage device 102 or memory device 103 via the drive device 101. However, the programs or instructions do not necessarily have to be installed from the storage medium; they may be downloaded from an external device via a network or the like. The storage device 102 is realized by a hard disk drive or the like and stores installed programs or instructions as well as files, data, etc. used to execute the programs or instructions. The memory device 103 is realized by a random access memory, static memory, etc. and, when a program or instruction is activated, reads and stores the program, instruction, data, etc. from the storage device 102. The storage device 102, memory device 103 and removable storage media may be collectively referred to as non-transitory storage media.

[0021] The processor 104 may be realized by one or more central processing units (CPUs), graphics processing units (GPUs), processing circuitry, etc., which may be configured with one or more processor cores, and executes various functions and processes of the training device 100 and the movement determination device 200, which will be described later, in accordance with programs, instructions, data such as parameters necessary to execute the programs or instructions, etc., stored in the memory device 103. The user interface (UI) device 105 may be configured with input devices such as a keyboard, mouse, camera, and microphone, output devices such as a display, speaker, headset, and printer, and input / output devices such as a touch panel, and implements an interface between the user and the training device 100 and the movement determination device 200. For example, a user operates a GUI (Graphical User Interface) displayed on a display or touch panel using a keyboard, mouse, etc. to operate the training device 100 and the movement determination device 200. The communication device 106 is realized by various communication circuits that execute communication processes with external devices, the Internet, a communication network such as a LAN (Local Area Network), etc.

[0022] However, the above-described hardware configuration is merely an example, and the training device 100 and the action determination device 200 according to the present disclosure may be realized by any other appropriate hardware configuration.

[0023] [Training Device] Next, a training device 100 according to an embodiment of the present disclosure will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the functional configuration of the training device 100 according to an embodiment of the present disclosure. As shown in Fig. 3, the training device 100 includes a training object model processing unit 110, a spatial teacher model processing unit 120, a temporal teacher model processing unit 130, and a training unit 140. For example, one or more functional units of the training object model processing unit 110, the spatial teacher model processing unit 120, the temporal teacher model processing unit 130, and the training unit 140 may be realized by one or more processors 104 executing one or more programs or instructions stored in the memory device 103.

[0024] The training object model processing unit 110 generates a predicted probability vector of an object, a predicted probability vector of a spatial token, and a predicted probability vector of a temporal token from the training video data using a machine learning model of the training object. Specifically, the training object model processing unit 110 acquires the training video data from the training data DB 30 and extracts a predetermined number of image frames (e.g., RGB frames) from the acquired training video data. Here, the extracted image frames may be consecutive adjacent image frames, or may be image frames extracted every few frames.

[0025] The training object model processing unit 110 may divide each extracted image frame into a predetermined number of image patches, and generate an image vector from each image patch to be input to the training object motion determination model 20. For example, as shown in FIG. 4, the training object model processing unit 110 may divide each extracted image frame into nine image patches x ｉ , x ｉ＋１ , ..., x ｉ＋８ and divides x into a predetermined number of image frames. １ , x ２ , ..., x ｎ For example, the image vector of each image patch may be set by flattening the pixel values ​​in the image patch into a one-dimensional vector. Note that the number of image patches divided from each image frame is not necessarily limited to this, and may be set to any appropriate number. Then, the training object model processing unit 110 generates an image vector x １ , x ２ , ..., x ｎ is input to the motion decision model 20 to be trained.

[0026] In the illustrated embodiment, the training target motion decision model 20 includes a video transformer. Specifically, given an input image vector x １ , x ２ , ..., x ｎ is first processed in the linear projection layer and the positional embedding layer to obtain the embedding vector z １ , z２ , ..., z ｎ Then, a class token indicating an action class, a space token indicating a spatial feature, and a time token indicating a temporal feature are added. The class token, the space token, and the time token may be, for example, a vector set randomly.

[0027] Then, the embedding vector z １ , z ２ , ..., z ｎ , the class tokens, spatial tokens, and temporal tokens are encoded in the video transformer layer, and the corresponding intermediate vectors z １ ', z ２ ', ..., z ｎ ', class tokens, spatial tokens, and temporal tokens are generated. Here, the video transformer layer may be configured by any known video transformer model, and detailed description thereof will be omitted here.

[0028] The class tokens, spatial tokens, and temporal tokens generated in the video transformer layer are subjected to dimensional transformation in an MLP (Multi-Layer Perception) head, and are converted into prediction probability vectors, spatial token prediction probability vectors, and temporal token prediction probability vectors, respectively, having dimensions corresponding to the number of action classes. The prediction probability vectors, spatial token prediction probability vectors, and temporal token prediction probability vectors output from the training target action decision model 20 can be compared with the action labels, spatial teacher prediction class vectors, and temporal teacher prediction class vectors, respectively, corresponding to the input training video data, as will be described in detail below.

[0029] The spatial teacher model processing unit 120 generates a spatial teacher prediction class vector from an image frame of the training video data using the spatial teacher model 40. Specifically, the spatial teacher model 40 may be a machine learning model such as a pre-trained convolutional neural network, and upon receiving an image frame, calculates the spatial feature of the image frame. In this embodiment, the spatial teacher model 40 is used as a teacher model for training the training target motion determination model 20 according to the distillation method.

[0030] Specifically, the spatial teacher model processing unit 120 inputs a predetermined number of image frames corresponding to the image vector input to the action determination model 20 of the training target into the spatial teacher model 40, and generates a spatial teacher prediction class vector from the spatial teacher model 40. The spatial teacher model processing unit 120 passes the generated spatial teacher prediction class vector to the training unit 140.

[0031] The temporal teacher model processing unit 130 uses the temporal teacher model 50 to generate a temporal teacher prediction class vector from the flow frames of the training video data. Specifically, the temporal teacher model 50 may be a machine learning model such as a pre-trained convolutional neural network, and upon receiving a flow frame, calculates the temporal feature of the flow frame. In this embodiment, the temporal teacher model 50 is used as a teacher model for training the motion decision model 20 to be trained according to the distillation method.

[0032] Specifically, the temporal teacher model processing unit 130 inputs a flow frame corresponding to the image vector input to the action decision model 20 of the training target into the temporal teacher model 50, and generates a temporal teacher prediction class vector from the temporal teacher model 50. The temporal teacher model processing unit 130 passes the generated temporal teacher prediction class vector to the training unit 140.

[0033] The training unit 140 trains the machine learning model to be trained based on a first error between a training action label corresponding to the training video data and a predicted probability vector, a second error between the spatial token predicted probability vector and the spatial teacher predicted class vector, and a third error between the temporal token predicted probability vector and the temporal teacher predicted class vector. Specifically, the training unit 140 calculates an error between a training action label (correct label) of an image frame corresponding to an image vector input to the training target action determination model 20 and a predicted probability vector output from the training target action determination model 20. Similarly, the training unit 140 calculates an error between a spatial teacher predicted class vector generated from the spatial teacher model 40 for an image frame corresponding to the image vector input to the training target action determination model 20 and a spatial token predicted probability vector output from the training target action determination model 20. Furthermore, the training unit 140 calculates an error between a temporal teacher predicted class vector generated from the temporal teacher model 50 for a flow frame corresponding to the image vector input to the training target action determination model 20 and a temporal token predicted probability vector output from the training target action determination model 20.

[0034] The training unit 140 adjusts the parameters of the training target motion determination model 20 based on these three errors. For example, the training unit 140 may calculate the error L according to the following formula: However, L ＣＥ represents the error between the training action label (correct label) and the predicted probability vector output from the training action decision model 20, and L ＣＥ Ｓｐａｔｉａｌ represents the error between the spatial teacher predicted class vector and the spatial token predicted probability vector output from the training target action decision model 20, and L ＣＥ Ｔｅｍｐｏｒａｌ represents the error between the temporal teacher predicted class vector and the temporal token predicted probability vector output from the training target action decision model 20. Also, σ represents softmax, Z represents logit, y is the correct label, and y ｔ Ｓｐａｔｉａｌ is the spatial teacher predicted class vector, and y ｔＴｅｍｐｏｒａｌ is the time teacher predicted class vector. Also, λ ０ , λ １ , λ ２ is any constant.

[0035] The training unit 140 adjusts the parameters of the movement determination model 20 to be trained, for example, according to the backpropagation algorithm based on the error L. Then, the training unit 140 performs the parameter adjustment described above on the training video data stored in the training data DB 30, and terminates the training process when a predetermined termination condition is satisfied. The training unit 140 may provide the finally acquired movement determination model 20 to, for example, the movement determination device 200 as a trained movement determination model 60. Here, the predetermined termination condition may be, for example, that the training process has been performed on all training data sets in the training data DB 30, that an evaluation value such as the accuracy of the movement determination model 20 has reached a predetermined threshold or more, or that improvement in the evaluation value such as the accuracy of the movement determination model 20 has converged, etc.

[0036] [Training Process] Next, a training process according to an embodiment of the present disclosure will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the training process according to an embodiment of the present disclosure. The training process is performed by the training device 100 described above, and more specifically, may be realized by one or more processors 104 of the training device 100 executing one or more programs or instructions stored in one or more memory devices 103.

[0037] 5, in step S101, the training device 100 generates a predicted probability vector of an object, a predicted probability vector of a spatial token, and a predicted probability vector of a temporal token from training video data using a machine learning model to be trained. Specifically, the training device 100 acquires training video data from the training data DB 30, and calculates an image vector of each image patch from a predetermined number of image frames of the training video data. The training device 100 calculates each calculated image vector x １ , x ２ , ..., x ｎare input to the training target action decision model 20, and a prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector are obtained from the training target action decision model 20.

[0038] In step S102, the training device 100 generates a spatial teacher prediction class vector from the image frames of the training video data using the spatial teacher model 40. Specifically, the training device 100 inputs a predetermined number of image frames corresponding to the image vector input to the training target action determination model 20 in step S101 into the spatial teacher model 40, and acquires the spatial teacher prediction class vector from the spatial teacher model 40.

[0039] In step S103, the training device 100 generates a temporal teacher prediction class vector from the image frame of the training video data using the temporal teacher model 50. Specifically, the training device 100 inputs a flow frame corresponding to the image vector input to the motion decision model 20 of the training target in step S101 to the temporal teacher model 50, and acquires the temporal teacher prediction class vector from the temporal teacher model 50.

[0040] In step S104, the training device 100 adjusts the parameters of the action decision model 20 to be trained based on the error between the predicted probability vector obtained in step S101 and the corresponding correct label, the error between the spatial token predicted probability vector obtained in step S101 and the spatial teacher predicted class vector obtained in step S102, and the error between the temporal token predicted probability vector obtained in step S101 and the temporal teacher predicted class vector obtained in step S103.

[0041] In step S105, the training device 100 determines whether a predetermined termination condition is satisfied. If the predetermined termination condition is satisfied (S105: YES), the training device 100 may terminate the training process and provide the finally acquired movement determination model 20 as the trained movement determination model 60. On the other hand, if the predetermined termination condition is not satisfied (S105: NO), the training device 100 returns to step S101 and repeats the above-described process for the next training video data.

[0042] According to the training device 100 and training process described above, the training target movement determination model 20 is trained according to the distillation method using the pre-prepared spatial teacher model 40 and temporal teacher model 50. This makes it possible to train the training target movement determination model 20 with a smaller amount of training video data compared to a case in which the training target movement determination model 20 is trained using only a training data set of training video data without using the spatial teacher model 40 and the temporal teacher model 50.

[0043] [Movement Determination Apparatus] Next, a description will be given of a movement determination apparatus 200 according to an embodiment of the present disclosure. Fig. 6 is a schematic diagram showing the movement determination apparatus 200 according to an embodiment of the present disclosure. As shown in Fig. 6, when the movement determination apparatus 200 acquires video data to be determined, it determines a movement class of an object captured in the video data by using a trained movement determination model 60 provided by the training apparatus 100.

[0044] 7 is a block diagram showing a functional configuration of an action determination device 200 according to an embodiment of the present disclosure. As shown in FIG. 7, the action determination device 200 includes an acquisition unit 210 and an action determination unit 220.

[0045] The acquisition unit 210 acquires video data. Specifically, the acquisition unit 210 acquires video data from a camera that is communicatively connected to the action determination device 200, and passes the acquired video data to the action determination unit 220. For example, as described in the training process, the acquisition unit 210 extracts a predetermined number of image frames from the acquired video data and generates image vectors corresponding to image patches from the extracted image frames. The generated image vectors are passed to the action determination unit 220.

[0046] The movement determination unit 220 uses a machine learning model to generate a predicted probability vector of an object from video data, and determines a movement class of the object based on the predicted probability vector. Here, the machine learning model may be one generated from the machine learning model of the training target by the training device 100 described above. Specifically, the movement determination unit 220 inputs the acquired image vector into the movement determination model 60, and acquires the predicted probability vector from the movement determination model 60. Then, the movement determination unit 220 may determine the movement class with the highest confidence in the predicted probability vector, and determine the determined movement class as the movement of the object.

[0047] [Action Determination Process] Next, an action determination process according to an embodiment of the present disclosure will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the action determination process according to an embodiment of the present disclosure. The action determination process is executed by the above-described action determination device 200, and more specifically, may be realized by one or more processors 104 of the action determination device 200 executing one or more programs or instructions stored in one or more memory devices 103.

[0048] 8, in step S201, the motion determination apparatus 200 acquires video data, extracts a predetermined number of image frames from the video data, and generates image vectors corresponding to image patches from the extracted image frames.

[0049] In step S202, the action determination device 200 generates a predicted probability vector using a machine learning model. For example, the action determination device 200 inputs an image vector to the action determination model 60 trained by the training device 100, and obtains a predicted probability vector indicating the action class of the object from the action determination model 60.

[0050] In step S203, the motion determination device 200 determines a motion class of the object based on the predicted probability vector. Specifically, the motion determination device 200 may determine a motion class with the highest degree of certainty in the predicted probability vector acquired in step S202, and determine the determined motion class as the motion of the object.

[0051] According to the above-described action determination device 200 and action determination process, the action determination model 60 trained according to the distillation method using the previously prepared spatial teacher model 40 and temporal teacher model 50 is used for action determination. This makes it possible to acquire the trained action determination model 60 with a smaller amount of training video data compared to a case in which the action determination model 20 to be trained is trained using only the training data set of training video data without using the spatial teacher model 40 and the temporal teacher model 50.

[0052] (Supplementary Note 1) A training device comprising: a training object model processing unit that generates an object prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector from training video data using a machine learning model to be trained; a spatial teacher model processing unit that generates a spatial teacher prediction class vector from an image frame of the training video data using a spatial teacher model; a temporal teacher model processing unit that generates a temporal teacher prediction class vector from a flow frame of the training video data using a temporal teacher model; and a training unit that trains the machine learning model to be trained based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector. (Supplementary Note 2) The training device of Supplementary Note 1, wherein the machine learning model to be trained includes a video transformer. (Supplementary Note 3) The training device of Supplementary Note 2, wherein the video transformer accepts class tokens, spatial tokens, and temporal tokens. (Supplementary Note 4) The training device according to any one of Supplements 1 to 3, wherein the training video data is composed of image patches of a predetermined number of frames. (Supplementary Note 5) The training device according to any one of Supplements 1 to 4, wherein the predicted probability vector indicates a motion class of the object.(Supplementary Note 6) A computer-implemented training method comprising: generating an object prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector from training video data using a machine learning model to be trained; generating a spatial teacher prediction class vector from an image frame of the training video data using a spatial teacher model; generating a temporal teacher prediction class vector from a flow frame of the training video data using a temporal teacher model; and training the machine learning model to be trained based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector. (Supplementary Note 7) A program that causes a computer to execute the following steps: generating an object prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector from training video data using a machine learning model to be trained; generating a spatial teacher prediction class vector from an image frame of the training video data using a spatial teacher model; generating a temporal teacher prediction class vector from a flow frame of the training video data using a temporal teacher model; and training the machine learning model to be trained based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector.(Supplementary Note 8) A machine learning model generated by a computer executing the following steps: generating an object prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector from training video data using a machine learning model to be trained; generating a spatial teacher prediction class vector from an image frame of the training video data using a spatial teacher model; generating a temporal teacher prediction class vector from a flow frame of the training video data using a temporal teacher model; and training the machine learning model to be trained based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector. and a motion determination unit that uses a machine learning model to generate a predicted probability vector of an object from the video data and determine a motion class of the object based on the predicted probability vector, wherein the machine learning model is generated by executing the following: using a machine learning model to be trained to generate a predicted probability vector of an object, a spatial token predicted probability vector, and a temporal token predicted probability vector from training video data; using a spatial teacher model to generate a spatial teacher predicted class vector from an image frame of the training video data; and using a temporal teacher model to generate a temporal teacher predicted class vector from a flow frame of the training video data; and training the machine learning model to be trained based on a first error between a training action label corresponding to the training video data and the predicted probability vector, a second error between the spatial token predicted probability vector and the spatial teacher predicted class vector, and a third error between the temporal token predicted probability vector and the temporal teacher predicted class vector.(Supplementary Note 10) An action determination method, wherein a computer executes the steps of: acquiring video data; generating a predicted probability vector of an object from the video data using a machine learning model; and determining an action class of the object based on the predicted probability vector, wherein the machine learning model is generated by executing the steps of: using a machine learning model to be trained to generate a predicted probability vector of an object, a spatial token predicted probability vector, and a temporal feature vector from training video data; using a spatial teacher model to generate a spatial teacher predicted class vector from image frames of the training video data; and using a temporal teacher model to generate a temporal teacher predicted class vector from flow frames of the training video data; and training the machine learning model to be trained based on a first error between a training action label corresponding to the training video data and the predicted probability vector, a second error between the spatial token predicted probability vector and the spatial teacher predicted class vector, and a third error between the temporal feature vector and the temporal teacher predicted class vector.(Supplementary Note 11) A program that causes a computer to execute the following steps: acquire video data; generate a predicted probability vector of an object from the video data using a machine learning model; and determine an action class of the object based on the predicted probability vector; wherein the machine learning model is generated by: using a machine learning model to be trained to generate a predicted probability vector of an object, a spatial token predicted probability vector, and a temporal token predicted probability vector from training video data; using a spatial teacher model to generate a spatial teacher predicted class vector from image frames of the training video data; and using a temporal teacher model to generate a temporal teacher predicted class vector from flow frames of the training video data; and training the machine learning model to be trained based on a first error between a training action label corresponding to the training video data and the predicted probability vector, a second error between the spatial token predicted probability vector and the spatial teacher predicted class vector, and a third error between the temporal token predicted probability vector and the temporal teacher predicted class vector.

[0053] Although examples of the present disclosure have been described in detail above, the present disclosure is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present disclosure as set forth in the claims.

[0054] The disclosures of the specification, drawings and abstract contained in Japanese Patent Application No. 2023-121715, filed on July 26, 2023, are incorporated herein by reference in their entirety.

[0055] The present disclosure is useful for an apparatus and method for training a machine learning model for processing video data.

[0056] 20 Motion determination model of training target 30 Training data DB 40 Spatial teacher model 50 Temporal teacher model 60 Trained motion determination model 100 Training device 110 Training target model processing unit 120 Spatial teacher model processing unit 130 Temporal teacher model processing unit 140 Training unit 200 Motion determination device 210 Acquisition unit 220 Motion determination unit

Claims

1. A training device comprising: a training object model processing unit that generates an object prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector from training video data using a machine learning model of a training object; a spatial teacher model processing unit that generates a spatial teacher prediction class vector from an image frame of the training video data using a spatial teacher model; a temporal teacher model processing unit that generates a temporal teacher prediction class vector from a flow frame of the training video data using a temporal teacher model; and a training unit that trains the machine learning model of the training object based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector.

2. The training device of claim 1, wherein the machine learning model to be trained includes a video transformer.

3. The training device of claim 2, wherein the video transformer accepts class tokens, spatial tokens and temporal tokens.

4. The training device of claim 1, wherein the training video data consists of image patches of a predetermined number of frames.

5. The training apparatus of claim 1, wherein the predicted probability vector indicates a motion class of the object.

6. A computer-implemented training method, comprising: using a machine learning model to be trained to generate an object prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector from training video data; using a spatial teacher model to generate a spatial teacher prediction class vector from an image frame of the training video data; and using a temporal teacher model to generate a temporal teacher prediction class vector from a flow frame of the training video data; and training the machine learning model to be trained based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector.

7. A program for causing a computer to execute the following steps: using a machine learning model to be trained to generate an object prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector from training video data; using a spatial teacher model to generate a spatial teacher prediction class vector from an image frame of the training video data; and using a temporal teacher model to generate a temporal teacher prediction class vector from a flow frame of the training video data; and training the machine learning model to be trained based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector.

8. A machine learning model generated by a computer executing the following steps: using a machine learning model to be trained to generate an object prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector from training video data; using a spatial teacher model to generate a spatial teacher prediction class vector from an image frame of the training video data; and using a temporal teacher model to generate a temporal teacher prediction class vector from a flow frame of the training video data; and training the machine learning model to be trained based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector.

9. An action determination device comprising: an acquisition unit for acquiring video data; and an action determination unit for generating a predicted probability vector of an object from the video data using a machine learning model, and determining an action class of the object based on the predicted probability vector, wherein the machine learning model is generated by executing the following: using a machine learning model to be trained to generate a predicted probability vector of an object, a spatial token prediction probability vector, and a temporal token prediction probability vector from training video data; using a spatial teacher model to generate a spatial teacher prediction class vector from an image frame of the training video data; using a temporal teacher model to generate a temporal teacher prediction class vector from a flow frame of the training video data; and training the machine learning model to be trained based on a first error between a training action label corresponding to the training video data and the predicted probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector.

10. A method for determining an action class of the object, the method being performed by a computer: acquiring video data; using a machine learning model to generate a predicted probability vector of an object from the video data; and determining an action class of the object based on the predicted probability vector, the machine learning model being generated by executing: using a machine learning model to be trained to generate a predicted probability vector of an object, a spatial token predicted probability vector, and a temporal token predicted probability vector from training video data; using a spatial teacher model to generate a spatial teacher predicted class vector from an image frame of the training video data; and using a temporal teacher model to generate a temporal teacher predicted class vector from a flow frame of the training video data; and training the machine learning model to be trained based on a first error between a training action label corresponding to the training video data and the predicted probability vector, a second error between the spatial token predicted probability vector and the spatial teacher predicted class vector, and a third error between the temporal token predicted probability vector and the temporal teacher predicted class vector.

11. A program for causing a computer to execute the following steps: acquiring video data; generating a prediction probability vector of an object from the video data using a machine learning model; and determining an action class of the object based on the prediction probability vector; wherein the machine learning model is generated by executing: generating a prediction probability vector of an object, a spatial token prediction probability vector, and a temporal token prediction probability vector from training video data using a machine learning model to be trained; generating a spatial teacher prediction class vector from an image frame of the training video data using a spatial teacher model; generating a temporal teacher prediction class vector from a flow frame of the training video data using a temporal teacher model; and training the machine learning model to be trained based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector.

Images