Training devices, training methods, programs, and machine learning models
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
- Filing Date
- 2023-07-26
- Publication Date
- 2026-07-03
AI Technical Summary
Existing machine learning models require a large amount of training data and computing resources in video data processing, resulting in time-consuming and resource consumption.
A method called knowledge distillation is adopted to guide the training process of video data using pre-trained spatial and temporal teacher models, and adjust model parameters by generating predicted probability vectors of spatial and temporal features to reduce dependence on training data.
It realizes efficient generation of machine learning models that can process video data with less training data, reducing computational and time costs.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a training device, a training method, a machine learning model, an action determination device, an action determination method, and a program. [Background technology]
[0002] With the recent evolution of deep learning technology, machine learning models are being used in a wide range of technical fields. For example, in image processing and language processing, research and development of machine learning models is progressing rapidly. Under these circumstances, research and development of machine learning models for video data, i.e., video data, is being attempted using technologies developed in image processing and language processing. For example, the application of convolutional neural networks, which are effectively used in image processing, and transformer models, which are utilized in language processing, to machine learning models for video data is being considered.
[0003] For example, as a machine learning model for video data, development of an action recognition model that recognizes human actions, object movements, etc. In a typical action recognition process, video data showing the movement of an object is input to an action recognition model, and the action class of the object (e.g., walking, moving, performing abnormal behavior, etc.) is output from the action recognition model. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Patent No. 6875058 Summary of the Invention [Problem to be solved by the invention]
[0005] In the motion recognition process of video data, an approach has been considered in which a machine learning model is used to extract spatial features for recognizing an object in each image frame and temporal features for recognizing the movement of the object across multiple image frames, and the motion of the object in the video data is determined based on the spatial features and temporal features. In such an approach, for example, a convolutional neural network for extracting spatial features and a convolutional neural network for extracting temporal features may be used. In addition, an approach has also been proposed in which a transformer model for processing time-series data such as language processing is used to process video data.
[0006] However, it is known that training convolutional neural networks and transformer models requires large training datasets, and the training process using these training datasets can require a lot of computational and time resources.
[0007] In view of the above problems, one objective of the present disclosure is to provide a technique for efficiently generating a machine learning model for processing video data. [Means for solving the problem]
[0008] One aspect of the present disclosure relates to a training device having a training object model processing unit that uses a machine learning model of a training object to generate a predicted probability vector of an object, a spatial token predicted probability vector, and a temporal token predicted probability vector from training video data; a spatial teacher model processing unit that uses a spatial teacher model to generate a spatial teacher predicted class vector from an image frame of the training video data; a temporal teacher model processing unit that uses a temporal teacher model to generate a temporal teacher predicted class vector from a flow frame of the training video data; and a training unit that trains the machine learning model of the training object based on a first error between a training action label corresponding to the training video data and the predicted probability vector, a second error between the spatial token predicted probability vector and the spatial teacher predicted class vector, and a third error between the temporal token predicted probability vector and the temporal teacher predicted class vector. Effect of the Invention
[0009] According to the present disclosure, it is possible to provide a technique for efficiently generating machine learning models for processing video data. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a schematic diagram illustrating a training process for an action decision model according to one embodiment of the present disclosure. [Diagram 2] FIG. 2 is a block diagram illustrating a hardware configuration of the training device and the action determination device according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 is a block diagram illustrating a functional configuration of a training device according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a schematic diagram illustrating the architecture of a training object's motion decision model according to one embodiment of the present disclosure. [Diagram 5] FIG. 5 is a flow chart illustrating a training process according to one embodiment of the present disclosure. [Figure 6] FIG. 6 is a schematic diagram illustrating an inference process of an action decision model according to one embodiment of the present disclosure. [Figure 7] FIG. 7 is a block diagram illustrating a functional configuration of an action determination device according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a flowchart illustrating an action determination process according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.
[0012] In the following embodiments, a training device for training a machine learning model for processing video data and an action decision device using the trained machine learning model are disclosed.
[0013] [Summary of the Disclosure] To summarize the present disclosure, as shown in FIG. 1, a training device 100 uses a training data database (DB) 30, a spatial teacher model 40, and a temporal teacher model 50 to train a motion determination model 20 of a training target. Here, the motion determination model 20 of a training target is a machine learning model that receives video data and determines the motion of an object such as a person captured in the video data. In an embodiment described later, the training device 100 uses the motion determination model 20 to generate a predicted probability vector, a spatial teacher predicted class vector, and a temporal teacher predicted class vector of an object from training video data stored in the training data DB 30. Specifically, the training device 100 extracts a predetermined number of image frames from the training video data, inputs the extracted image frames to the motion determination model 20 of a training target, and obtains a predicted probability vector, a spatial teacher predicted class vector, and a temporal teacher predicted class vector from the motion determination model 20 of a training target.
[0014] On the other hand, the training device 100 uses the spatial teacher model 40 and the temporal teacher model 50 prepared in advance as teacher models to train the motion decision model 20 of the training target according to the distillation method. Specifically, the training device 100 inputs the image frame input to the motion decision model 20 of the training target into the spatial teacher model 40, and obtains a spatial teacher prediction class vector from the spatial teacher model 40. In addition, the training device 100 extracts a flow frame such as an optical flow for the image frame set, inputs the extracted flow frame into the temporal teacher model 50, and obtains a temporal teacher prediction class vector from the temporal teacher model 50. For example, the spatial teacher prediction class vector may represent the motion class of an object predicted based on the spatial features of the object in the input image frame, and the temporal teacher prediction class vector may represent the motion class of an object predicted based on the temporal features of the object in the input flow frame.
[0015] After obtaining the spatial teacher prediction class vector and the temporal teacher prediction class vector in this manner, the training device 100 adjusts the parameters of the training target action decision model 20 based on the error between the training action label (correct label) corresponding to the image frame and the prediction probability vector output from the training target action decision model 20, the error between the spatial teacher prediction class vector and the spatial token prediction probability vector output from the training target action decision model 20, and the error between the temporal teacher prediction class vector and the temporal token prediction probability vector output from the training target action decision model 20.
[0016] When a predetermined termination condition is satisfied, the training device 100 may provide the finally acquired motion determination model 20 to the motion determination device 200 as a trained motion determination model 60. Upon acquiring the trained motion determination model 60, the motion determination device 200 determines a motion class indicating a motion of an object in the video data to be recognized, using the trained motion determination model 60, as described later.
[0017] In this way, the motion determination model 20 to be trained is trained according to the distillation method using the spatial teacher model 40 and the temporal teacher model 50 prepared in advance. This makes it possible for the training device 100 to train the motion determination model 20 to be trained using a smaller amount of training video data compared to a case in which the motion determination model 20 to be trained is trained using only the training data set of the training video data without using the spatial teacher model 40 and the temporal teacher model 50.
[0018] In the above-described embodiment, the motion determination model 20 that determines the motion class of an object captured in the video data from the video data is described as the machine learning model to be trained, however, the training process according to the present disclosure is not necessarily limited to this and can be similarly applied to a machine learning model that performs any other task of processing video data.
[0019] Here, the training device 100 and the action determination device 200 are each realized by a computing device such as a server, and may have, for example, a hardware configuration as shown in Fig. 2. That is, the training device 100 and the action determination device 200 each have a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106, which are interconnected via a bus B.
[0020] The programs or instructions for implementing various functions and processes of the training device 100 and the motion determination device 200 described later may be stored in a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or a flash memory. When the storage medium is set in the drive device 101, the programs or instructions are installed from the storage medium to the storage device 102 or the memory device 103 via the drive device 101. However, the programs or instructions do not necessarily need to be installed from the storage medium, and may be downloaded from any external device via a network or the like. The storage device 102 is realized by a hard disk drive or the like, and stores files, data, etc. used to execute the programs or instructions together with the installed programs or instructions. The memory device 103 is realized by a random access memory, a static memory, etc., and when the programs or instructions are activated, reads out the programs, instructions, data, etc. from the storage device 102 and stores them. The storage device 102, the memory device 103, and the removable storage medium may be collectively referred to as a non-transitory storage medium.
[0021] The processor 104 may be realized by one or more central processing units (CPUs), graphics processing units (GPUs), processing circuitry, etc., each of which may be configured with one or more processor cores, and executes various functions and processes of the training device 100 and the motion determination device 200 described below according to programs, instructions, data such as parameters necessary to execute the programs or instructions, etc., stored in the memory device 103. The user interface (UI) device 105 may be configured by input devices such as a keyboard, mouse, camera, microphone, etc., output devices such as a display, speaker, headset, printer, etc., and input / output devices such as a touch panel, and realizes an interface between the user and the training device 100 and the motion determination device 200. For example, the user operates the training device 100 and the motion determination device 200 by operating a graphical user interface (GUI) displayed on a display or a touch panel using a keyboard, mouse, etc. The communication device 106 is realized by various communication circuits that execute communication processes with external devices, the Internet, a communication network such as a LAN (Local Area Network), etc.
[0022] However, the above-described hardware configuration is merely an example, and the training device 100 and the movement determination device 200 according to the present disclosure may be realized by any other appropriate hardware configuration.
[0023] [Training device] Next, a training device 100 according to an embodiment of the present disclosure will be described with reference to FIG. 3. FIG. 3 is a block diagram showing a functional configuration of the training device 100 according to an embodiment of the present disclosure. As shown in FIG. 3, the training device 100 has a training object model processing unit 110, a spatial teacher model processing unit 120, a temporal teacher model processing unit 130, and a training unit 140. For example, one or more functional units of the training object model processing unit 110, the spatial teacher model processing unit 120, the temporal teacher model processing unit 130, and the training unit 140 may be realized by one or more processors 104 executing one or more programs or instructions stored in the memory device 103.
[0024] The training object model processing unit 110 generates a predicted probability vector of an object, a spatial token predicted probability vector, and a temporal token predicted probability vector from the training video data using a machine learning model of the training object. Specifically, the training object model processing unit 110 acquires training video data from the training data DB 30, and extracts a predetermined number of image frames (e.g., RGB frames, etc.) from the acquired training video data. Here, the extracted image frames may be consecutive adjacent image frames, or may be image frames extracted every few frames.
[0025] The training object model processor 110 may divide each extracted image frame into a predetermined number of image patches and generate an image vector from each image patch for input to the training object motion determination model 20. For example, as shown in FIG. 4, the training object model processor 110 may divide each extracted image frame into 9 image patches x i ,x i+1 ,···,x i+8 For a given number of image frames, nFor example, the image vector of each image patch may be set by flattening the pixel values in the image patch into a one-dimensional vector. Note that the number of image patches divided from each image frame is not necessarily limited to this, and may be set to any appropriate value. Then, the training object model processing unit 110 generates an image vector x1, x2,..., x n is input to the motion decision model 20 to be trained.
[0026] In the illustrated embodiment, the motion decision model 20 to be trained includes a video transformer. Specifically, given input image vectors x1, x2,..., x n is first processed in the linear projection layer and the positional embedding layer to obtain the embedding vectors z1, z2, . . . , z n Then, a class token indicating an action class, a spatial token indicating a spatial feature, and a time token indicating a time feature are added. The class token, the spatial token, and the time token may be, for example, a vector set randomly.
[0027] Then, the embedding vectors z1, z2, , z n ,The class tokens, spatial tokens and temporal tokens are encoded in the video transformer layer, and the corresponding intermediate vectors z1', z2',..., z n ', class tokens, spatial tokens and time tokens are generated. Here, the video transformer layer may be configured by any known video transformer model, and the detailed description thereof will be omitted here.
[0028] The class tokens, spatial tokens, and temporal tokens generated in the video transformer layer are dimensionally transformed in the MLP (Multi Layer Perception) head, and are converted into a prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector, each having a dimension corresponding to the number of action classes. The prediction probability vector, the spatial token prediction probability vector, and the temporal token prediction probability vector output from the training target action decision model 20 can be compared with the action label, the spatial teacher prediction class vector, and the temporal teacher prediction class vector, respectively, corresponding to the input training video data, as will be described in detail below.
[0029] The spatial teacher model processing unit 120 uses the spatial teacher model 40 to generate a spatial teacher prediction class vector from an image frame of the training video data. Specifically, the spatial teacher model 40 may be a machine learning model such as a pre-trained convolutional neural network, and upon receiving an image frame, calculates the spatial feature amount of the image frame. In this embodiment, the spatial teacher model 40 is used as a teacher model for training the motion determination model 20 to be trained according to the distillation method.
[0030] Specifically, the spatial teacher model processing unit 120 inputs a predetermined number of image frames corresponding to the image vector input to the action determination model 20 of the training target to the spatial teacher model 40, and generates a spatial teacher prediction class vector from the spatial teacher model 40. The spatial teacher model processing unit 120 passes the generated spatial teacher prediction class vector to the training unit 140.
[0031] The temporal teacher model processing unit 130 uses the temporal teacher model 50 to generate a temporal teacher prediction class vector from the flow frame of the training video data. Specifically, the temporal teacher model 50 may be a machine learning model such as a pre-trained convolutional neural network, and upon receiving a flow frame, calculates the temporal feature amount of the flow frame. In this embodiment, the temporal teacher model 50 is used as a teacher model for training the motion decision model 20 to be trained according to the distillation method.
[0032] Specifically, the temporal teacher model processing unit 130 inputs a flow frame corresponding to the image vector input to the motion decision model 20 of the training target into the temporal teacher model 50, and generates a temporal teacher prediction class vector from the temporal teacher model 50. The temporal teacher model processing unit 130 passes the generated temporal teacher prediction class vector to the training unit 140.
[0033] The training unit 140 trains the machine learning model to be trained based on the first error between the training action label corresponding to the training video data and the predicted probability vector, the second error between the spatial token predicted probability vector and the spatial teacher predicted class vector, and the third error between the temporal token predicted probability vector and the temporal teacher predicted class vector. Specifically, the training unit 140 calculates the error between the training action label (correct answer label) of the image frame corresponding to the image vector input to the motion determination model 20 to be trained and the predicted probability vector output from the motion determination model 20 to be trained. Similarly, the training unit 140 calculates the error between the spatial teacher predicted class vector generated from the spatial teacher model 40 for the image frame corresponding to the image vector input to the motion determination model 20 to be trained and the spatial token predicted probability vector output from the motion determination model 20 to be trained. Furthermore, the training unit 140 calculates the error between the temporal teacher predicted class vector generated from the temporal teacher model 50 for the flow frame corresponding to the image vector input to the motion determination model 20 to be trained and the temporal token predicted probability vector output from the motion determination model 20 to be trained.
[0034] The training unit 140 adjusts the parameters of the training target motion determination model 20 based on these three errors. For example, the training unit 140 may calculate the error L according to the following formula.
number
[0035] The training unit 140 adjusts the parameters of the motion determination model 20 to be trained according to, for example, the backpropagation method based on the error L. Then, the training unit 140 performs the above-mentioned parameter adjustment on the training video data stored in the training data DB 30, and ends the training process when a predetermined end condition is satisfied. The training unit 140 may provide the finally acquired motion determination model 20 as a trained motion determination model 60 to, for example, the motion determination device 200. Here, the predetermined end condition may be, for example, that the training process has been performed on all training data sets in the training data DB 30, that an evaluation value such as the accuracy of the motion determination model 20 has reached a predetermined threshold value or more, that an improvement in the evaluation value such as the accuracy of the motion determination model 20 has converged, and the like.
[0036] [Training process] Next, a training process according to an embodiment of the present disclosure will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the training process according to an embodiment of the present disclosure. The training process is performed by the training device 100 described above, and more specifically, may be realized by one or more processors 104 of the training device 100 executing one or more programs or instructions stored in one or more memory devices 103.
[0037] 5, in step S101, the training device 100 generates a prediction probability vector of an object, a prediction probability vector of a spatial token, and a prediction probability vector of a temporal token from training video data using a machine learning model of a training target. Specifically, the training device 100 obtains training video data from the training data DB 30, and calculates an image vector of each image patch from a predetermined number of image frames of the training video data. The training device 100 calculates each of the calculated image vectors x1, x2,..., x n is input to the training object motion decision model 20, and a prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector are obtained from the training object motion decision model 20.
[0038] In step S102, the training device 100 generates a spatial teacher prediction class vector from the image frames of the training video data using the spatial teacher model 40. Specifically, the training device 100 inputs a predetermined number of image frames corresponding to the image vector input to the motion determination model 20 of the training target in step S101 to the spatial teacher model 40, and obtains a spatial teacher prediction class vector from the spatial teacher model 40.
[0039] In step S103, the training device 100 generates a temporal teacher prediction class vector from the image frame of the training video data by using the temporal teacher model 50. Specifically, the training device 100 inputs a flow frame corresponding to the image vector input to the motion decision model 20 of the training target in step S101 into the temporal teacher model 50, and obtains a temporal teacher prediction class vector from the temporal teacher model 50.
[0040] In step S104, the training device 100 adjusts parameters of the action decision model 20 to be trained based on the error between the prediction probability vector obtained in step S101 and the corresponding correct label, the error between the spatial token prediction probability vector obtained in step S101 and the spatial teacher prediction class vector obtained in step S102, and the error between the temporal token prediction probability vector obtained in step S101 and the temporal teacher prediction class vector obtained in step S103.
[0041] In step S105, the training device 100 determines whether a predetermined end condition is satisfied. If the predetermined end condition is satisfied (S105: YES), the training device 100 may terminate the training process and provide the finally acquired action determination model 20 as a trained action determination model 60. On the other hand, if the predetermined end condition is not satisfied (S105: NO), the training device 100 returns to step S101 and repeats the above-mentioned process for the next training video data.
[0042] According to the above-mentioned training device 100 and training process, the motion determination model 20 of the training target is trained according to the distillation method using the spatial teacher model 40 and the temporal teacher model 50 prepared in advance. As a result, the motion determination model 20 of the training target can be trained with a smaller amount of training video data, compared to a case in which the motion determination model 20 of the training target is trained using only the training data set of the training video data without using the spatial teacher model 40 and the temporal teacher model 50.
[0043] [Motion determination device] Next, a description will be given of an action determination device 200 according to an embodiment of the present disclosure. Fig. 6 is a schematic diagram showing the action determination device 200 according to an embodiment of the present disclosure. As shown in Fig. 6, when the action determination device 200 acquires video data to be determined, the action determination device 200 determines the action class of an object captured in the video data by using the trained action determination model 60 provided by the training device 100.
[0044] 7 is a block diagram showing a functional configuration of an action determination device 200 according to an embodiment of the present disclosure. As shown in FIG. 7, the action determination device 200 includes an acquisition unit 210 and an action determination unit 220.
[0045] The acquisition unit 210 acquires video data. Specifically, the acquisition unit 210 acquires video data from a camera that is communicatively connected to the motion determination device 200, and passes the acquired video data to the motion determination unit 220. For example, as described in the training process, the acquisition unit 210 extracts a predetermined number of image frames from the acquired video data, and generates an image vector corresponding to an image patch from the extracted image frames. The generated image vector is passed to the motion determination unit 220.
[0046] The movement determination unit 220 uses a machine learning model to generate a predicted probability vector of an object from video data, and determines a movement class of the object based on the predicted probability vector. Here, the machine learning model may be generated from a machine learning model of a training target by the above-mentioned training device 100. Specifically, the movement determination unit 220 inputs an acquired image vector to the movement determination model 60, and obtains a predicted probability vector from the movement determination model 60. Then, the movement determination unit 220 may determine a movement class with the highest confidence in the predicted probability vector, and determine the determined movement class as the movement of the object.
[0047] [Action decision process] Next, an action determination process according to an embodiment of the present disclosure will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the action determination process according to an embodiment of the present disclosure. The action determination process is executed by the above-mentioned action determination device 200, and more specifically, may be realized by one or more processors 104 of the action determination device 200 executing one or more programs or instructions stored in one or more memory devices 103.
[0048] 8, in step S201, the motion determination apparatus 200 acquires video data. The motion determination apparatus 200 extracts a predetermined number of image frames from the video data, and generates image vectors corresponding to image patches from the extracted image frames.
[0049] In step S202, the action determination device 200 generates a predicted probability vector using a machine learning model. For example, the action determination device 200 inputs an image vector to the action determination model 60 trained by the training device 100, and obtains a predicted probability vector indicating an action class of an object from the action determination model 60.
[0050] In step S203, the motion determination device 200 determines a motion class of the object based on the predicted probability vector. Specifically, the motion determination device 200 may determine a motion class with the highest confidence in the predicted probability vector acquired in step S202, and determine the determined motion class as the motion of the object.
[0051] According to the above-mentioned motion determination device 200 and motion determination process, the motion determination model 60 trained according to the distillation method using the spatial teacher model 40 and the temporal teacher model 50 prepared in advance is used for motion determination. This makes it possible to obtain the trained motion determination model 60 with a smaller amount of training video data, compared to a case in which the motion determination model 20 to be trained is trained using only the training data set of the training video data without using the spatial teacher model 40 and the temporal teacher model 50.
[0052] (Appendix 1) a training object model processing unit that generates an object prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector from the training video data using a machine learning model to be trained; a spatial teacher model processing unit that uses a spatial teacher model to generate a spatial teacher prediction class vector from the image frames of the training video data; A temporal teacher model processing unit uses a temporal teacher model to generate a temporal teacher predicted class vector from the flow frames of the training video data; a training unit that trains the machine learning model of the training target based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector; A training device having the above features. (Appendix 2) 2. The training apparatus of claim 1, wherein the machine learning model to be trained includes a video transformer. (Appendix 3) 3. The training apparatus of claim 2, wherein the video transformer accepts class tokens, spatial tokens, and temporal tokens. (Appendix 4) 4. The training device according to any one of appendices 1 to 3, wherein the training video data is composed of image patches of a predetermined number of frames. (Appendix 5) 5. The training device according to any one of Supplementary Notes 1 to 4, wherein the predicted probability vector indicates a motion class of the object. (Appendix 6) generating an object prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector from the training video data using the machine learning model to be trained; generating spatially trained predicted class vectors from image frames of the training video data utilizing a spatially trained model; generating a temporally-supervised predicted class vector from the flow frames of the training video data utilizing a temporally-supervised model; training the machine learning model of the training target based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector; A training method performed by a computer. (Appendix 7) generating an object prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector from the training video data using the machine learning model to be trained; generating spatially trained predicted class vectors from image frames of the training video data utilizing a spatially trained model; generating a temporally-supervised predicted class vector from the flow frames of the training video data utilizing a temporally-supervised model; training the machine learning model of the training target based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector; A program that causes a computer to execute the following. (Appendix 8) generating an object prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector from the training video data using the machine learning model to be trained; generating spatially trained predicted class vectors from image frames of the training video data utilizing a spatially trained model; generating a temporally-supervised predicted class vector from the flow frames of the training video data utilizing a temporally-supervised model; training the machine learning model of the training target based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector; A machine learning model is generated by a computer running the above. (Appendix 9) An acquisition unit for acquiring video data; a motion determination unit that uses a machine learning model to generate a predicted probability vector of an object from the video data and determines a motion class of the object based on the predicted probability vector; having The machine learning model is generating an object prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector from the training video data using the machine learning model to be trained; generating spatially trained predicted class vectors from image frames of the training video data utilizing a spatially trained model; generating a temporally-supervised predicted class vector from the flow frames of the training video data utilizing a temporally-supervised model; training the machine learning model of the training target based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector; The action determination device is generated by executing the above. (Appendix 10) Obtaining video data; and utilizing a machine learning model to generate a predicted probability vector of an object from the video data; determining a behavioral class of the object based on the predicted probability vector; The computer executes The machine learning model is generating an object prediction probability vector, a spatial token prediction probability vector, and a temporal feature vector from training video data using a machine learning model to be trained; generating spatially trained predicted class vectors from image frames of the training video data utilizing a spatially trained model; generating a temporally-supervised predicted class vector from the flow frames of the training video data utilizing a temporally-supervised model; training the machine learning model of the training target based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal feature vector and the temporal teacher prediction class vector; The action determination method is generated by executing (Appendix 11) Obtaining video data; and utilizing a machine learning model to generate a predicted probability vector of an object from the video data; determining a behavioral class of the object based on the predicted probability vector; on the computer, The machine learning model is generating an object prediction probability vector, a spatial token prediction probability vector, and a temporal token prediction probability vector from the training video data using the machine learning model to be trained; generating spatially trained predicted class vectors from image frames of the training video data utilizing a spatially trained model; generating a temporally-supervised predicted class vector from the flow frames of the training video data utilizing a temporally-supervised model; training the machine learning model of the training target based on a first error between a training action label corresponding to the training video data and the prediction probability vector, a second error between the spatial token prediction probability vector and the spatial teacher prediction class vector, and a third error between the temporal token prediction probability vector and the temporal teacher prediction class vector; A program generated by running
[0053] Although the examples of the present disclosure have been described in detail above, the present disclosure is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present disclosure described in the claims. [Industrial Applicability]
[0054] The present disclosure is useful for an apparatus and method for training a machine learning model for processing video data. [Explanation of symbols]
[0055] 20 Training target motion decision model 30 Training Data DB 40 Spatial Teacher Model 50-Hour Teacher Model 60 trained action decision models 100 training equipment 110 Training object model processing unit 120 Spatial teacher model processing unit 130 hours teacher model processing section 140 Training Department 200 Motion determination device 210 Acquisition Department 220 Action Decision Unit
Claims
1. A machine learning model to be trained generates predicted probability vectors for objects, spatial tokens, and time tokens from training video data using the target machine learning model, and a training model processing unit. A spatial teacher model processing unit that generates spatial teacher prediction class vectors from image frames of the training video data using a spatial teacher model, A time-teaching model processing unit that generates a time-teaching prediction class vector from the flow frame of the training video data using a time-teaching model, A training unit that trains the machine learning model to be trained based on a first error between the training motion label corresponding to the training video data and the predicted probability vector, a second error between the spatial token predicted probability vector and the spatial teacher predicted class vector, and a third error between the time token predicted probability vector and the time teacher predicted class vector. A training device having the following features.
2. The training apparatus according to claim 1, wherein the machine learning model to be trained includes a video transformer.
3. The training apparatus according to claim 2, wherein the video transformer accepts class tokens, spatial tokens, and time tokens.
4. The training device according to claim 1, wherein the training video data consists of image patches of a predetermined number of frames.
5. The training apparatus according to claim 1, wherein the predicted probability vector indicates the behavior class of the object.
6. Using the machine learning model to be trained, generate predicted probability vectors for objects, spatial tokens, and time tokens from training video data. Using a spatial teacher model, generate spatial teacher prediction class vectors from the image frames of the training video data, Using a time-based teacher model, a time-based teacher prediction class vector is generated from the flow frame of the training video data. Training the machine learning model to be trained based on a first error between the training motion labels corresponding to the training video data and the predicted probability vector, a second error between the spatial token predicted probability vector and the spatial teacher predicted class vector, and a third error between the time token predicted probability vector and the time teacher predicted class vector. A training method for a computer to perform this task.
7. Using the machine learning model to be trained, generate predicted probability vectors for objects, spatial tokens, and time tokens from training video data. Using a spatial teacher model, generate spatial teacher prediction class vectors from the image frames of the training video data, Using a time-based teacher model, a time-based teacher prediction class vector is generated from the flow frame of the training video data. Training the machine learning model to be trained based on a first error between the training motion labels corresponding to the training video data and the predicted probability vector, a second error between the spatial token predicted probability vector and the spatial teacher predicted class vector, and a third error between the time token predicted probability vector and the time teacher predicted class vector. A program that causes a computer to execute something.
8. Using the machine learning model to be trained, generate predicted probability vectors for objects, spatial tokens, and time tokens from training video data. Using a spatial teacher model, generate spatial teacher prediction class vectors from the image frames of the training video data, Using a time-based teacher model, a time-based teacher prediction class vector is generated from the flow frame of the training video data. Training the machine learning model to be trained based on a first error between the training motion labels corresponding to the training video data and the predicted probability vector, a second error between the spatial token predicted probability vector and the spatial teacher predicted class vector, and a third error between the time token predicted probability vector and the time teacher predicted class vector. A machine learning model generated by a computer executing a program.