Systems, computer-implemented methods, and programs for vision-based joint and postural motion prediction - Patents.com

Abstract

To provide a computer implemented method, a program, and a system capable of jointly predicting human motion and posture.SOLUTION: A method includes the steps of: extracting each frame of a video frame; estimating a current posture of an object in each frame; determining a joint position of a joint associated with an object in each frame; calculating an optical flow between each pair of successive video frames; and extracts motion characteristics from each video frame based on the optical flow; encoding state information based on the current posture and the motion characteristics; determining a current motion label of each motion characteristic in state information of the first frame; predicting a future motion label of each motion characteristic of a second frame following the first frame based on the current posture, the motion label, and the state information; predicting, based on the current posture and the status information, a future posture of each motion characteristic of the second frame; and modifying the current motion label, the future motion label, and the future posture.SELECTED DRAWING: Figure 7B

Description

[Technical field]

[0001] Aspects of the embodiments relate to methods, systems, and user experiences related to image-based prediction in an environment, and more specifically, to vision-based methods for predicting human motion and posture. [Background technology]

[0002] Predicting human motion and movement trajectories corresponds to the field of predicting what a person will do next and how they will do it, and is important in a wide range of related applications such as assisted living and future collaborative robotic situations.

[0003] Prior art approaches can model or predict human motion and human posture independently. As described herein, human motion refers to what people do or cause, such as walking, falling, and opening a door. Human motion is closely related to daily life and related activities, which include a series of actions. Each action can affect a person's state, positively or negatively, independently or jointly. Analyzing human motion can enable machines to understand the world better. Thus, it is expected that human-oriented intelligence can better serve humans, such as preventing harmful actions.

[0004] Some related art systems focus on human action recognition, including recognition of human actions in appropriately trimmed video clips, and can provide action labels after analyzing the entire video clip. These related art systems can also consider temporal action localization, including localizing specific meaningful actions in a timeline after analyzing an untrimmed video. However, in the aforementioned related art examples, when the system provides an action label after the fact, the action has already been completed. That is, an injury may have already occurred when the action analysis is completed.

[0005] One of the main problems with related art systems is that the predicted pose converges quickly to the mean pose, making the system perform worse than a zero-velocity predictor (i.e., simply repeating the last observation). Therefore, existing systems are not able to jointly predict human motion and pose. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] FARHA, YA, et al., When Will You do What? - Anticipating Temporal Occurrences of Activities, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 5343-5352. [Non-Patent Document 2] CARREIRA, J., et al., Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017,10 pgs. [Non-Patent Document 3] HAN,T.,et al., Human Action Forecasting by Learning Task Grammars, arXiv:1709.06391vl,September 19, 2017, 8 pgs. [Non-Patent Document 4] JAIN, M., et al., Action Localization with Tubelets from Motion, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 740-747. [Non-Patent Document 5] KE, Q., et al., Time-Conditioned Action Anticipation in One Shot, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 9925-9934. [Non-Patent Document 6] KOPPULA, HS, et al., Anticipating Human Activities Using Object Affordances for Reactive Robotic Response, IEEE Tranactions on Pattern Analysis and Machine Intelligence, 2015, 38(1),pp.14-29. [Non-Patent Document 7] Ll, Y., et al., Online Human Action Detection Using Joint Classification-Regression Recurrent Neural Networks, European Conference on Computer Vision, 2016, pp. 203-220. [Non-Patent Document 8] MA., S., et al., Learning Activity Progression on LSTMs for Activity Detection and Early Detection, Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1942-1950.

Non-Patent Document 9

Non-Patent Document 10

Non-Patent Document 11

Non-Patent Document 12

[0007] Thus, there is an unmet need in the art for a system that can predict future movements and the motion trajectories of those movements. The disclosed technique aims to provide a computer-implemented method, program, and system capable of collaboratively predicting human motion and posture. [Means for solving the problem]

[0008] According to one aspect of an embodiment, a computer-implemented method is provided that includes extracting each frame of a video sequence at a single frame rate, the video sequence comprising a series of video frames, estimating a current pose of an object in each video frame and determining joint positions of joints associated with the object in each video frame, calculating optical flow between each pair of successive video frames of each time step of the video sequence, extracting motion characteristics from each video frame of the video sequence based on the optical flow, and encoding state information based on the current pose and the motion characteristics. A second aspect of the present invention relates to a method for determining a current action label for each motion feature included in the state information of a first frame of the series of video frames, the series of video frames being in red-green-blue (RGB) format, and further including: determining, by a decoder, a future action label for each motion feature in a second frame of the series of video frames following the first frame based on the current pose, action label, and state information; predicting, by a decoder, a future pose for each motion feature in the second frame based on the current pose and the state information; and adjusting the current action label, the future action label, and the future pose based on a loss function. A third aspect is the method of the second aspect, further comprising predicting at least one future pose and future motion of the second frame based on the model and the video sequence. A fourth aspect is the first aspect, wherein the encoder includes one or more processors. A fifth aspect is the second aspect, wherein the decoder includes one or more processors. A sixth aspect is the fourth aspect, wherein the encoder and the decoder are implemented using a recurrent neural network implementing one or more gated recurrent networks. A seventh aspect is the fifth aspect, wherein the encoder and the decoder are implemented using a recurrent neural network implementing one or more gated recurrent networks. In an eighth aspect, in the first aspect, the calculating the optical flow forms a two-channel flow frame, each channel including displacement in an x-axis and a y-axis, respectively. The ninth aspect is the first aspect, wherein the frame rate is 24 fps to 60 fps. A tenth aspect is the first aspect, wherein each joint position of the one or more joint positions includes a pair of two-dimensional coordinates (eg, X, Y) or 3D coordinates (eg, X, Y, Z). The program of an eleventh aspect causes a processor to perform processes including extracting, by an encoder, each frame of a video sequence at a single frame rate, the video sequence including a series of video frames; estimating a current pose of one or more objects in each video frame and determining joint positions of joints associated with the objects in each video frame of the video sequence; calculating an optical flow between each pair of successive video frames for each time step of the video sequence; extracting motion characteristics from each video frame of the video sequence based on the optical flow; and encoding, by an encoder, state information based on the current pose and the motion characteristics. A twelfth aspect of the invention relates to the eleventh aspect of the invention, further comprising: determining a current action label for each motion feature included in the state information for a first frame of the series of video frames; predicting, by a decoder, a future action label for each motion feature in a second frame of the series of video frames following the first frame based on the current pose, action label, and state information; predicting, by a decoder, a future pose for each motion feature in the second frame based on the current pose and the state information; and revising the current action label, the future action label, and the future pose based on a loss function. A thirteenth aspect is the method of the twelfth aspect, further comprising predicting at least one future pose and future motion of the second frame based on the model and the video sequence. A fourteenth aspect is the eleventh aspect, wherein the encoder includes one or more processors, and the encoder and the decoder are implemented using a recurrent neural network implementing one or more gated recurrent networks. A fifteenth aspect is the twelfth aspect, wherein the decoder includes one or more processors, and the encoder and decoder are implemented using a recurrent neural network implementing one or more gated recurrent networks. A sixteenth aspect of the present invention is the eleventh aspect, wherein the calculating the optical flow forms a two-channel flow frame, each channel including displacement in an x-axis and a y-axis, respectively. The seventeenth aspect is the eleventh aspect, wherein the frame rate is 24 fps to 60 fps. An eighteenth aspect is the eleventh aspect, wherein each joint position of the one or more joint positions includes a pair of two-dimensional coordinates (eg, X, Y) or 3D coordinates (eg, X, Y, Z). A system of a 19th aspect is a system for vision-based joint action and posture motion prediction, comprising a processor and a memory device, the system including an encoder configured to extract each frame of a video sequence at a single frame rate, the video sequence including a series of video frames, the encoder estimating a current pose of one or more objects in each video frame and determining joint positions of one or more joints associated with one or more objects in each video frame of the video sequence, the processor calculating an optical flow between each pair of successive video frames for each time step of the video sequence, the processor extracting motion characteristics from each video frame of the video sequence based on the optical flow, and an encoder encoding state information based on the current pose and the motion characteristics. A twentieth aspect is directed to a method of implementing the present invention in accordance with the nineteenth aspect, wherein the processor is configured to determine a current action label for each motion feature included in the state information for a first frame of the series of video frames; predicting, by a decoder, a future action label for each motion feature in a second frame of the series of video frames following the first frame based on the current pose, action label, and the state information; predicting, by a decoder, a future pose for each motion feature in the second frame based on the current pose and the state information; revising the current action label, the future action label, and the future pose based on a loss function; and predicting at least one future pose and future motion for the second frame based on a model and the video sequence. A twenty-first aspect is the sixteenth aspect, wherein the encoder includes one or more processors, and the encoder and decoder are implemented using a recurrent neural network implementing one or more gated recurrent networks. A 22nd aspect is the 20th aspect, wherein the decoder includes one or more processors, and the encoder and decoder are implemented using a recurrent neural network implementing one or more gated recurrent networks. A twenty-third aspect is the nineteenth aspect, wherein the processor that calculates the optical flow forms a two-channel flow frame, each channel including displacement in the x-axis and y-axis, respectively. A twenty-fourth aspect is the nineteenth aspect, wherein each joint position of the one or more joint positions includes a pair of two-dimensional coordinates (eg, X, Y) or 3D coordinates (eg, X, Y, Z).

[0009] An embodiment may also include a non-transitory computer-readable medium having a storage device and a processor, the processor capable of executing instructions related to image-based prediction in an environment, and more specifically, a vision-based method for predicting human motion and posture.

[0010] Details of embodiments of the present disclosure, both as to their structure and operation, can be gleaned in part from study of the accompanying drawings, in which like reference characters refer to like parts, and in which: [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a flow diagram of one embodiment of a method for predicting future movements and postures from current and previous observations.

[0012] [Diagram 2] FIG. 2 is a functional block diagram of one embodiment of a system for predicting joint and postural motion according to the present disclosure.

[0013] [Diagram 3] FIG. 3 is a flow diagram of one embodiment of a method for estimating motion characteristics and human pose from video according to this disclosure.

[0014] [Figure 4] FIG. 4 is a flow diagram of one embodiment of a method for multi-task training of sequence-to-sequence models in accordance with the present disclosure.

[0015] [Diagram 5] FIG. 5 is a flow diagram of one embodiment of a method for providing inference using a trained model according to the present disclosure.

[0016] [Figure 6] FIG. 6 is a graphical illustration of motion prediction accuracy for different prediction lengths using test data.

[0017] [Figure 7A] FIG. 7A is a flow diagram of one embodiment of a method for vision-based joint and posture motion prediction according to the present disclosure. [Figure 7B] FIG. 7B is a flow diagram of one embodiment of a method for vision-based joint action and posture motion prediction according to the present disclosure.

[0018] [Figure 8] FIG. 8 is a functional block diagram of one embodiment of a computing environment according to the present disclosure.

[0019] [Figure 9] FIG. 9 is a functional block diagram of an exemplary operating environment according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The following detailed description provides further details of the figures and examples of the present application. Reference numbers and descriptions of overlapping elements between the figures have been omitted for clarity. The terms used throughout this specification are provided by way of example and are not intended to be limiting.

[0021] Aspects of the embodiments are directed to deep learning methods for predicting or otherwise anticipating human motion based on observation of previous motions. Predicting human motion and motion trajectories is important in a wide range of applications, such as AI assisted living and human-robot interaction. Early prediction of motion and posture can benefit a variety of applications, including AI assisted medical care, for example, to prevent patient falls. If an AI system can predict falls in a hospital or assisted living facility, falls can be prevented to reduce injuries, thus reducing medical costs and improving quality of life.

[0022] 1 is a flow diagram of one embodiment of a method for predicting future actions and postures from current and previous observations. Humans can naturally anticipate actions and the position of an actor before the action is performed based on observation of the current action or posture and their individual past experience in similar situations. Similarly, the present disclosure provides systems and methods for jointly anticipating or otherwise predicting one or more human action categories and skeletal joints (postures) in future video frames based on previous video frames that have been previously analyzed.

[0023] 1 can extract motion features 110 and poses 120 from one or more video streams 130. The method 100 can be modeled as a multi-task learning problem including action category classification and skeleton-joint regression, both for future frames related to the video stream 130. Given the video stream 130, the motion features 110 (optical flow) and the poses 120 of a subject can be extracted for every frame of the video stream 130.

[0024] As used herein, a skeletal pose can utilize a number of (e.g., 14 or 25, where 14 represents the total number of joints at the top of the body and 25 represents the total number of joints in the body. Other numbers of joints can be used instead without departing from the scope of the present invention) joint points to represent a given position on the human body. These skeletal poses or joint points can include, describe, or be associated with joints of the body, such as the hands, wrists, elbows, shoulders, ankles, knees, hips, etc. As described herein, by way of example and not by way of limitation, using RGB (red-green-blue) images from a video stream provides a low-dimensional object that is easy to model dynamically. For example and not by way of limitation, using a skeleton (e.g., size 25 joints x 3) can represent a subject more compactly than using raw pixels (e.g., size 480X640) from the RGB images.

[0025] At every time step t in the video stream 130, the motion characteristics 110 and the poses 120 can be combined to form a deep model 140, which can include a sequence transformation model. The method 100 can then be used to progressively predict future sequences of actions and poses 150 based on the deep model 140.

[0026] 2 is a functional block diagram of an embodiment of a system for predicting joint and posture movements according to the present disclosure. A system 200 can be used to perform one or more aspects of the method for predicting joint and posture movements from a video stream (e.g., the video stream 130). The system 200 can include an encoder 210 communicatively coupled to a decoder 220. The system 200 can receive and process the (e.g., RGB) video stream 130 to jointly predict future motion and posture sequences. The system 200 can incorporate a multi-task encoder-decoder recurrent network structure.

[0027] At each time step t of the video stream 130, the encoder 210 can encode multi-modality characteristics from past video stream information and recognize current actions seen in the video stream 130. The encoded information serves to initialize the decoder 220 with relevant information to perform predictions incrementally.

[0028] In contrast to systems directed at motion recognition and detection that make decisions based on entire videos of past events, motion prediction as described herein can involve predicting or anticipating immediate future movements or postures using observations available at the moment or occurring in real time, which can be accomplished using the sequential encoder-decoder recurrent network structure of the system 200 as shown.

[0029] The encoder 210 may include a recurrent neural network (RNN), such as a gated recurrent network (GRU) or a long short-term memory (LSTM). The encoder 210 may have one or more RNNs, denoted as GRUs 212a, 212b, 212c. The encoder 210 may encode multimodal features from past video sequences (e.g., the video stream 130) and determine a current action type based on the past video sequences. The current action type may include, for example, a description of an action seen in the video stream 130. As shown in the video stream 130 of FIG. 1, for example, a user stands next to a microwave oven, then reaches over and opens the microwave oven. The current state information may describe aspects of such a scene.

[0030] The decoder 220 may receive current state information 230 of the encoder 210. The current state information 230, also referred to herein as state 230, may be encoded by the encoder 210 as a hidden state or otherwise obscured by a hash function, compression, or other applicable process. Thus, the state 230 may also be referred to herein as a "hidden state" or an "encoder state."

[0031] The decoder 220 can receive three pieces of information: the state information 230, the predicted current motion 200, and the detected pose as a starting point: a “loss function” is shown above 240. In FIG. 2, “at” means motion type, which corresponds to “yt” in the loss function. The decoder 220 can then implement an RNN 222 (shown as GRU 222a, GRU 222b) to progressively generate future sequences of motion and pose pairs 240 (shown as 240a, 240b) based on a model (e.g., the deep model 140) created by the encoded past frames and the recognized current motion.

[0032] At time t, a sequence of past video frames V (e.g., from video stream 130) 0:t Given Y, the system 200 can be used to predict a sequence of action labels and corresponding poses at future time steps T. This can be expressed as (Y t:t+T’ P t:t+T ) can be shown as Y t denotes the behavior category at time step t and is encoded as a one-hot vector, where 1 is the true class and 0 is all other classes. P t is a two-dimensional (2D) pose at time t defined by the (x, y) coordinates of a set of major body joints, or alternatively, a 3D pose indicated by the (x, y, z) coordinates. Major body joints in this sense can include, for example, nose, neck, shoulders, elbows, wrists, ankles, knees, eyes, ears, toes, heels, elbows, etc.

[0033] 3 is a flow diagram of an embodiment of a method for estimating motion characteristics and human pose from video according to the present disclosure. Method 300 may be performed, for example, by system 200. Method 300 leverages advances in deep learning representations to estimate the motion characteristics of an RGB image f rgb , optical flow f motion , and human posture f pose Features can be extracted from multiple modalities, including the RGB modality, which represents visual appearance.

[0034] The latter two modalities, i.e., optical flow f motion and human posture f pose Let f represent short-term and long-term motion information, respectively, associated with a sequence of video frames, e.g., from video stream 130. To capture short-term temporal relationships, method 300 may be implemented with a stack of L video frames (e.g., from video stream 130) as input, as opposed to a single image. rgb , f motion , and f pose The following describes this.

[0035] RGB feature extraction. Frame-level features are extracted from a ResNet-18 convolutional neural network (CNN). A standard ResNet-18 CNN takes in an RGB image and has 3 input channels. To accommodate a stack of L frames as input, the input channels of the ResNet-18 CNN network are adjusted to L × 3. The final RGB representation f rgb is the 512-d vector extracted from the last average pooling layer. In other embodiments, in addition to ResNet-18, alternative off-the-shelf CNN networks such as VGG, Densenet, or different layers such as ResNet-34 can be utilized.

[0036] Optical Flow Feature Extraction. A deep learning based optical flow estimator (such as FlowNet2.0) can be utilized to compute the optical flow between two consecutive frames of the video stream 130. For each time step t, such a system can generate a two-channel flow frame, where each channel contains displacements on the x-axis and y-axis, respectively. Similar to the RGB modality, we use the L-stacked optical flow frames as input to a ResNet-18 CNN, changing the number of input channels to L×2. The final representation f motion is a 512-d vector.

[0037] Pose extraction. Pose has recently attracted a lot of interest due to the success of real-time body point detection from a single image. For the posture modality, an off-the-shelf pose estimator (such as OpenPose) can be utilized to determine the joint positions of key body points from each frame. Since each joint position can be represented by a pair of 2D coordinates, each pose is encoded by a 1D vector with num_joints × 2elements. The final pose representation f poseis the concatenation of poses from L frames, with a total size of 1×L×num_joints×2elements. Given a video (e.g., the video stream 130), the system can first extract all the frames at a fixed sampling rate (e.g., 30 fps). At each time step t, RGB, optical flow, and pose features are used separately, or f t These are concatenated into a final multimodal feature representation denoted as

[0038] At block 310, the system 200 can extract all video frames from the video stream 130 at a fixed sampling rate (e.g., 30 frames per second (fps)), or can extract frames that allow for the depiction of smooth motion. Fixing the sampling rate across all video frames is important to maintain consistent comparisons for predicting the future motion and posture frame by frame. If the sampling or frame rate is inconsistent, the speed of motion may be ambiguous. Thus, with an inconsistent frame rate, it is difficult for the model to learn the dynamics of motion. As used herein, 30 fps is not limiting of the disclosure. Other sampling rates below or above 30 fps are possible, including intervening frame rate values, such as, for example, 40, 45, 50, 60 fps or more. Common video frame rates include, but are not limited to, 24, 29.97, 30, 59.94, and 60 fps. For processing purposes, the rate may be normalized to 29.97 or 30 fps (24 fps is typically seen in movies), therefore examples may include frame rates from 24 fps to 60 fps.

[0039] The extracted video frames are then used to compute the features in two different domains: skeletal posture features and kinematic features of the subject in the video.

[0040] In block 330, pose estimation can model skeletal pose characteristics 332. During the pose estimation in block 330, one or more human joint positions in each video frame can be estimated using a prefabricated pose estimation deep model. Each joint position in the pose characteristics 332 can be represented by two-dimensional (2D) coordinates or 3D coordinates. Thus, each pose can be represented by a 1×num joints *2It can be written as a vector.

[0041] In block 320, the optical flow estimation can be used to model motion characteristics. In block 320, or in the motion branch of the method 300, an optical flow estimation deep model can be implemented to calculate the optical flow 322 between each pair of consecutive video frames (e.g., of the video stream 130). Note that the width and height of the optical flow is the same as an RGB frame associated with the video stream 130, but the optical flow 322 has two channels representing displacement values ​​on each of the x and y axes.

[0042] In block 340, a pre-trained feature extractor can generate motion features 342 for each video frame. The feature extractor in block 340 can receive as input a stacked optical flow, which is a concatenation of the optical flow of the current time and the previous num_stack time steps. The output of block 340 can include short-term motion features 342 with dimensions of motion-dim. Early fusion is used for each frame to concatenate the motion features and the posture features as the final features for subsequent processes.

[0043] The method 300 can also perform late fusion to efficiently represent the combined features of both modalities (e.g., pose and optical flow / motion). Unlike early fusion, which utilizes a single encoder (e.g., the encoder 210 with a single RNN) to capture temporal patterns from the concatenated features, late fusion can include different encoders for each modality before the concatenation, resulting in separate hidden states for motion and pose. In some examples, the encoder (e.g., the encoder 210) can be an RNN, such as a GRU cell, similar to that described above in connection with FIG. 2.

[0044] Compared to early fusion, late fusion introduces an additional encoder structure, which increases the model complexity. However, since movement features are dense descriptors of local variance, and posture can be considered as a higher level of abstraction of continuous dynamics, the partitioned latent space of late fusion is more flexible than early fusion for modeling such different types of time dependencies.

[0045] Therefore, to further improve the decoding process of the method 300, the (first or primary) GRUs 222 and 212 can be augmented by secondary GRU cells during the training phase to form a bidirectional structure. In addition to future pose prediction via the first GRU 222, the second GRU cell 222 is initialized with the same hidden state, but aims to reconstruct previous inputs in the reverse order. The bidirectional GRU captures the context better when performing inputs from past to future and future to past, so that information from both past and future is preserved. More specifically, at each time step, the reconstructed pose of the current frame is provided to reconstruct the input pose from the last frame. This forces the hidden state (e.g., state 230) to maintain sufficient and necessary history information about pose and motion that may be important to recover all past frames. Thus, even if such an augmented structure is reduced to an initial single GRU decoder during the testing phase after training and the secondary GRU is removed, the hidden state may be more capable of capturing long-term patterns.

[0046] 4 is a flow diagram 400 of one embodiment of a method for multi-task training of a sequence transfer model according to the present disclosure. A sequence transfer model can address the prediction problem described above. The method of the present disclosure can include an encoder 410 configured to encode past information and a decoder 430 configured to predict future actions and postures (e.g., of an individual in the video stream 130). The encoder 410 can implement an RNN (e.g., a GRU) to encode multi-modal features from past video sequences (e.g., the video stream 130) and determine a current action type.

[0047] The decoder 430 is set to the same value as the current hidden state 412 of the encoder 410. By feeding the predicted current motion and detected pose as a starting point, the decoder 430 can generate future motion sequences and pose pairs incrementally (e.g., using a GRU). Both the encoder 410 and the decoder 430 can be RNN-based models. At every time step, the encoder 410 can receive as input the final characteristics of the current frame and all previous frames. Since RNNs can model dependencies between different elements in a sequence, the encoder 410 encodes all observed frames into a hidden state, which is the long-term information of the sequence.

[0048] Given the encoded hidden state 412 from the encoder 410, the decoder 430 can progressively output both action labels 432 and poses 434. Since the input of the decoder 430 and the output of the encoder 410 must be the same, the input to the decoder 430 can also be action labels 432 and poses 434, except that there are no action labels at any time step.

[0049] In some implementations, the encoder 410 can also recognize the motion category of the current frame. The motion label and the calculated pose at the current time step can then be provided to the decoder 430 as input for the first time step. The decoder 430 then outputs the prediction result for the first time step, i.e., the motion label and pose. For subsequent time steps, the decoder 430 takes as input the output of the previous time steps, until a maximum time step is reached. As a result, the system 400 can provide predicted future motion labels and poses for every time step.

[0050] The method 400 may include three tasks, labeled as Task 1, Task 2, and Task 3 in FIG. 4. These tasks include recognition of the current action label (Task 1), prediction of the future action label (Task 2), and prediction of the future pose (Task 3). Although these are separate tasks, they are not completely independent of each other. For example, knowing the current action label 436 helps to predict the future action label evolved from the current action label. On the one hand, the future action label serves as an additional validation for predicting the future pose. In this way, the pose prediction module can model only the poses that belong to a specific action, which reduces the burden on this module and makes learning easier. On the other hand, apart from the action label of the current time step, the future poses can serve as additional information for the prediction of the future poses. In summary, the three tasks are closely related to each other and are therefore natural candidates for joint learning.

[0051] As shown in Figure 4, the proposed model consists of one weight-sharing backbone network and three heads to perform these tasks. By using multi-task learning, the proposed model is trained jointly. The multi-task encoder-decoder recurrent network is jointly optimized end-to-end by a loss function 450. The loss function can be expressed as the following Equation (1):

number

[0052] Here, L cls is the sum of the motion classification losses for each frame from time t to t+T. regis the sum of L1 smooth regression losses of pose joint positions for each frame from time t to t+T. λ is a scalar parameter that balances the two types of losses. Note that motion classification is performed by a motion classifier at 420 to define the current motion label 436. The system 400 can of course handle different prediction lengths based on the incorporation of the RNN model in the decoder 430. As used herein, prediction length is the amount of time into the future for which the system 400 can provide motion and pose predictions.

[0053] Given the coded hidden state 412 from the encoder 410, the decoder 430 can progressively output both the action label and the pose by itself. In the first prediction time step, the decoder 430 also receives as input the recognized action label and the pose at the last seen time step. In the subsequent time steps, the decoder 430 always receives the output of the previous time step as the input of the current time step, i.e., the hidden state, the action label, and the pose. Therefore, the prediction length is set as a variable parameter of the system. It can be predicted as long as it is set during inference.

[0054] 5 is a flow diagram of one embodiment of a method for providing inference using a trained model according to the present disclosure. In block 505 (similar to block 405), motion characteristics 110 and pose 120 are extracted from a video stream 130. In block 510, the extracted motion characteristics 110 and pose 120 may be encoded (e.g., by the encoder 210 or 510) as state information 230.

[0055] Based on the trained model generated by the method 400 , the action classifier 520 may perform action classification in block 520 to define a current action label 536 .

[0056] In block 530, the decoder 220 can decode the state 230 and predict a future label 532 and a future pose 534 based on the model from the method 400. The current action label 436, the future action label 432, and the future pose 434 are distinct from the current action label 536, the future action label 532, and the future pose 534 for situations in which the method 500 is used for motion and pose prediction as opposed to training the model.

[0057] 6 is a graphical illustration of motion prediction accuracy for different prediction lengths using test data. The table methods 400 and 500 can be evaluated based on test data. For example, a simple procedure such as assembling furniture can be used as a surrogate for another, possibly more complex task. An exemplary dataset includes a set of videos (e.g., the video stream 130) of an individual assembling a small piece of furniture.

[0058] The exemplary dataset has 101 short videos, each approximately 2-4 minutes long, shot at 30 frames per second. An example scenario includes 14 actors in the videos, with sequences from 11 actors performed for training and validation using the method 400, and testing performed using the method 500 in a balanced manner. Half of the sequences show assembly on the floor, while the other half show assembly on a workbench. Exemplary action labels for the videos include, for example, four "attach legs" actions (one for each leg), four "remove legs" actions, a "pick up legs" action, a "flip table" action, a "spin in", a "spin out" action, and a null action for frames that were not or could not be labeled by the method 400.

[0059] The system 200 can be evaluated based on two aspects: motion prediction accuracy and performance of future pose prediction. In the first aspect, the motion prediction accuracy can be compared with the motion prediction accuracy of state-of-the-art or off-the-shelf models and different prediction lengths. The two control methods selected are 1) LSTM using a two-stream prediction system and 2) LSTM using improved dense trajectory (iDT).

[0060] As shown in Table 1 below, the next first prediction (Acc 0 ) accuracy (e.g., sequence-to-sequence or seq2seq) is over 11.1% higher than both controls. Furthermore, the models implemented by the methods 400 and 500 can predict multiple consecutive time steps or sequences, whereas the other two prediction methods can only predict the next one time step.

[0061] In the disclosed test, the method 500 determines the next fifth prediction (Acc4), or Acc 0 We were able to achieve an accuracy of 74.3% in the four additional steps past Acc., which is only slightly lower than the accuracy of the first prediction. The reduction in prediction accuracy is due to the reduction in the amount of information about the future posture / movement, and therefore the reduction in prediction accuracy. As shown in Figure 4, the relationship between the motion prediction accuracy and the prediction length is 0 ) to 61 for every 5th time step. As can be seen, the accuracy decreases as the prediction length increases, which illustrates the point above.

[0062] [Table 1]

[0063] 7A and 7B are flow diagrams of an embodiment of a method for vision-based joint motion and posture motion prediction according to the present disclosure. Method 700 can be implemented by one or more processors in a computing environment (e.g., computing environment 1000 described below). In some embodiments, as shown in FIG. 7A, the method 700 can be a combination of multiple sub-processes including estimating current posture and joint motion at 701, training (e.g., learning) a deep learning model at 702, and predicting future motion and posture based on the model at 703.

[0064] 7B shows further details of the estimation 701, learning 702, and prediction 703 according to the embodiment of the method 700. Elements 705-725 represent additional details related to the estimation 701, elements 730-745 represent additional details related to the learning 702, and element 750 represents additional details related to the prediction 703.

[0065] In block 705, the system 200 may extract each frame of a video sequence (e.g., the video sequence 130) at a single frame rate. The single frame rate may be constant across all video frames of the video sequence. The video frames may be formatted, for example, as a Red-Green-Blue (RGB) video format. However, this embodiment is not so limited. For example, and without limitation, the video frames may be extracted as RGB, but the video frames may also be represented as YUV in one or more other well-known video formats.

[0066] At block 710, the system 200 may estimate a current pose of one or more objects in each video frame. The system 200 may further determine joint positions of one or more joints associated with one or more objects in each video frame of the video sequence. The joint position determination and pose estimation may occur simultaneously, in parallel, or sequentially, in any order.

[0067] In block 715, the system 200 may calculate optical flow between each pair of successive video frames for each time step of the video sequence to form a two-channel flow frame, with each channel containing displacement in the x-axis and y-axis, respectively.

[0068] At block 720, the system 200 can extract motion characteristics from each video frame of the video sequence based on the optical flow.

[0069] In block 725, the system 200 may encode state information based on the current pose and the motion characteristics.

[0070] In block 730, the system 200 may determine a current action label for each movement characteristic included in the state information for a first frame of a sequence of video frames.

[0071] In block 735, the system 200 may predict a future action label for each motion feature in a second frame of the series of video frames that follows the first frame based on the current action label and state information.

[0072] In block 740, the system 200 may predict a future pose for each movement characteristic in the second frame based on the current pose and the state information.

[0073] In block 745, the system 200 may refine the current action labels, the future action labels, and the future poses based on a loss function.

[0074] In block 750, the system 200 can predict at least one future pose and future motion of the second frame based on the model and the video sequence.

[0075] For example, embodiments of the method 700 and other aspects disclosed herein may be applicable to a robot or robotic system. In some examples, a robotic system may implement the method 700 in various medical or surgical procedures to improve the speed of various procedures or speed up training by anticipating the needs of a human operator. In at least one example, a robotic assistant in a medical application may anticipate when a patient is about to fall based on a recently captured video of the patient / subject, or pass surgical tools to a surgeon during surgery based on the actions of the surgical team.

[0076] According to another exemplary application, activities performed by a surgeon during a surgical procedure may be subject to posture and motion prediction. Thus, when the surgeon begins to engage in postures and motions that correspond to potential risks, such as placing an incision or tool in an improper location, angle, setting, etc., an embodiment may provide a predictive output that gives the surgeon an opportunity to take corrective action and avoid harm to the patient.

[0077] In another exemplary application, the method 700 may be implemented in a retail environment for loss or theft prevention to anticipate motions of subjects captured on video (e.g., the video stream 130). For example, but not by way of limitation, motion prediction may be used to perform posture motion predictions for individuals entering a retail environment such that postures and motions associated with shoplifting activities may be detected prior to theft. Similarly, postures and motions associated with in-store activities that may injure a user or damage merchandise, such as certain postures of individuals coming into contact with products (e.g., individuals testing or holding products that are too large, climbing on shelves to reach products that are higher on the shelves, etc.). Thus, the embodiment may be used to reduce the risk of harm to individuals in a retail environment, such as customers or employees, as well as identify and provide opportunities to intervene in a manner that reduces such risks and loss or theft behaviors.

[0078] Further, the embodiments can be utilized in consumer-facing settings such as the home. Appliances, robotics, communication devices, etc. can perform actions based on posture and motion predictions. For example, but not by way of limitation, cooking-related activities can be predicted based on a series of postures and motions, and appliances in the kitchen can be set to one or more modes, such as pre-heat defrost, to provide additional convenience to the consumer. In other embodiments, the consumer can be provided with an assistant that can identify when the consumer is performing tasks in the wrong order (e.g., the consumer is cooking and forgets to add ingredients) and provide corrective action to avoid injury, errors, or other harm. Home assistant products can sense such activities through audio, video, or other sensors or IoT devices, and provide corrective feedback or suggestions to the user in a variety of consumer and home use situations.

[0079] The embodiments may also be used in industrial environments such as manufacturing, assembly, etc., where poses and motions associated with manufacturing defects may be used to predict defects before they occur, such as, for example, but not limited to, a total quality control approach to quality control. In such embodiments, the poses or motions of an individual or robot may detect poses or motions associated with corresponding defects, such as not fully installing a part, skipping an assembly step, or using an improper tool or an improper number of parts. In response, corrective actions may be suggested, additional training may be provided, or other recommendations or actions may be provided based on the output predictions related to the poses and motions.

[0080] 8 is a functional block diagram of one embodiment of a computing environment according to the present disclosure. Computing environment 800 having an exemplary computer device 805 suitable for use in some embodiments. Computing device 805 in computing environment 800 can include one or more processing units, cores, or processors 810, memory 815 (e.g., RAM, ROM, etc.), internal storage 820 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interfaces 825, any of which may be coupled to communicate information over a communication mechanism or bus 830 or may be incorporated into computing device 805. The environment 800 can support operations related to the system 200 and the methods 80, 300, and 400.

[0081] According to this embodiment, processing related to neural activity may be performed on a processor 810 that is a central processing unit (CPU). Alternatively, other processors may be used instead without departing from the concept of the present invention. For example, but not limited to, a graphics processing unit (GPU) and / or a neural processing unit (NPU) may be used in place of or in combination with the CPU to perform the processing of the exemplary implementations described above. In various implementations or embodiments of the disclosed systems and methods, the RNN may be implemented using one or more GRUs and / or LSTMs to perform the functions and achieve the objectives described herein.

[0082] The computing device 805 may be communicatively coupled to an input / interface 835 and an output device / interface 840. One or both of the input / interface 835 and the output device / interface 840 may be a wired or wireless interface and may be detachable. The input / interface 835 may include any device, component, sensor, or interface, physical or virtual, that can be used to provide input (e.g., buttons, touch screen interfaces, keyboards, pointing / cursor control devices, microphones, cameras, Braille, motion sensors, optical readers, etc.).

[0083] Output devices / interfaces 840 may include displays, televisions, monitors, printers, speakers, Braille, etc. In some embodiments, input / interfaces 835 (e.g., user interfaces) and output devices / interfaces 840 may be embedded in or physically coupled to computing device 805. In other embodiments, other computing devices may function as or provide the functionality of input / interfaces 835 and output devices / interfaces 840 for computing device 805.

[0084] Examples of computing devices 805 include, but are not limited to, highly mobile devices (e.g., smart phones, devices in vehicles and other machines, devices carried by humans and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed to be portable (e.g., desktop computers, server devices, other computers, information kiosks, televisions, radios with and / or coupled to one or more processors embedded therein, etc.).

[0085] Computing device 805 may be communicatively coupled (e.g., via I / O interface 825) to external storage 845 and network 850 for communicating with any number of network components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 805 or any connected computing device may function, provide services, or be referred to as a server, client, thin server, general purpose machine, special purpose machine, or another label. By way of example and not limitation, network 850 may include a blockchain network and / or a cloud.

[0086] I / O interface 825 may include, without limitation, wired and / or wireless interfaces using any communications or I / O protocol or standard (e.g., Ethernet, 802.11xs, Universal System Bus, WiMAX, modem, cellular network protocols, etc.) for communicating information to and from at least all connected components, devices and networks in computing environment 800. Network 850 may be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0087] The computing device 805 may use and / or communicate using computer usable or computer readable media, including transitory media and non-transitory media. Transitory media include transmission media (e.g., metallic cables, optical fibers), signals, carrier waves, etc. Non-transitory media include magnetic media (e.g., disks and tapes), optical media (e.g., CD-ROMs, digital video disks, Blu-ray® disks), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage devices), and other non-volatile storage or memory.

[0088] The computing device 805 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some exemplary computing environments. The computer-executable instructions can be retrieved from a transitory medium and stored in and retrieved from a non-transitory medium. The executable instructions can be from one or more of any programming language, scripting language, and machine language (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0089] The processor(s) 810 may run under any operating system (OS) (not shown), in a native or virtual environment. One or more applications may be deployed, including a logic unit 855, an application programming interface (API) unit 860, an input unit 865, an output unit 870, a training unit 875, a feature extraction unit 880, a prediction unit 885, and an inter-unit communication mechanism 895 for different units (e.g., the encoder 110 and the decoder 120) to communicate with each other, with an OS, and with other applications (not shown).

[0090] The training unit 875 may perform functions associated with the encoder 110 and train the model as in the methods 300 and 400. The feature extraction unit 880 may perform functions associated with the method 300, for example. In some embodiments, the feature extraction unit 880 may perform one or more of the frame extraction of block 310, the pose estimation of block 330 (FIG. 3), in addition to the feature extraction of block 340 (FIG. 3). The prediction unit 885 may perform functions associated with the decoder 120, including prediction or anticipation of pose and motion based on the state information 230 in the methods 100, 300, and 400.

[0091] For example, the training unit 875, the feature extraction unit 880, and the prediction unit 885 may perform one or more processes set forth above with respect to the structures described above, in addition to the method 300 and the method 400. The described units and elements may be modified in design, function, configuration, or implementation and are not limited to the descriptions provided.

[0092] In some embodiments, once information or instructions to execute are received by API unit 860, they may be communicated to one or more other units (e.g., logic unit 855, input unit 865, training unit 875, feature extraction unit 880, and prediction unit 885).

[0093] For example, the training unit 875 may receive and process simulated data, historical data, or information from one or more sensors, at least as described above in connection with the method 400. The output of the training unit 875 is provided to the feature extraction unit 880, which performs necessary operations based on, for example, the application of a neural network as described above and illustrated in Figures 1-5. Furthermore, the prediction unit 885 may perform operations based on the outputs of the training unit 875 and the feature extraction unit 880 to minimize reprojection errors and provide an output signal.

[0094] In some examples, the logic unit 855 may be configured to control the flow of information between the units and direct the services provided by the API unit 860, the input unit 865, the training unit 875, the feature extraction unit 880, and the prediction unit 885 in some of the implementations described above. For example, the flow of one or more processes or implementations may be controlled by the logic unit 855 alone or in conjunction with the API unit 860.

[0095] 9 is a functional block diagram of an example operating environment according to the present disclosure. Environment 900 can be suitable for some embodiments disclosed herein. Environment 900 includes devices 905-945, each communicatively connected to at least one other device, e.g., via a network 960 (e.g., by a wired and / or wireless connection). Some devices can be communicatively connected to one or more storage devices 930 and 945.

[0096] An example of one or more of the devices 905-945 may each be the computing device 805 illustrated in Figure 8. The devices 905-945 may include, but are not limited to, a computer 905 (e.g., a laptop computing device) having a monitor and associated webcam, a mobile device 910 (e.g., a smartphone or tablet), a television 915, a device associated with a vehicle 920, a server computer 925, computing devices 935-940, and storage devices 930 and 945, as described above.

[0097] In some implementations, devices 905-920 can be considered user devices associated with a user that can remotely obtain sensed inputs that are used as inputs for the above examples. In this example, one or more of these user devices 905-920 can be associated with one or more sensors, such as cameras, embedded in the user's body, either temporarily or permanently, remotely from the patient care facility, that can sense information as required for this example, as described above.

[0098] While the foregoing examples are provided to illustrate the scope of the invention, they are not intended to be limiting, and other techniques or implementations may be substituted or added without departing from the scope of the invention. For example, but not limited to, imaging techniques other than those disclosed herein may be utilized.

[0099] According to one embodiment, algorithms such as SuperPoint can be used to train image point detection and determination. Additionally, the embodiment can utilize alternative image classification algorithms and / or use other neural network structures (e.g., Siamese networks). Additional approaches incorporate zone class operation expertise, apply processing to two images using techniques such as formation, lighting and illumination, and / or use a single image for depth methods.

[0100] Although this is not required, the embodiments may have various advantages and benefits. For example, and not by way of limitation, the embodiments may be able to operate on small data sets. Additionally, the embodiments may allow for location constraints within a target tissue, such as the colon or lungs. Thus, the surgeon may be able to more accurately pinpoint the location of the scope by anyone using video. Additionally, the embodiments provide a much higher degree of accuracy than related art approaches.

[0101] Although some examples have been shown and described, these examples are provided to convey the subject matter described herein to those skilled in the art. It should be understood that the subject matter described herein may be embodied in various forms, without being limited to the described examples. The subject matter described herein may be embodied without the matters specifically defined or described, or with other or different elements or matters not described. Those skilled in the art will understand that changes can be made to these examples without departing from the subject matter described herein, as defined in the appended claims and their equivalents.

[0102] Aspects of certain non-limiting embodiments of the present disclosure address the above-mentioned characteristics and / or other characteristics not mentioned above, however, aspects of non-limiting embodiments need not address the above-mentioned characteristics, and aspects of non-limiting embodiments of the present disclosure may not address the above-mentioned characteristics.

Claims

1. extracting each frame of a video sequence at a single frame rate, the video sequence comprising a series of video frames; estimating a current pose of an object in each video frame and determining joint positions of joints associated with said object in each video frame; calculating optical flow between each pair of successive video frames of each time step of the video sequence; extracting motion characteristics from each video frame of the video sequence based on the optical flow; and an encoder encoding state information based on a current pose and the motion characteristics; Including, the encoder comprises one or more processors; Computer-implemented method.

2. determining a current action label for each motion characteristic included in the state information for a first frame of the series of video frames, the series of video frames being in a Red-Green-Blue (RGB) format; predicting, by a decoder, a future action label for each motion feature in a second frame of the series of video frames subsequent to the first frame based on the current pose, the action label, and the state information. predicting, by a decoder, a future pose for each motion characteristic in the second frame based on the current pose and the state information; and adjusting the current action label, the future action label, and the future pose based on a loss function; The computer-implemented method of claim 1 , further comprising:

3. The computer-implemented method of claim 2 , further comprising predicting at least one of a future pose and a future motion for the second frame based on a model and the video sequence.

4. The computer-implemented method of claim 2 , wherein the decoder comprises one or more processors.

5. 10. The computer-implemented method of claim 1, wherein the encoder is implemented using a recurrent neural network that implements one or more gated recurrent networks.

6. 5. The computer-implemented method of claim 4, wherein the decoder is implemented using a recurrent neural network that implements one or more gated recurrent networks.

7. The computer-implemented method of claim 1 , wherein the computing the optical flow forms a two-channel flow frame, each channel including displacement in an x-axis and a y-axis, respectively.

8. The computer-implemented method of claim 1 , wherein the frame rate is between 24 fps and 60 fps.

9. The computer-implemented method of claim 1 , wherein each joint position of the one or more joint positions comprises a pair of two-dimensional coordinates (e.g., X, Y) or 3D coordinates (e.g., X, Y, Z).

10. The processor: extracting, by an encoder, each frame of a video sequence at a single frame rate, the video sequence including a series of video frames; estimating a current pose of one or more objects in each video frame and determining joint positions of joints associated with the objects in each video frame of the video sequence; calculating optical flow between each pair of successive video frames of each time step of the video sequence; extracting motion characteristics from each video frame of the video sequence based on the optical flow; and encoding, by the encoder, state information based on the current pose and the motion characteristics; A program for executing a process including the steps of: the encoder-related functions are performed by one or more processors; program.

11. determining a current action label for each motion feature included in the state information for a first frame of the sequence of video frames; predicting, by a decoder, a future action label for each motion feature in a second frame of the series of video frames subsequent to the first frame based on the current pose, the action label, and the state information. predicting, by the decoder, a future pose for each motion characteristic in the second frame based on the current pose and the state information; and revising the current action label, the future action label, and the future pose based on a loss function; The program of claim 10, further comprising:

12. 12. The program of claim 11, further comprising predicting at least one of a future pose and a future motion of the second frame based on the model and the video sequence. 。

13. 11. The program of claim 10, wherein the encoder-related functions are performed using a recurrent neural network implementing one or more gated recurrent networks.

14. 12. The program of claim 11, wherein the decoder-related functions are performed by one or more processors, and the decoder-related functions are performed using a recurrent neural network that implements one or more gated recurrent networks.

15. The method of claim 10 , wherein the calculating the optical flow forms a two-channel flow frame, each channel including displacement in an x-axis and a y-axis, respectively.

16. The program according to claim 10, wherein the frame rate is 24 fps to 60 fps.

17. The method of claim 10 , wherein each joint position of the one or more joint positions comprises a pair of two-dimensional coordinates (e.g., X, Y) or 3D coordinates (e.g., X, Y, Z).

18. 1. A system for vision-based joint and posture motion prediction, comprising a processor and a memory device, an encoder configured to extract frames of a video sequence at a single frame rate, the video sequence comprising a series of video frames; the processor estimating a current pose of one or more objects in each video frame and determining joint positions of one or more joints associated with the one or more objects in each video frame of the video sequence; said processor computing optical flow between each pair of successive video frames for each time step of said video sequence; the processor extracting motion characteristics from each video frame of the video sequence based on the optical flow; and an encoder for encoding state information based on the current pose and the motion characteristics; Including, the encoder comprises one or more processors; system.

19. The processor, determining a current action label for each motion feature included in the state information for a first frame of the sequence of video frames; predicting, by a decoder, a future action label for each motion feature in a second frame of the series of video frames subsequent to the first frame based on the current pose, the action label, and the state information. predicting, by a decoder, a future pose for each motion characteristic in the second frame based on the current pose and the state information; revising the current action label, the future action label, and the future pose based on a loss function; and predicting at least one future pose and a future motion of the second frame based on the model and the video sequence; 20. The system of claim 18, configured to implement:

20. 20. The system of claim 18, wherein the encoder is implemented using a recurrent neural network implementing one or more gated recurrent networks.

21. 20. The system of claim 19, wherein the decoder comprises one or more processors, and wherein the decoder is implemented using a recurrent neural network that implements one or more gated recurrent networks.

22. 20. The system of claim 18, wherein the processor for computing the optical flow forms a two-channel flow frame, each channel including displacement in an x-axis and a y-axis, respectively.

23. The system of claim 18 , wherein each joint position of the one or more joint positions comprises a pair of two-dimensional coordinates (e.g., X, Y) or 3D coordinates (e.g., X, Y, Z).

Images