System, method and medium for vision-based prediction of joint action and posture motion

Abstract

Systems, methods and media for vision-based prediction of joint actions and gesture motion. A computer-implemented method includes: extracting frames of a video sequence; estimating the current pose of an object in each frame and determining the joint positions of joints associated with the object in each frame; calculating optical flow between each pair of consecutive video frames; extracting motion features from each video frame of the video sequence based on the optical flow; encoding state information based on the current pose and the motion features; determining a current action label for each motion feature in the state information for a first frame; predicting a future action label for each motion feature in a second frame after the first frame based on the current pose, the current action label and the state information; predicting a future pose for each motion feature in the second frame based on the current pose and the state information; and refining the current action label, the future action label and the future pose.

Description

Technical Field

[0001] Aspects of example implementations relate to methods, systems, and user experiences associated with image-based predictions in an environment, and more particularly, to a vision-based approach for predicting human motion and gestures. Background Art

[0002] Anticipating human motion and movement trajectories addresses the area of ​​predicting what a person is going to do next and how they are going to do it, and is critical in a variety of current technology applications such as assisted living and future collaborative robotics settings.

[0003] Prior art methods can independently model or predict human actions and human gestures. As described herein, human actions refer to what a person does or causes to happen, such as walking, falling, and opening a door. Human actions are highly related to daily life and associated activities (including a series of actions). Each action can positively or negatively, independently or jointly affect the state of a person. Analyzing human actions can enable machines to better understand the world. Therefore, it is expected to better serve people with human-oriented intelligence (e.g., preventing harmful actions).

[0004] Some prior art systems focus on human action recognition, including recognizing human actions in well-cropped video clips, and can provide action labels after analyzing the entire video clip. These prior art systems can also consider temporal action localization, including locating specific meaningful actions on the timeline after analyzing the uncropped video. However, in the above-mentioned prior art examples, the action has been completed (after the fact) when the system provides the action label. That is, the injury may have occurred when the analysis of the action is completed.

[0005] One of the main problems in prior art systems is that the predicted pose quickly converges to the average pose and the system performs worse than a zero-velocity predictor (ie, only repeats the last observation). Therefore, prior art systems are unable to jointly predict human motion and pose.

[0006] Therefore, there is still a need in the art for a system that can predict future actions and the motion trajectories of those actions. Summary of the invention

[0007] According to one aspect of an example implementation, a computer-implemented method is provided, the method comprising: extracting individual frames of a video sequence at a single frame rate, the video sequence comprising a series of video frames; estimating a current pose of an object within each video frame and determining joint positions of joints associated with the object within each video frame; calculating an optical flow between each pair of consecutive video frames for each time step of the video sequence; extracting motion features from each video frame of the video sequence based on the optical flow; and encoding state information based on the current pose and the motion features.

[0008] Example implementations may also include a non-transitory computer-readable medium having a storage device and a processor capable of executing instructions associated with image-based predictions in an environment, and more particularly, a vision-based method for predicting human motion and gestures. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The patent or application file contains at least one drawing drawn in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0010] Details of embodiments of the present disclosure as to their structure and operation may be gleaned in part by studying the accompanying drawings, in which like numerals represent like parts, and in which:

[0011] Figure 1 is a flow chart of an embodiment of a method for predicting future motion and posture from current observations and previous observations;

[0012] Figure 2 is a functional block diagram of an embodiment of a system for predicting joint actions and gesture motions according to the present disclosure;

[0013] Figure 3 is a flowchart of an embodiment of a method for estimating motion features and human posture from a video according to the present disclosure;

[0014] Figure 4 is a flowchart of an embodiment of a method for multi-task training of a sequence-to-sequence model according to the present disclosure;

[0015] Figure 5 is a flow chart of an embodiment of a method for providing inference using a trained model according to the present disclosure;

[0016] Figure 6 is a graphical depiction of the accuracy of action predictions using different prediction lengths of the test data;

[0017] Fig. 7A and Figure 7B is a flow chart of an embodiment of a method for vision-based joint action and posture motion prediction according to the present disclosure;

[0018] Figure 8 is a functional block diagram of an implementation of a computing environment according to the present disclosure; and

[0019] Fig. 9 is a functional block diagram of an exemplary operating environment according to the present disclosure. DETAILED DESCRIPTION

[0020] The following detailed description provides further details of the drawings and example implementations of the present application. For clarity, the reference numerals and descriptions of redundant elements between the drawings are omitted. The terms used throughout the specification are provided as examples and are not intended to be limiting.

[0021] Various aspects of example implementations relate to deep learning methods for predicting or anticipating human motion based on observations of previous motions. Anticipating human motion and motion trajectories is critical in various applications such as AI-assisted living and human-computer interaction. Early anticipation of motion and gestures can be beneficial for various applications including AI-assisted health care (e.g., preventing patient falls). If an AI system can predict falls in a hospital or assisted living setting, falls can be prevented and injuries reduced, thereby reducing medical costs and improving quality of life.

[0022] Figure 1 Flowchart of an embodiment of a method for predicting future actions and gestures from current observations and previous observations. Humans are naturally able to anticipate actions and actor positions before they occur based on observations of current actions or gestures and the individual's past experiences in similar environments. In a similar manner, the present disclosure provides systems and methods for jointly anticipating or predicting one or more personal action categories and skeletal joints (gestures) in future video frames based on previous video frames that have been previously analyzed.

[0023] Figure 1 The illustrated method 100 can extract motion features 110 and poses 120 from one or more video streams 130. The method 100 can be modeled as a multi-task learning problem, including both action category classification and skeleton joint regression for future frames associated with the video stream 130. Given a video stream 130, motion features 110 (optical flow) and poses 120 of a human object can be extracted for each frame of the video stream 130.

[0024] As used herein, a skeletal pose may utilize multiple (e.g., 14 or 25, 14 represents the total number of joints in the upper body, 25 represents the total number of joints in the body; other numbers of joints may be substituted without departing from the scope of the invention) joint points to represent a given location on the human body. These skeletal poses or joint points may include, describe, or be associated with joints on the body, such as hands, wrists, elbows, shoulders, ankles, knees, hips, and the like. As described herein, for example but not by way of limitation, using RGB (red, green, and blue) images from a video stream provides a lower dimensional object that is easier to model dynamically. For example but not by way of limitation, using a skeleton (e.g., of size 25 joints × 3) is more compact to represent a human object than using raw pixels from an RGB image (e.g., of size 480 × 640).

[0025] At each time step t within the video stream 130 , the motion features 110 and poses 120 may be combined to form a deep model 140 (which may include a sequence-to-sequence model). The method 100 may then be used to progressively predict a sequence 150 of future actions and poses based on the deep model 140 .

[0026] Figure 2 2 is a functional block diagram of an embodiment of a system for predicting joint actions and gesture motions according to the present disclosure. System 200 may be used to perform one or more aspects of a method for predicting joint actions and gesture motions from a video stream (e.g., video stream 130). System 200 may include an encoder 210 communicatively coupled to a decoder 220. System 200 may receive and process (e.g., RGB) video stream 130 and jointly predict future action and gesture sequences. System 200 may include a multi-task encoder-decoder recursive network structure.

[0027] At each time step t of the video stream 130, the encoder 210 may encode multimodal features from past video stream information and identify the current action seen in the video stream 130. The encoded information is used to initialize the decoder 220 with relevant information, which will perform prediction step by step.

[0028] In contrast to systems involving action recognition and detection that make decisions based on an entire video of past events, action prediction as described herein may include using observations available at the moment or occurring in real time to predict or anticipate a motion or gesture immediately after the current time. As shown, this may be accomplished using the sequential encoder-decoder recurrent network structure of system 200.

[0029] The encoder 210 may include a recurrent neural network (RNN) such as a gated recurrent network (GRU) or a long short-term memory (LSTM). The encoder 210 may have one or more RNNs shown as GRUs 212a, 212b, 212c. The encoder 210 may encode multimodal features from past video sequences (e.g., video stream 130) and determine the current action type based on the past video sequence. For example, the current action type may include a description of an action seen in the video stream 130. For example, Figure 1 As shown in video stream 130 of FIG. 1 , a user stands next to a microwave oven and then reaches out to open the microwave oven. The current state information may describe various aspects of such a scene.

[0030] Decoder 220 may receive current state information 230 from encoder 210. Current state information 230 (also referred to herein as state 230) may be encoded as a hidden state or masked by a hash function, compression, or other suitable processing by encoder 210. Therefore, state 230 may also be referred to herein as a "hidden state" or "encoder state."

[0031] The decoder 220 may receive three pieces of information: state information 230, the predicted current action 200, and the detected pose as a starting point. A "loss function" is shown above 240. Figure 2 In , "at" means the action type, corresponding to "yt" in the loss function. The decoder 220 may then implement an RNN 222 (shown as GRU 222a, GRU 222b) to progressively generate a future sequence 240 (shown as 240a, 240b) of action and pose pairs based on a model (e.g., deep model 140) created by the encoded past frames and the recognized current action.

[0032] At time t, given a series of past video frames V 0:t (e.g., from video stream 130), system 200 can be used to predict a sequence of action labels and corresponding poses for the next T time steps. This can be expressed as (Y t:t+T ,P t:t+T ). t P represents the action category at time step t and is encoded as a one-hot vector where 1 is the true class and 0 is all others. t A two-dimensional (2D) pose at time t defined by the (x,y) coordinates of a set of key body joints (alternatively, a 3D pose represented by (x,y,z) coordinates). Key body joints in this sense may include, for example, nose, neck, shoulders, elbows, wrists, ankles, knees, eyes, ears, toes, heels, elbows, etc.

[0033] Figure 3 300 is a flowchart of an embodiment of a method for estimating motion features and human posture from a video according to the present disclosure. For example, method 300 may be performed by system 200. Method 300 may utilize the progress of deep learning representation from a RGB image f rgb , optical flow f motion And people posture pose The RGB modality represents the visual appearance.

[0034] For example, the optical flows of the latter two modalities f motion And people posture pose represents short-term and long-term motion information about a sequence of video frames from the video stream 130, respectively. In order to capture short-term relationships, as opposed to a single image, the method 300 may implement a stack of L video frames (e.g., from the video stream 130) as input. The following describes the feature representation f rgb 、f motion and f pose .

[0035] RGB feature extraction. Frame-level features are extracted from the ResNet-18 convolutional neural network (CNN). The standard ResNet-18 CNN accepts RGB images and has 3 input channels. In order to accommodate a stack of L frames as input, the input channels of the ResNet-18 CNN network are adjusted to L×3. The final RGB representation f rgb is the 512-d vector extracted from the last average pooling layer. In addition to ResNet-18, alternative off-the-shelf CNN networks such as VGG, Densenet or with different layers such as ResNet-34 may be employed in other example implementations.

[0036] Optical flow feature extraction. A deep learning-based optical flow estimator (e.g., FlowNet 2.0) can be used to calculate the optical flow between two consecutive frames of the video stream 130. For each time step t, such a system can generate a two-channel flow frame, where each channel contains the displacement of the x-axis and the y-axis respectively. Similar to the RGB modality, we use L stacked optical flow frames as the input of the ResNet-18 CNN and change the number of input channels to L×2. The final representation f motion is a 512-d vector.

[0037] Pose Extraction. Pose has recently received significant attention due to the success of real-time body point detection from a single image. For the pose modality, an off-the-shelf pose estimator (e.g., OpenPose) can be employed to determine the joint positions of key body points from each frame. Since each joint position can be represented by a pair of 2D coordinates, each pose is encoded by a 1D vector with num_joints×2 elements. The final pose representation f pose is the concatenation of poses from L frames, with a total size of 1×L×num_joints×2 elements. Given a video (e.g., video stream 130), the system can first extract all frames at a fixed sampling rate (e.g., 30fps). At each time step t, RGB, optical flow, and pose features are used separately or concatenated together as represented by f t The final multimodal feature representation of .

[0038] In block 310, the system 200 may extract all video frames from the video stream 130 at a fixed sampling rate (e.g., 30 frames per second (fps)), or frames that will allow for depiction of smooth motion. In order to maintain consistent frame-by-frame comparisons of predictions of future actions and gestures, it is important to fix the sampling rate across all video frames. If the sampling rate or frame rate is inconsistent, the speed of the action may become unclear. Therefore, for inconsistent frame rates, it is difficult for the model to learn the dynamics of the action. As used herein, 30fps does not limit the present disclosure. Other sampling rates lower or higher than 30fps are also possible, such as 40, 45, 50, 60fps or higher, including any intermediate frame rate values. Common video frame rates include (but are not limited to) 24, 29.97, 30, 59.94 and 60fps. For processing, the ratio may be standardized to 29.97 or 30fps (24fps usually appears in movies). Therefore, an example implementation may include a frame rate between 24fps and 60fps.

[0039] The extracted video frames are then used to compute features in two different domains, namely, skeletal pose features and motion features of the objects within the video.

[0040] At block 330, pose estimation may model a skeletal pose feature 332. During pose estimation at block 330, an off-the-shelf pose estimation deep model may be used to estimate the joint positions of one or more people in each video frame. Each joint position of the pose feature 332 may be represented by a two-dimensional (2D) coordinate or a 3D coordinate. Thus, each pose may be described as 1×num joints *2vector.

[0041] At block 320, optical flow estimation may be used to model motion features. In block 320 or the motion branch of method 300, an optical flow estimation depth model may be implemented to compute an optical flow 322 between each pair of consecutive video frames (e.g., of video stream 130). Note that the width and height of the optical flow are the same as the RGB frames associated with video stream 130, however, the optical flow 322 has two channels representing displacement values ​​for each of the x-axis and y-axis.

[0042] At block 340, the pre-trained feature extractor may generate motion features 342 for each video frame. The feature extraction of block 340 may receive a stacked optical flow as input, which concatenates the optical flow of the current time and the previous num_stack time steps. The output of block 340 may include short-term motion features 342 with dimensions motion-dim. Using early fusion of each frame, the motion features and pose features are concatenated together as the final features for subsequent processing.

[0043] Method 300 may also implement late fusion to effectively represent the combined features of two modalities (e.g., pose and optical flow / motion). Unlike early fusion, which employs only one encoder (e.g., encoder 210 with a single RNN) to capture temporal patterns from the concatenated features, late fusion may include different encoders for each modality before concatenation, resulting in separate hidden states for motion and pose. In some examples, similar to the above combined Figure 2 As described above, the encoder (eg, encoder 210) may be a RNN such as a GRU unit.

[0044] Compared to early fusion, late fusion increases model complexity due to the introduction of an additional encoder structure. However, since motion features are dense descriptors of local variations, while pose can be viewed as a higher-level abstraction of sequential dynamics, the partitioned latent space of late fusion allows for greater flexibility than early fusion in modeling these different types of temporal correlations.

[0045] Therefore, in order to further improve the decoding process of method 300, the (first or main) GRU 222 and 212 can be enhanced with auxiliary GRU units during the training phase to form a bidirectional structure. In addition to the future posture prediction via the first GRU 222, the second GRU unit 222 is initialized with the same hidden state, but is intended to reconstruct the previous input in reverse order. The bidirectional GRU better captures the context when running the input from the past to the future and from the future to the past, which will retain the information of the past and the future. More specifically, at each time step, the reconstructed posture of the current frame is provided for reconstructing the input posture from the last frame. This forces the hidden state (e.g., state 230) to maintain sufficient necessary historical information about posture and motion, which is crucial for restoring each past frame. Therefore, after training, the hidden state may be more able to capture long-term patterns, although this enhanced structure is simplified to the initial single GRU decoder and the auxiliary GRU is removed during the test phase.

[0046] Figure 4 400 is a flowchart of an embodiment of a method for multi-task training of a sequence-to-sequence model according to the present disclosure. The sequence-to-sequence model can solve the above-mentioned prediction problem. The disclosed method may include: an encoder 410, which is configured to encode past information; and a decoder 430, which is configured to predict future actions and postures (e.g., of individuals in video stream 130). Encoder 410 can implement an RNN (e.g., GRU) to encode multimodal features from past video sequences (e.g., from video stream 130) and determine the current action type.

[0047] Decoder 430 is set to the same value as the current hidden state 412 of encoder 410. By feeding the predicted current action and the detected posture as a starting point, decoder 430 can (for example, using GRU) gradually generate a sequence of future action and posture pairs. Encoder 410 and decoder 430 can both be RNN-based models. At each time step, encoder 410 can receive the final features of the current frame and all previous frames as input. Since RNN can model the correlation between different elements in the sequence, encoder 410 encodes all observed frames into hidden states (long-term information of the sequence).

[0048] Given the encoded hidden state 412 from the encoder 410, the decoder 430 may output step-by-step action labels 432 and poses 434. Since the input of the decoder 430 and the output of the encoder 410 should be the same, the input of the decoder 430 may also be the action labels 432 and poses 434. However, there are no action labels at any time step.

[0049] In some implementations, the encoder 410 may also identify the action category of the current frame. The action label and the pose calculated at the current time step may then be fed into the decoder 430 as input for its first time step. The decoder 430 then outputs the prediction result (i.e., action label and pose) for the first time step. For subsequent time steps, the decoder 430 takes the output of the previous time step as input until it meets the maximum time step. As a result, the system 400 may provide the future action label and pose predicted for each time step.

[0050] Method 400 may include three tasks: Figure 4 Labeled as Task 1, Task 2, Task 3 in . These tasks include identifying the current action label (Task 1), predicting future action labels (Task 2), and predicting future postures (Task 3). Although they are separate tasks, they are not completely independent of each other. For example, knowing the current action label 436 will help predict future action labels because they evolve from the current action labels. On the other hand, future action labels will serve as additional verification for predicting future postures. In this way, the posture prediction module can only model postures belonging to specific actions, which will reduce the burden on the module and therefore make it easier to learn. On the other hand, in addition to the action label of the current time step, future postures can be used as additional information for future posture prediction. In summary, these three tasks are closely related to each other and are therefore natural candidates for joint learning.

[0051] like Figure 4As shown, the proposed model consists of a backbone network with shared weights and three heads that implement these tasks. With the help of multi-task learning, the proposed model is jointly trained. The multi-task encoder-decoder recurrent network is jointly optimized end-to-end using the loss function 450. The loss function can be expressed as follows (1):

[0052]

[0053] here, It is the sum of the action classification losses of each frame from time t to t+T. is the pose joint position of each frame from time t to t+T The sum of the smoothed regression losses. λ is a scalar parameter that balances the two types of losses. It should be noted that action classification is performed by the action classifier at 420 to define the current action label 436. Based on the incorporation of the RNN model within the decoder 430, the system 400 is naturally able to handle different prediction lengths. As used herein, the prediction length is the amount of time in the future that the system 400 can provide motion and posture predictions.

[0054] Given the encoded hidden state 412 from the encoder 410, the decoder 430 is able to output both the action label and the pose by itself step by step. At the first prediction time step, the decoder 430 also takes as input the action label and pose identified at the last time step seen. For subsequent time steps, the decoder 430 always takes as input the output of the previous time step, i.e., the hidden state, action label, and pose. Therefore, the prediction length is set as a variable parameter of the system. It can be predicted as long as it is set during inference.

[0055] Figure 5 is a flow chart of an embodiment of a method for providing inference using a trained model according to the present disclosure. At block 505 (similar to block 405), motion features 110 and poses 120 are extracted from video stream 130. At block 510, the extracted motion features 110 and poses 120 may be encoded (e.g., by encoder 210 or 510) as state information 230.

[0056] Based on the trained model generated by method 400 , the action classifier 520 may perform action classification at block 520 to define a current action label 536 .

[0057] At block 530, decoder 220 may decode state 230 to predict future labels 532 and future poses 534 based on the model from method 400. Unlike for training the model, for the case where method 500 is used for motion and pose prediction, current action labels 436, future action labels 432, and future poses 434 are different from current action labels 536, future action labels 532, and future poses 534.

[0058] Figure 6 is a graphical depiction of the accuracy of action predictions using different prediction lengths for test data. Table Methods 400 and 500 can be evaluated based on test data. For example, a simple program such as furniture assembly can be used as a proxy for another, potentially more complex task. The example dataset has a set of videos (e.g., video stream 130) of individuals assembling small pieces of furniture.

[0059] The exemplary data set has 101 short videos, each of which is about 2-4 minutes long and shot at 30 frames per second. The example scene includes 14 actors in the video, and sequences from 11 actors are implemented for training and validation using method 400, while testing is completed using method 500 as a balance. Half of the sequences are shown assembled on the floor, while the other half are shown assembled on a workbench. Exemplary action labels for the video include, for example, four "attach leg" actions (one for each leg), four "disassemble leg" actions, one "pick up leg" action, one "flip table" action, "spin in" and "spin out" actions, and empty actions for frames that are not or cannot be labeled by method 400.

[0060] The system 200 can be evaluated based on two aspects: motion prediction accuracy and performance of future pose prediction. In the first aspect, motion prediction accuracy can be compared with the latest or existing models and motion prediction accuracy of different prediction lengths. The two selected comparison methods are 1) LSTM using a two-stream prediction system and 2) LSTM using improved dense trajectories (iDT).

[0061] As shown in Table 1 below, the accuracy of the next first prediction (Acc0) of method 500 (e.g., sequence to sequence or seq2seq) is more than 11.1% higher than that of the two controls. In addition, the two other prediction methods can only predict the next time step, while the models implemented by methods 400 and 500 can predict multiple consecutive time steps or sequences.

[0062] In the disclosed test, method 500 was able to achieve 74.3% accuracy on the fifth prediction (Acc4) (or four steps after Acc0), which is only slightly lower than the first prediction. The decrease in prediction accuracy is due to the decrease in the amount of information about future poses / motions, so the prediction accuracy decreases. Figure 4 As shown, the relationship between action prediction accuracy and prediction length is shown every 5 time steps from the next 1st (Acc0) to the 61st. As shown, the accuracy decreases as the prediction length increases, which proves the above point.

[0063] Table 1:

[0064]

[0065] Fig. 7A and Figure 7B 700 is a flow chart of an embodiment of a method for vision-based joint action and posture motion prediction according to the present disclosure. The method 700 may be implemented by one or more processors in a computing environment (e.g., computing environment 1000 described below). In some embodiments, Fig. 7A As shown, method 700 may be a combination of multiple sub-processes, including estimating current posture and joint motion at 701 , training a deep learning model (e.g., learning) at 702 , and predicting future motion and posture based on the model at 703 .

[0066] In more detail, Figure 7B Further details of estimation 701, learning 702, and prediction 703 of method 700 according to an example implementation are shown. Elements 705-725 represent additional details associated with estimation 701, elements 730-745 represent additional details associated with learning 702, and element 750 represents additional details associated with prediction 703.

[0067] At block 705, the system 200 may extract individual frames of a video sequence (e.g., video sequence 130) at a single frame rate. The single frame rate may be constant across all video frames of the video sequence. For example, the video frames may be formatted as a red, green, and blue (RGB) video format. However, the example implementation is not limited thereto. For example, but not by way of limitation, although the video frames may be extracted as RGB, the video frames may be represented as YUV in one or more other well-known video formats.

[0068] At block 710, the system 200 may estimate the current pose of one or more objects within each video frame. The system 200 may also determine joint positions of one or more joints associated with the one or more objects within each video frame of the video sequence. The determination of joint positions and the estimation of pose may occur simultaneously, in parallel, or sequentially in any order.

[0069] At block 715 , the system 200 may calculate optical flows between each pair of consecutive video frames at each time step of the video sequence to form a two-channel flow frame, where each channel includes displacements in the x-axis and the y-axis, respectively.

[0070] At block 720 , the system 200 may extract motion features from each video frame of the video sequence based on optical flow.

[0071] At block 725 , system 200 may encode state information based on the current pose and motion characteristics.

[0072] At block 730 , the system 200 may determine, for a first frame in a series of video frames, a current action label for each motion feature included in the state information;

[0073] At block 735 , the system 200 may predict future action labels for each motion feature in a second frame following a first frame in a series of video frames based on the current action labels and the state information.

[0074] At block 740 , system 200 may predict a future pose of each motion feature in the second frame based on the current pose and the state information.

[0075] At block 745 , the system 200 may refine the current action label, future action labels, and future poses based on the loss function.

[0076] At block 750 , the system 200 may predict at least one future pose and a future motion of the second frame based on the model and the video sequence.

[0077] For example, method 700 and other embodiments disclosed herein may be applicable to a robot or robotic system. In some examples, a robotic system may implement method 700 in various medical or surgical procedures to improve the speed of various procedures by anticipating the needs of human actors, or to enhance the speed of training. In at least one example, a robotic assistant in a medical application may anticipate when a patient is about to fall based on a video of the patient / human subject just captured, or provide surgical tools to a surgeon during surgery based on the actions of the surgical team.

[0078] According to another example use, the activities performed by the surgeon during a surgical procedure may be the subject of posture and motion prediction. Thus, when the surgeon begins to perform postures and motions corresponding to potential risks (e.g., placing an incision or tool in an incorrect position, angle, setting, etc.), the example implementation may provide a prediction output that provides the surgeon with an opportunity to take corrective action and avoid harming the patient.

[0079] In another example use, method 700 may be implemented in a retail environment for loss or theft prevention, anticipating the motion of an object captured on video (e.g., video stream 130). For example, but not by way of limitation, motion prediction may be used to perform gesture motion prediction on an individual entering a retail environment, such that gestures and motions associated with shoplifting behavior may be detected prior to theft. Similarly, gestures and motions associated with activities in a store that may harm a user or damage merchandise, such as certain gestures of an individual interacting with a product (e.g., testing or holding an oversized product individually, climbing up a shelf to reach a product on a higher shelf, etc.). Thus, example implementations may be used to reduce the risk of harm to an individual (e.g., a customer or employee) in a retail environment, as well as identify and provide intervention opportunities in a manner that reduces the risk of behaviors such as loss or theft.

[0080] In addition, example implementations may be used in consumer-oriented settings (e.g., at home). Consumer appliances, robots, communication devices, and the like may implement actions based on gesture and motion predictions. For example, but not by way of limitation, based on a series of gestures and motions, activities associated with cooking may be predicted, and appliances in the kitchen may be set to one or more modes (e.g., preheating, thawing, etc.) to provide additional convenience to consumers. In other example implementations, an assistant may be provided to consumers that can identify when a consumer is performing a task in an incorrect order (e.g., a consumer is cooking and forgets to add ingredients, etc.), and provide corrective actions to avoid injury, errors, or other harm. Home assistant products may sense such activities through audio, video, or other sensors or IoT devices in various consumer and home settings and provide corrective feedback or suggestions to the user.

[0081] Example implementations may also be used in industrial settings such as manufacturing, assembly, and the like, for example, but not by way of limitation, where gestures and motions associated with defects in manufacturing (e.g., a total quality management approach for quality control) may be used to predict defects before they occur. In such example implementations, gestures or motions associated with corresponding defects (e.g., not adequately attaching a part, skipping an assembly step, or using an incorrect tool or incorrect number of parts) may be detected by the gestures or motions of an individual or robot. In response, based on the output predictions associated with the gestures and motions, corrective actions may be suggested, additional training may be provided, or other recommendations or actions may be provided.

[0082] Figure 8 800 has an example computer device 805 suitable for use in some example implementations. The computing device 805 in the computing environment 800 may include one or more processing units, cores or processors 810, memory 815 (e.g., RAM, ROM, etc.), internal storage 820 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interfaces 825, any of which may be coupled to a communication mechanism or bus 830 for communicating information or embedded in the computing device 805. The computing environment 800 may support operations associated with the system 200 and the methods 80, 300, and 400.

[0083] According to this example implementation, processing associated with neural activity may occur on a processor 810 that is a central processing unit (CPU). Alternatively, other processors may be used instead without departing from the inventive concept. For example, but not limited to, a graphics processing unit (GPU) and / or a neural processing unit (NPU) may be used instead of or in combination with the CPU to perform the processing of the above example implementation. In various implementations or embodiments of the disclosed systems and methods, the RNN may be implemented using one or more GRUs and or LSTMs to perform the functions and achieve the purposes described herein.

[0084] The computing device 805 may be communicatively coupled to an input / interface 835 and an output device / interface 840. Either or both of the input / interface 835 and the output device / interface 840 may be wired or wireless interfaces and may be detachable. The input / interface 835 may include any device, component, sensor, or interface (physical or virtual) that can be used to provide input (e.g., buttons, touch screen interfaces, keyboards, pointing / cursor controls, microphones, cameras, Braille, motion sensors, optical readers, etc.).

[0085] Output devices / interfaces 840 may include displays, televisions, monitors, printers, speakers, Braille, etc. In some example implementations, input / interfaces 835 (e.g., user interfaces) and output devices / interfaces 840 may be embedded in or physically coupled to computing device 805. In other example implementations, other computing devices may serve as or provide functionality for input / interfaces 835 and output devices / interfaces 840 for computing device 805.

[0086] Examples of computing devices 805 may include, but are not limited to, highly mobile devices (e.g., smart phones, devices in vehicles and other machines, devices carried by people and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices that are not designed for mobility (e.g., desktop computers, server devices, other computers, kiosks, televisions, radios embedded with and / or coupled to one or more processors, etc.).

[0087] The computing device 805 may be communicatively coupled to an external storage device 845 and a network 850 (e.g., via an I / O interface 825) for communicating with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. The computing device 805 or any connected computing device may function as, provide services for, or be referred to as a server, client, thin server, general purpose machine, special purpose machine, or another label. For example, and not by way of limitation, the network 850 may include a blockchain network and / or a cloud.

[0088] I / O interface 825 may include, but is not limited to, wired and / or wireless interfaces using any communication or I / O protocol or standard (e.g., Ethernet, 802.11xs, universal system bus, WiMAX, modems, cellular network protocols, etc.) for communicating information to and / or from at least all connected components, devices, and networks in computing environment 800. Network 850 may be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0089] The computing device 805 may use and / or communicate with computer-usable or computer-readable media (including both transitory and non-transitory media). Transitory media include transmission media (e.g., metal cables, optical fibers), signals, carrier waves, etc. Non-transitory media include magnetic media (e.g., magnetic disks and tapes), optical media (e.g., CD ROMs, digital video disks, Blu-ray disks), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage devices), and other non-volatile storage devices or memories.

[0090] The computing device 805 may be used to implement techniques, methods, applications, processes, or computer-executable instructions in some example computing environments. The computer-executable instructions may be retrieved from a transient medium, as well as stored on and retrieved from a non-transitory medium. The executable instructions may originate from one or more of any programming, scripting, and machine languages ​​(e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0091] The processor 810 can be executed under any operating system (OS) (not shown) in a native or virtual environment. One or more applications can be deployed, which include a logic unit 855, an application programming interface (API) unit 860, an input unit 865, an output unit 870, a training unit 875, a feature extraction unit 880, a prediction unit 885, and an inter-unit communication mechanism 895 for different units (e.g., encoder 110 and decoder 120) to communicate with each other, communicate with the OS, and communicate with other applications (not shown).

[0092] The training unit 875 can perform the functions associated with the encoder 110 and train the model as in the methods 300 and 400. For example, the feature extraction unit 880 can perform the functions associated with the method 300. In some implementations, in addition to the feature extraction ( Figure 3 ), the feature extraction unit 880 may also perform frame extraction of block 310, posture estimation of block 330 ( Figure 3). The prediction unit 885 may perform functions associated with the decoder 120, including predicting or anticipating gestures and motions based on the state information 230 in the methods 100, 300, and 400.

[0093] For example, in addition to method 300 and method 400, training unit 875, feature extraction unit 880, and prediction unit 885 may implement one or more of the processes shown above with respect to the above structures. The design, function, configuration, or implementation of the described units and elements may vary and are not limited to the description provided.

[0094] In some example implementations, when information is received or instructions are executed via the API unit 860, it may be communicated to one or more other units (e.g., the logic unit 855, the input unit 865, the training unit 875, the feature extraction unit 880, and the prediction unit 885).

[0095] For example, the training unit 875 may receive and process information from simulated data, historical data, or one or more sensors, as described above at least in conjunction with method 400. The output of the training unit 875 is provided to the feature extraction unit 880, which extracts the feature information based on, for example, the information described above and in Figures 1 to 5 The necessary operations are performed by applying the neural network shown in FIG. 875. In addition, based on the outputs of the training unit 875 and the feature extraction unit 880, the prediction unit 885 may perform operations and minimize the reprojection error to provide an output signal.

[0096] In some cases, the logic unit 855 may be configured to control the flow of information between the units and direct the services provided by the API unit 860, the input unit 865, the training unit 875, the feature extraction unit 880, and the prediction unit 885 in some of the above-mentioned example implementations. For example, the flow of one or more processes or implementations may be controlled by the logic unit 855 alone or in combination with the API unit 860.

[0097] Fig. 9 900 is a functional block diagram of an exemplary operating environment according to the present disclosure. Environment 900 may be suitable for some of the example implementations disclosed herein. Environment 900 includes devices 905-945, and each device is communicatively connected to at least one other device via, for example, network 960 (e.g., via wired and / or wireless connections). Some devices may be communicatively connected to one or more storage devices 930 and 945.

[0098] Examples of one or more devices 905-945 may be Figure 8905-945 may include, but are not limited to, a computer 905 (e.g., a laptop computing device) with a monitor and associated webcam as described above, a mobile device 910 (e.g., a smart phone or tablet), a television 915, a device associated with a vehicle 920, a server computer 925, computing devices 935-940, storage devices 930 and 945.

[0099] In some implementations, devices 905-920 may be considered user devices associated with users who may remotely obtain sensed inputs used as inputs for the example implementations described above. In this example implementation, as described above, one or more of these user devices 905-920 may be associated with one or more sensors that may sense information required for this example implementation, such as a camera temporarily or permanently embedded in the user's body away from a patient care facility.

[0100] Although the above example implementations are provided to indicate the scope of the present invention, they are not intended to be limiting, and other methods or implementations may be substituted or added without departing from the scope of the present invention. For example, but not by way of limitation, image technologies other than those disclosed herein may be employed.

[0101] According to an example implementation, algorithms such as SuperPoint can be used to train image point detection and determination. In addition, example implementations can employ alternative image classification algorithms, and / or use other neural network structures (e.g., Siamese networks). Additional methods incorporate expert knowledge in region-like actions, apply enhancements to both images using techniques such as formation, lighting, and illumination, and / or use a single image for deep methods.

[0102] The example implementation may have various advantages and benefits, although this is not required. For example, but not by way of limitation, the example implementation is feasible on small data sets. In addition, the example implementation provides constraints for locations inside target tissues such as the colon or lungs. Therefore, surgeons may be able to more accurately locate locations within any human body using video. In addition, the example implementation provides much higher accuracy than prior art methods.

[0103] Although some example implementations have been shown and described, these example implementations are provided in order to convey the subject matter described herein to those familiar with the art. It should be understood that the subject matter described herein can be implemented in various forms, without being limited to the described example implementations. The subject matter described herein can be practiced without those specifically defined or described matters or with other or different elements or matters that are not described. Those familiar with the art will understand that these exemplary implementations can be changed without departing from the subject matter defined in the attached claims and their equivalents as described herein.

[0104] Aspects of certain non-limiting embodiments of the present disclosure address the features discussed above and / or other features not described above. However, aspects of non-limiting embodiments need not address the above features, and aspects of non-limiting embodiments of the present disclosure may not address the above features.

Claims

1. A computer-implemented method comprising the following steps: Extracting individual frames of a video sequence at a single frame rate, the video sequence comprising a series of video frames; estimating a current pose of an object within each video frame and determining joint positions of joints associated with the object within each video frame; calculating the optical flow between each pair of consecutive video frames for each time step of the video sequence; extracting motion features from each video frame of the video sequence based on the optical flow; as well as The encoder encodes the state information based on the current posture and the motion characteristics, Wherein, the encoder is implemented using a recursive neural network implementing one or more gated recursive networks or long short-term memories.

2. The method according to claim 1, further comprising the following steps: Determining, for a first frame in the series of video frames, a current action label for each motion feature included in the state information, the series of video frames being in a red, green, and blue (RGB) format; Predicting, by a decoder, a future action label for each motion feature in a second frame following the first frame in the series of video frames based on the current posture, the current action label and the state information; predicting, by the decoder, a future pose for each motion feature in the second frame based on the current pose and the state information; as well as The current action label, the future action label, and the future posture are refined based on a loss function. 3 . The method of claim 2 , further comprising predicting at least one future pose and a future motion for the second frame based on a model and the video sequence.

4. The method according to claim 2, wherein: The encoder and the decoder include one or more processors.

5. The method according to claim 4, wherein: The decoder is implemented using a recurrent neural network implementing one or more gated recurrent networks.

6. The method according to claim 1, wherein: The step of calculating the optical flow forms a two-channel flow frame, wherein each channel contains the displacement of the x-axis and the y-axis respectively.

7. The method according to claim 1, wherein: The frame rate is between 24fps and 60fps.

8. The method according to claim 1, wherein: Each of the one or more joint positions includes a pair of two-dimensional coordinates or 3D coordinates.

9. A non-transitory computer-readable medium comprising instructions executable on a processor, the instructions comprising: Extracting, by an encoder, individual frames of a video sequence at a single frame rate, the video sequence comprising a series of video frames; estimating a current pose of one or more objects within each video frame and determining joint positions of joints associated with the objects within each video frame of the video sequence; calculating the optical flow between each pair of consecutive video frames for each time step of the video sequence; extracting motion features from each video frame of the video sequence based on the optical flow; as well as The encoder encodes state information based on the current posture and the motion feature, Wherein, the encoder is implemented using a recursive neural network implementing one or more gated recursive networks or long short-term memories.

10. The non-transitory computer readable medium of claim 9, wherein: The instructions also include: determining, for a first frame in the series of video frames, a current action label for each motion feature included in the state information; Predicting, by a decoder, a future action label for each motion feature in a second frame following the first frame in the series of video frames based on the current posture, the current action label and the state information; predicting, by the decoder, a future pose for each motion feature in the second frame based on the current pose and the state information; and The current action label, the future action label, and the future posture are refined based on a loss function.

11. The non-transitory computer readable medium of claim 10, wherein: The instructions also include predicting at least one future pose and a future motion for the second frame based on a model and the video sequence.

12. The non-transitory computer readable medium of claim 10, wherein: The encoder and the decoder include one or more processors, and the decoder is implemented using a recurrent neural network implementing one or more gated recurrent networks.

13. The non-transitory computer readable medium of claim 9, wherein: The process of calculating the optical flow forms a two-channel flow frame, where each channel contains the displacement of the x-axis and the y-axis respectively.

14. The non-transitory computer readable medium of claim 9, wherein: The frame rate is between 24fps and 60fps.

15. The non-transitory computer readable medium of claim 9, wherein: Each of the one or more joint positions includes a pair of two-dimensional coordinates or 3D coordinates.

16. A system for vision-based joint action and posture motion prediction comprising a processor and a storage device, the system comprising: an encoder configured to extract individual frames of a video sequence at a single frame rate, the video sequence comprising a series of video frames; The processor estimates a current pose of one or more objects within each video frame and determines joint positions of one or more joints associated with the one or more objects within each video frame of the video sequence; The processor calculates the optical flow between each pair of consecutive video frames for each time step of the video sequence; The processor extracts motion features from each video frame of the video sequence based on the optical flow; as well as wherein the encoder is configured to encode state information based on the current posture and the motion feature, and Wherein, the encoder is implemented using a recursive neural network implementing one or more gated recursive networks or long short-term memories.

17. The system of claim 16, wherein: The processor is configured to perform: determining, for a first frame in the series of video frames, a current action label for each motion feature included in the state information; Predicting, by a decoder, a future action label for each motion feature in a second frame following the first frame in the series of video frames based on the current posture, the current action label and the state information; predicting, by the decoder, a future pose for each motion feature in the second frame based on the current pose and the state information; Refining the current action label, the future action label, and the future posture based on a loss function; and At least one future pose and a future motion for the second frame is predicted based on the model and the video sequence.

18. The system of claim 17, wherein: The encoder and the decoder include one or more processors, and wherein the decoder is implemented using a recurrent neural network implementing one or more gated recurrent networks.

19. The system of claim 16, wherein: The processor calculates the optical flow to form a dual-channel flow frame, wherein each channel contains displacements of the x-axis and the y-axis respectively.

20. The system of claim 16, wherein: Each of the one or more joint positions includes a pair of two-dimensional coordinates or 3D coordinates.

Images