Apparatus and method for three-dimensional pose estimation

Abstract

Apparatuses and methods for three-dimensional pose estimation are disclosed herein. An apparatus comprises an image synchronizer (214) for synchronizing a first image (114) generated by a first image capture device (104) and a second image (116) generated by a second image capture device (106), the first image (114) and the second image (116) comprising an object (102); a two-dimensional pose detector (216) for predicting a first position of a keypoint of the object (102) based on the first image (114) and by executing a first neural network model to generate first two-dimensional data, and predicting a second position of the keypoint based on the second image (116) and by executing the first neural network model to generate second two-dimensional data; and a three-dimensional pose calculator (218) for generating a three-dimensional graphical model (1100) representing a pose of the object (102) in the first image (114) and the second image (116) based on the first two-dimensional data, the second two-dimensional data and by executing a second neural network model.

Description

[0001] Divisional Application Instructions

[0002] This application is a divisional application of the Chinese invention patent application with application date of June 26, 2020, application number 202080101140.1, and titled “Device and Method for Three-Dimensional Posture Estimation”. Technical Field

[0003] The present disclosure relates generally to pose estimation, and more particularly to apparatus and methods for three-dimensional pose estimation. Background Art

[0004] Pose estimation uses image data to determine the pose (e.g., position and orientation) of an object (e.g., a human) or target. The image data is analyzed to, for example, identify the positions of the object's joints (e.g., elbows, knees, ankles) in the image data that indicate the object's pose. The pose information obtained from the image data can be used to analyze the characteristics of the subject's body during activities such as sports. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 is a block diagram of an example system that includes a three-dimensional (3D) model generator for generating a 3D graphical model of an object based on image data.

[0006] Figure 2 yes Figure 1 A block diagram of an example system including an example implementation of a 3D model generator and including one or more computing systems for training a neural network to generate a model for use during 3D pose estimation according to the teachings of the present disclosure.

[0007] Figure 3-Figure 5 The flowchart shows the Figure 1 The example 3D model generator implements an example algorithm.

[0008] Figure 6 Illustrated are example two-dimensional (2D) graphics model data generated according to the teachings of the present disclosure.

[0009] Figure 7 Illustrated are example normalized 2D graphics model data generated according to the teachings of this disclosure.

[0010] Figure 8 An example 3D graphics model generated according to the teachings of the present disclosure is illustrated.

[0011] Figure 9 and Figure 10 Illustrated are example reference body data that may be used to generate a 3D graphical model according to the teachings of this disclosure.

[0012] Figure 11 Another example 3D graphics model generated according to the teachings of this disclosure is illustrated.

[0013] Figure 12 is a flowchart representing example machine-readable instructions that, when executed Figure 2 The example system, when executed by the first computing system, causes the first computing system to train a neural network to predict locations of key points of objects in image data.

[0014] Figure 13 is a flowchart representing example machine-readable instructions that, when executed Figure 2 When executed by a second computing system of the example system, the second computing system trains a neural network to perform mapping of 2D skeleton data to a joint depth offset map.

[0015] Figure 14 is a flowchart representing example machine-readable instructions that, when executed, cause Figure 1 and / or Figure 2 An example 3D model generator generates 3D graphical model(s) of object(s) based on image data.

[0016] Figure 15 is constructed to execute Figure 12 Instructions to achieve Figure 2 A block diagram of an example processing platform of an example first computing system.

[0017] Figure 16 is constructed to execute Figure 12 Instructions to achieve Figure 2 A block diagram of an example processing platform of an example second computing system.

[0018] Figure 17 is constructed to execute Figure 14 Instructions to achieve Figure 1 and / or Figure 2 Block diagram of an example processing platform for an example 3D model generator.

[0019] Figure 18 is a block diagram of an example software distribution platform for distributing software (e.g., Figure 12 、 Figure 13 and / or Figure 14 The software (e.g., software corresponding to example computer-readable instructions) is distributed to client devices, such as consumers (e.g., for licensing, sale, and / or use), retailers (e.g., for sale, resale, licensing, and / or sublicensing), and / or original equipment manufacturers (OEMs) (e.g., for inclusion in products to be distributed to, for example, retailers and / or direct purchasing customers).

[0020] The accompanying drawings are not to scale. Instead, the thickness of layers or regions may be exaggerated in the accompanying drawings. Generally, like reference numerals will be used throughout the drawings and accompanying written description to refer to like or similar components.

[0021] Unless otherwise specifically stated, descriptors such as "first," "second," "third," and the like are used herein without importing or otherwise indicating any priority, physical order, arrangement in a list, and / or ordering in any manner, but are merely used as labels and / or arbitrary names to distinguish elements so as to facilitate understanding of the disclosed examples. In some examples, the descriptor "first" may be used to refer to an element in the detailed description, while the same element may be referred to by a different descriptor, such as "second" or "third," in the claims. In such cases, it should be understood that such descriptors are merely used to explicitly identify those elements that, for example, might otherwise share the same name. DETAILED DESCRIPTION

[0022] Pose estimation uses image data to determine the pose (e.g., position and orientation) of an object (e.g., a human) or target. The image data is analyzed to, for example, identify the positions of the object's joints (e.g., elbows, knees, ankles) in the image data that indicate the object's pose. The pose information obtained from the image data can be used to analyze the characteristics of the object's body during activities such as sports. A two-dimensional (2D) and / or three-dimensional (3D) graphical model (e.g., a skeletal model) of the object can be generated based on the pose estimation data to illustrate the object's pose.

[0023] In some known 3D pose estimation techniques, image data generated by multiple cameras is analyzed to estimate the 2D pose of an object captured in the image data based on joint or keypoint recognition. To generate a 3D graphical model of the object in this pose, known pose estimation techniques can use 2D pose data and methods such as triangulation, statistical modeling (e.g., image structure modeling), and / or 3D geometric modeling techniques (e.g., volume voting, factor graph optimization) to calculate the positions of the user's joints.

[0024] However, such known 3D pose estimation techniques operate based on the assumption that the camera generating the image data is static and that the camera's intrinsic parameters (e.g., focal length) and the camera's extrinsic parameters (e.g., the camera's position and orientation relative to the environment) are known before analyzing the image data. In other words, known 3D pose estimation techniques rely on multi-camera calibration. However, performing multi-camera calibration is resource-intensive and time-consuming. Additionally, multi-camera calibration involving different types of cameras (e.g., pan-tilt camera(s), handheld camera(s)) is difficult. Consequently, known 3D pose estimation techniques are difficult to implement for image data that is collected using a camera with varying zoom levels and / or a camera that moves during image generation. However, such cameras are often used in environments such as sporting events to capture the movements of athletes.

[0025] Disclosed herein are apparatus and methods for performing 3D pose estimation, wherein calibration of an image capture device (e.g., camera(s)) used to generate image data for deriving pose information is not required. To generate a 3D graphical model of an object in a certain pose, the examples disclosed herein use image data generated by multiple image capture devices to perform 2D pose estimation to predict the positions of the object's joints. In the examples disclosed herein, the image capture devices may include different types of image capture devices (e.g., long-range cameras(s), pan-tilt cameras(s). In the examples disclosed herein, neural network regression analysis is performed to generate (one or more) 3D graphical models based on the 2D pose estimation data, which models show the object in the pose. The examples disclosed herein perform optimization to estimate 3D pose parameters (e.g., joint positions) and parameters of the image capture device (e.g., extrinsic parameters, such as device orientation). The examples disclosed herein output (one or more) 3D graphical models (e.g., skeletal models) that show the object in the pose and take into account the estimated parameters of the image capture device without requiring the parameters of the image capture device to be known in advance. The example 3D graphics model(s) disclosed herein may be used for applications such as biomechanical analysis, 3D animation, human interaction recognition, and the like.

[0026] In the examples disclosed herein, a feedforward neural network is implemented to regress a 3D pose from 2D pose image data. The regressed 3D pose is used as calibration data to estimate the parameters of the image capture device and to take into account multiple views of the object in the pose obtained by the image capture device. Parameters of the regressed 3D pose (e.g., bone orientation) are identified to indicate joint rotations relative to a 3D skeleton template. Optimization is performed using the 2D and 3D pose data extracted from the image data and the estimated image capture device parameters to generate a 3D graphical model of the object in the pose. Since the examples disclosed herein do not rely on multi-camera calibration to perform 3D pose estimation, the examples disclosed herein can be implemented to provide accurate 3D pose estimation using image data generated in dynamic environments (including sporting events, such as figure skating and speed skating) where multi-camera calibration is difficult to perform.

[0027] Figure 1 An example system 100 constructed according to the teachings of the present disclosure is illustrated for estimating and tracking the pose of an object 102 located in an environment 103 and generating (one or more) 3D graphical models of the object 102 based on the pose estimate. The example system 100 includes an image capture device for generating image data of the object 102. Figure 1 , example system 100 includes a first image capture device 104 , a second image capture device 106 , a third image capture device 108 , and a fourth image capture device 110 in environment 103 . Figure 1 The example system 100 may include additional or fewer image capture devices (e.g., two image capture devices, six image capture devices). Image capture devices 104, 106, 108, 110 may include, for example, video cameras, still cameras, movable cameras (e.g., pan-tilt cameras), long-range cameras, and / or other types of image capture devices. In some examples, two or more of image capture devices 104, 106, 108, 110 are different types of image capture devices and / or image capture devices with different properties. For example, first image capture device 104 may be a still video camera, while second image capture device 106 may include a movable video camera.

[0028] exist Figure 1 In the example of , the image capture devices 104, 106, 108, 110 are uncalibrated because one or more intrinsic parameters (e.g., focal length, focus) and / or extrinsic parameters (e.g., rotation, position) of each image capture device 104, 106, 108, 110 are unknown and / or vary when the image capture devices 104, 106, 108, 110 generate image data. Figure 1In the example of FIG, image capture devices 104, 106, 108, 110 can be spaced apart from one another in environment 103 such that each image capture device 104, 106, 108, 110 has a different view of at least a portion of environment 103. Each image capture device 104, 106, 108, 110 defines a respective image capture device coordinate system, wherein the position (e.g., (X, Y, Z) position) of a 3D point can be defined relative to the particular image capture device coordinate system. The coordinate system defined by each image capture device 104, 106, 108, 110 is based on the position of the image capture device 104, 106, 108, 110 in environment 103.

[0029] Figure 1 Each image capture device 104, 106, 108, 110 generates image data representing an object 102 (e.g., a human being). Figure 1 As shown, the first image capture device 104 generates a first image 114 including the object 102, and the second image capture device 106 generates a second image 116 including the object 102. The first image 114 and the second image 116 are generated by the respective devices 104, 106 at substantially the same time (e.g., time synchronized). Figure 1 As shown, due to the different positions of the first and second image capture devices 104, 106 in the environment 103, the view of the object 102 in the first image 114 (e.g., a side profile view) is different from the view of the object 102 in the second image data 116 (e.g., a front view). Figure 1 , the third image capture device 108 generates a third image 118 of the object 102, and the fourth image capture device 108 generates a fourth image 120 of the object 102. Figure 1 As shown, based on the different fields of view of the image capture devices 104 , 106 , 108 , 110 , the view of the object 102 captured in each image 114 , 116 , 118 , 120 is different.

[0030] exist Figure 1In the example of , each of the image capture devices 104, 106, 108, 110 captures images of the object 102 over time to generate an image data stream (e.g., a sequence or series of images, such as video frames, still images, etc.). The image capture devices are synchronized in time such that the image data generated by each image capture device captures the object in the same pose, but from different angles based on the position and / or orientation of the image capture device. Thus, the images captured by the image capture devices 104, 106, 108, 110 can be used to track the movement of the object 102 over time, thereby tracking changes in the pose of the object 102. As disclosed herein, image data generated by each image capture device 102, 104, 106, 108 (where the image data generated by each device includes a respective image 114, 116, 118, 120) is used to identify (e.g., predict) a pose of object 102 at a particular time corresponding to the time at which the respective image capture device 104, 106, 108, 110 captured the image 114, 116, 118, 120 including object 102. The image data is used to generate a 3D graphical model of object 102 in that pose.

[0031] Figure 1 The example system 100 includes one or more semiconductor-based processors to process image data generated by the image capture devices 104, 106, 108, 110. In some examples, the processor(s) are located at the image capture devices 104, 106, 108, 110. For example, the second, third, and fourth image capture devices 106, 108, 110 can transmit data to the onboard processor 122 of the first image capture device 104. Similarly, each of the first, second, third, and / or fourth image capture devices 104, 106, 108, 110 can transmit data to the onboard processor 124 of the second image capture device 106, the onboard processor 126 of the third image capture device 108, and / or the onboard processor 128 of the fourth image capture device 110. In other examples, the image capture devices 104, 106, 108, 110 can transmit data to a processor 130 of another user device 132, such as a smartphone, a personal computing device (e.g., a laptop), or the like. In other examples, the image capture devices 104 , 106 , 108 , 110 may transmit data to a cloud-based device 134 (eg, one or more servers, processor(s), and / or virtual machine(s)).

[0032] In some examples, the processor(s) 122, 124, 126, 128 of the image capture device(s) 104, 106, 108, 110 are communicatively coupled to one or more other processors. In such examples, for example, the second, third, and fourth image capture devices 106, 108, 110 can transmit image data including images 116, 118, 120 to the onboard processor 122 of the first image capture device 104. The onboard processor 122 of the first image capture device 104 can then transmit the image data (including image data including the image 114 generated by the first image capture device 104) to the processor 130 of the user device 132 and / or the cloud-based device(s) 134. In some such examples, the image capture device 102 (e.g., the onboard processor 122) and the processor(s) 130, 134 are communicatively coupled via one or more wired connections (e.g., cables) or wireless connections (e.g., cellular, Wi-Fi, or Bluetooth connections). Any onboard processor 122, 124, 126, 128 of an image capture device 104, 106, 108, 110 may be communicatively coupled with one or more other processors 130, 134. In other examples, image data may be processed only by one or more of the onboard processors 122, 124, 126, 128 of the respective image capture device 104, 106, 108, 110.

[0033] exist Figure 1 In the example of , image data generated by image capture devices 104, 106, 108, 110 is processed by 3D model generator 136 to identify pose(s) of object 102 and generate 3D graphical model(s) of object 102 in the pose(s). Figure 1 The 3D model generator 136 generates (one or more) 3D models using image data generated by uncalibrated image capture devices 104, 106, 108, 110 (e.g., one or more intrinsic and / or extrinsic parameters of the image capture devices 104, 106, 108, 1110 are unknown to the 3D model generator 136). The 3D model generator 136 outputs (one or more) 3D graphical models for display and / or analysis by (one or more) user applications 138 (e.g., a body posture analysis application) installed on, for example, the user device 132. Figure 1In the example of the 3D model generator 136, the 3D model generator 136 is implemented by executable instructions executed on one or more processors 122, 124, 126, 128 of the image capture device(s) 104, 106, 108, 110. However, in other examples, the 3D model generator 136 is implemented by executable instructions executed on the processor 130 of the user device 132 and / or the cloud-based device(s) 134. In other examples, the 3D model generator 136 is implemented by dedicated circuitry located on one or more of the image capture devices 104, 106, 108, 110 and / or the user device 132. In some examples, one or more components of the example 3D model generator 136 are implemented by the onboard processor(s) 122, 124, 126, 128 of the image capture device(s) 104, 106, 108, 110, while one or more other components are implemented by the processor 130 of the user device 132 and / or the cloud-based device(s) 134. These components may be implemented in software, firmware, hardware, or a combination of two or more of software, firmware, and hardware.

[0034] exist Figure 1 In some examples, 3D model generator 136 processes image data generated by image capture devices 104, 106, 108, and 110 to perform 3D pose estimation and generate (one or more) 3D graphical models representing (one or more) poses of object 102 in the image data. In some examples, 3D model generator 136 receives image data from each of image capture devices 104, 106, 108, and 110 in substantially real time (as used herein, "substantially real time" means occurring in a nearly instantaneous manner (e.g., within one second), acknowledging that there may be delays in computing time, transmission, etc. in the real world). In other examples, 3D model generator 136 receives image data at a later time (e.g., periodically based on one or more settings and / or aperiodically (e.g., after a few seconds)). 3D model generator 136 can perform one or more operations on the image data generated by each image capture device 104, 106, 108, and 110, such as filtering the image data and / or analyzing the data.

[0035] As disclosed herein, a 3D model generator 136 extracts images (e.g., video frames) from the image data feeds generated by each image capture device 104, 106, 108, 110 and temporally synchronizes the images obtained from each device 104, 106, 108, 110. The 3D model generator analyzes each set of synchronized images to predict the locations of keypoints or joints (e.g., elbows, wrists, pelvis) of the object 102 in the image and estimates the 2D pose of the object based on the keypoint locations. The 3D model generator 136 can identify the locations of keypoints in the image data based on (one or more) keypoint recognition models generated via neural network training. In an example disclosed herein, the 3D model generator 136 computes (e.g., regresses) the 3D pose of the object 102 from the 2D pose data based on a learned neural network model, the model including a mapping of the 2D pose data to a joint depth offset map, wherein the joint depth offset map provides a depth offset of a joint relative to a root joint (i.e., a reference joint) of the object 102 (e.g., a pelvic joint).

[0036] As described above, each image capture device 104, 106, 108, 110 defines its own coordinate system. Figure 1 In the example, Figure 1 The 3D model generator 136 selects one of the image capture device coordinate systems, such as the coordinate system of the first image capture device 104, to use as the world coordinate system, wherein the orientation of the first image capture device 104 defines the three coordinate axes (X, Y, Z) of the world coordinate system. Figure 1 In the example, the 3D model generator 136 estimates intrinsic and extrinsic parameters of the image capture devices 104, 106, 108, 110, such as device orientation (e.g., defined by a rotation matrix), so that 3D joint information (e.g., (X, Y, Z) joint positions) of the object 102 obtained from images of the respective image capture devices 104, 106, 108, 110 can be transformed from the coordinate system of the respective image capture devices 104, 106, 108, 110 to the world coordinate system.

[0037] Figure 1 The 3D model generator 136 uses the 2D pose data, the 3D pose data and the estimated parameters of the image capture devices 104, 106, 108, 110 to solve an optimization problem (e.g., by minimizing a least squares non-objective function). As a result of the optimization, Figure 1The 3D model generator 136 generates a 3D graphical model of the object in the pose relative to a world coordinate system (e.g., the coordinate system of a selected image capture device, such as the first image capture device 104). Thus, the 3D model generator 136 uses the different views of the object 102 obtained from the image capture devices 104, 106, 108, 110 to generate a 3D graphical model that represents the positions of the object's joints, regardless of the positions of the image capture devices 104, 106, 108, 110 in the environment 103. The 3D graphical model(s) generated by the 3D model generator 136 can be transmitted to, for example, the user application(s) 138 of the user device 132 for analysis.

[0038] Figure 2 yes Figure 1 FIG. 1 is a block diagram of an example implementation of the system 100 including an example implementation of the 3D model generator 136. As described above, the 3D model generator 136 is constructed to use an image capture device (e.g., Figure 1 to identify (e.g., predict) an object (e.g., Figure 1 (one or more) poses of the object 102) and generates (one or more) 3D graphics models of the object in a certain pose. Figure 2 In the example of , the 3D model generator 136 is executed by the processor(s) 122, 124, 126, 128 of the image capture device(s) 104, 106, 108, 110, the processor 130 of the user device 132, and / or the cloud-based device 134 (e.g., Figure 1 The 3D model generator 136 may be implemented by one or more of the server(s), processor(s), and / or virtual machine(s) in the cloud 134. In some examples, some of the image data analysis is implemented by the 3D model generator 136 via the cloud computing environment, while one or more other portions of the analysis are implemented by one or more of the processor(s) 122, 124, 126, 128 of the image capture device(s) 104, 106, 108, 110 and / or the processor 130 of the user device 132 (e.g., a smartphone).

[0039] As described above, each of the image capture devices 104, 106, 108, 110 generates image data, where the image data includes a sequence or series of images (eg, video frames, still images) of the object 102 captured over time. Figure 2 As shown in the example 3D model generator 136 receives the Figure 1104, a first image data stream 200 from the first image capture device 104, a second image data stream 202 from the second image capture device 106, a third image data stream 204 from the third image capture device 108, and a fourth image data stream 206 from the fourth image capture device 110. The image data streams 200, 202, 204, 206 may be stored in a database 208. In some examples, the 3D model generator 136 includes the database 208. In other examples, the database 208 is located external to the 3D model generator 136, in a location accessible to the 3D model generator 136, such as Figure 2 shown. Figure 2 The example database 208 of the illustrated example is implemented by any memory, storage device, and / or storage disk for storing data, such as flash memory, magnetic media, optical media, etc. Furthermore, the data stored in the example database 208 can be in any data format, such as binary data, comma-delimited data, tab-delimited data, structured query language (SQL) structures, image data, etc.

[0040] Figure 2 The example 3D model generator 136 includes an image capture device controller 210. In this example, the image capture device controller 210 provides a means for controlling the operation of the image capture devices 104, 106, 108, 110. For example, the image capture device controller 210 can control the power state of the image capture devices 104, 106, 108, 110 and / or settings of the image capture devices 104, 106, 108, 110, such as frame rate, zoom level, position (e.g., for a movable camera), clock synchronization between two or more image capture devices 104, 106, 108, 110, etc. The image capture device controller 210 can control the image capture devices 104, 106, 108, 110 based on one or more image capture device rules 212 defined by user input(s) and stored in the database 208.

[0041] Figure 2The example 3D model generator 136 includes an image synchronizer 214. In the illustrated example, the image synchronizer 214 provides a means for extracting images (e.g., video frames, still images) from the image data streams 200, 202, 204, 206 generated by each of the image capture devices 104, 106, 108, 110 and synchronizing the images captured by each device 104, 106, 108, 110 based on time (e.g., to provide a synchronized set of images for analysis). For example, the image synchronizer 214 can synchronize or align the images in the respective image data streams 200, 202, 204, 206 on a frame-by-frame basis based on timestamps associated with the images generated by each image capture device 104, 106, 108, 110.

[0042] exist Figure 2 In the example of FIG, the images extracted and time-synchronized by the image synchronizer 214 are used by the 3D model generator 136 to detect (e.g., predict) the 2D pose(s) of the object(s) in the image data and generate the 3D graphical model(s) of the object(s) in the pose(s). The example 3D model generator 136 includes a 2D pose detector 216 and a 3D pose calculator 218. For illustration, the 3D pose calculator 218 is used to detect (e.g., predict) the 2D pose(s) of the object(s) in the image data and generate the 3D graphical model(s) of the object(s) in the pose(s). Figure 3-Figure 5 The flowcharts 300, 400, 500 and Figures 6-11 The example models shown are used to discuss the 2D pose detector 216 and the 3D pose calculator 218 . Figure 3-Figure 5 Flowcharts 300 , 400 , 500 represent example algorithms that may be executed by 2D pose detector 216 and / or 3D pose calculator 218 to predict pose(s) of object(s) in image data and generate 3D graphics model(s). Figures 6-11 The model illustrates Figure 2 Example 2D and / or 3D model(s) generated by the 3D model generator 136 when performing 3D pose estimation.

[0043] Figure 3 Flowchart 300 illustrates an overview of the algorithm executed by 2D pose detector 216 and / or 3D pose calculator 218 to perform 3D pose estimation. Figure 3 As shown, Figure 2 The 2D pose detector 218 receives synchronized images from C uncalibrated image capture devices 104, 106, 108, 110 (e.g., video frames). Figure 2The 2D pose detector 216 analyzes the synchronized images and predicts the positions of key points (e.g., joints such as elbows, wrists, knees, etc.) of each object in each synchronized image. The 2D pose detector 216 generates 2D skeleton data for each object in the image based on the prediction of the key point positions of the object and assigns confidence scores to the predicted positions of the key points (block 302). 2D skeleton data and associated confidence scores Used by the 3D pose calculator 218 to generate 3D graphical model(s) (e.g., skeleton) of the object(s) and estimate the parameters Π of the image capture devices 104, 106, 108, 110 that generated the image data 200, 202, 204, 206 used to derive the 3D skeleton data (block 304). As disclosed herein, the 3D skeleton data of the object(s) and the parameters Π of the image capture devices 104, 106, 108, 110 may be provided to, for example, a computer installed on the user device 132 ( Figure 1 ) on the user application(s) 138 for display and / or analysis (block 306).

[0044] Will contact now Figure 2 The example system 100 is discussed in more detail Figure 3 Each block 302 and 304 of the flowchart 300 is shown in FIG. Figure 3 As described at block 302 of , the 2D pose detector 216 analyzes each synchronized image to identify the locations of key points or joints of each object in the image data. In the examples disclosed herein, machine learning is used to improve the efficiency of the 2D pose detector 216 in detecting key points of each object in the image.

[0045] Artificial intelligence (AI), including machine learning (ML), deep learning (DL), and / or other artificial machine-driven logic, enables a machine (e.g., a computer, logic circuit, etc.) to process input data using a model to generate outputs based on patterns and / or associations that the model has previously learned through a training process. For example, the model may be trained with data to recognize patterns and / or associations, and follow such patterns and / or associations when processing input data, so that (one or more) other inputs result in (one or more) outputs consistent with the recognized patterns and / or associations.

[0046] There are many different types of machine learning models and / or machine learning architectures. In the examples disclosed herein, a deep neural network model is used. Generally speaking, a machine learning model / architecture suitable for use in the example methods disclosed herein will be based on supervised learning. However, other types of machine learning models, such as unsupervised learning, may be used in addition or in lieu thereof.

[0047] Generally speaking, implementing an ML / AI system involves two phases, a learning / training phase and an inference phase. In the learning / training phase, a training algorithm is used to train the model to operate according to patterns and / or associations based on, for example, training data. Generally speaking, the model includes internal parameters that guide how input data is transformed into output data, such as by a series of nodes and connections within the model. In addition, hyperparameters are used as part of the training process to control how learning is performed (e.g., learning rate, number of layers to be used in the machine learning model, etc.). Hyperparameters are defined as training parameters that are determined before initiating the training process.

[0048] Different types of training can be performed based on the type and / or expected output of the ML / AI model. For example, supervised training uses inputs and corresponding expected (e.g., labeled) outputs to select parameters for the ML / AI model (e.g., by iterating over combinations of selected parameters) that reduce the model error. As used herein, labeling refers to the expected output of a machine learning model (e.g., a classification, an expected output value, etc.). Alternatively, unsupervised training (e.g., as used in deep learning, a subset of machine learning, etc.) involves inferring patterns from inputs to select parameters for an ML / AI model (e.g., without the benefit of expected (e.g., labeled) outputs).

[0049] In some examples disclosed herein, a training algorithm such as stochastic gradient descent is used to train the ML / AI model. However, any other training algorithm may be used in addition or alternatively. In the examples disclosed herein, training may be performed based on an early stopping principle, where training continues until (one or more) models stop improving. In the examples disclosed herein, it may be performed remotely or locally. In some examples, training may initially be performed remotely. Further training (e.g., retraining) may be performed locally based on data generated as a result of model execution. Training is performed using hyperparameters that control how learning is performed (e.g., learning rate, the number of layers to be used in the machine learning model, etc.). In the examples disclosed herein, hyperparameters include batch size, iteration, number of rounds, optimizer, learning rate, etc. Such hyperparameters are selected by trial and error based on, for example, a specific training data set.

[0050] Training is performed using training data. In examples disclosed herein, the training data is derived from previously generated 2D and / or 3D images that include (one or more) objects in (one or more) different poses. Because supervised training is used, the training data is labeled. In examples disclosed herein, the labels are applied to the training data based on, for example, the locations of key points of (one or more) objects in the image data. In some examples, the training data is subdivided so that a portion of the data is used for validation purposes.

[0051] Once trained, the model(s) are stored in one or more databases (e.g. Figure 2 The one or more models may then be executed by, for example, the 2D pose detector 216 and / or the 3D pose calculator 218, as disclosed below.

[0052] Once trained, the deployed model can be operated to process data in the inference phase. In the inference phase, the data to be analyzed (e.g., live data) is input to the model, and the model executes to create output. This inference phase can be thought of as the AI "thinking" to generate output based on what it has learned from training (e.g., by executing the model to apply the learned patterns and / or associations to live data). In some examples, the input data undergoes preprocessing before being used as input to the machine learning model. Additionally, in some examples, the output data may undergo post-processing after it is generated by the AI model to transform the output into a useful result (e.g., a display of data, instructions to be executed by the machine, etc.).

[0053] In some examples, the output of the deployed model can be captured and provided as feedback. By analyzing the feedback, the accuracy of the deployed model can be determined. If the feedback indicates that the accuracy of the deployed model is below a threshold or other standard, the feedback and updated training dataset, hyperparameters, etc. can be used to trigger training of an updated model to generate an updated deployed model.

[0054] refer to Figure 2 The example system 100 includes a first computing system 220 to train a neural network to detect locations of key points of objects in image data. The example first computing system 220 includes a first neural network processor 222. In the examples disclosed herein, the first neural network processor 222 implements a first neural network.

[0055] Figure 2 The example first computing system 220 includes a first neural network trainer 226. Figure 2The example first neural network trainer 226 performs training on the neural network implemented by the first neural network processor 222. In some examples disclosed herein, the training is performed using a stochastic gradient descent algorithm. However, other methods of training neural networks may also be used in addition or in lieu thereof.

[0056] Figure 2 The example first computing system 220 includes a first training controller 228. The example training controller 228 instructs the first neural network trainer 226 to perform training of the neural network based on the first training data 230. Figure 2 In the example of FIG. 2 , first training data 230 used by first neural network trainer 226 to train the neural network is stored in database 232 . Figure 2 The example database 232 of the illustrated example is implemented by any memory, storage device, and / or storage disk for storing data, such as flash memory, magnetic media, optical media, etc. Furthermore, the data stored in the example database 232 may be in any data format, such as binary data, comma-delimited data, tab-delimited data, Structured Query Language (SQL) structures, image data, etc. Although the database 232 is illustrated as a single element in the illustrated example, the database 232 and / or any other data storage element described herein may be implemented by any number and / or any (one or more) types of memory.

[0057] exist Figure 2 In an example of the present invention, training data 230 may include images generated for training purposes that include object(s) in various poses. In some examples, the training data includes image data streams 200, 202, 204, 206 generated by image capture device(s) 104, 106, 108, 110. The training data 230 is labeled with an (X, Y) joint or keypoint location for each relevant keypoint (e.g., joint) of the object(s) in each pose relative to the coordinate system of each image in the training data. A first neural network trainer 226 trains a neural network implemented by the neural network processor 222 using the training data 230. Based on the locations of the keypoints of the object(s) performing different poses in the training data 230, the first neural network trainer 226 trains the neural network 222 to recognize (e.g., predict) the position of the keypoints of the object(s) performing different poses in the training data 230. The locations of key points of various objects in (eg, images 114 , 116 , 118 , 120 ).

[0058] As a result of the neural network training, a keypoint prediction model 234 is generated. The keypoint prediction model 234 is stored in a database 236. The databases 232 and 236 may be the same storage device or different storage devices.

[0059] Key point prediction model 234 is composed of Figure 2 Specifically, the 2D pose detector 216 performs a 3D pose detector on each synchronized image. (For example, Figure 1 The keypoint prediction model 234 is executed on the images 114, 116, 118, 120) to detect the positions (e.g., (X, Y) coordinate positions) of the keypoints of the object(s) in the images and to generate (one or more) 2D skeletons of the object(s) for each image. (e.g., where k is the index of a particular keypoint of the identified object 102, there are at most J keypoints in total, and p represents the (X, Y) coordinate position of the corresponding keypoint.) In some examples, the 2D pose detector 216 generates a 2D skeleton data S 2d Apply a filter (e.g., a low-pass filter, such as a "one-ohm" filter) to remove noise and smooth the data. 2D skeleton data S 2d For example, the 2D skeleton data 240 is stored in the database 208 in the form of matrix or vector data. In some examples, the 2D skeleton data 240 is used as the first training data 230 for training the neural network 222.

[0060] 2D skeleton data 240 (ie, )'s (XY) coordinates can be plotted to generate (one or more) 2D graphical models. Figure 6 The diagram shows the 2D pose detector 216 based on the Figure 1 2D skeleton data generated for each synchronized image 114, 116, 118, 120 Specifically, Figure 6 The diagram shows the Figure 1 2D skeleton data of the first image 114 The first graphical 2D skeleton 600 of the object 102 is based on the 2D skeleton data of the second image 116 The second graphical 2D skeleton 602 of the object 102 is based on the 2D skeleton data of the third image 118 The third graphical 2D skeleton 604 of the object 102 and the 2D skeleton data based on the fourth image 120 The fourth graphical 2D skeleton 606 of the object 102 is used to generate Figure 6The 2D skeleton data (eg, matrices and / or vector(s)) of the 2D graphics models 600, 602, 604, 606 may be stored as 2D skeleton data 240 in the Figure 2 Additionally or alternatively, in some examples, the graphical images 600 , 602 , 604 , 606 generated from the 2D skeletal data 240 are stored in the database 208 .

[0061] Figure 6 Each of the 2D skeleton models 600, 602, 604, 606 is defined by key points of the object 102, which are identified by the 2D pose detector 216 in the respective images 114, 116, 118, 120 using the key point prediction model 134. For example, the first graphical 2D skeleton 600 includes a first key point 608 at a first (X, Y) position in the coordinate system of the first image capture device 104. The first key point 608 corresponds to Figure 1 The first knee of the subject 102 can be seen in the first image 114 of the second image capture device 108. The second graphical 2D skeleton 602 includes a first keypoint 608 at a first (X, Y) position and a second keypoint 610 at a second (X, Y) position in the coordinate system of the second image capture device 108. The second keypoint 610 corresponds to the first keypoint 608 at the second (X, Y) position. Figure 1 A second knee of subject 102 is visible in second image 116 .

[0062] In the examples disclosed herein, Figure 6 The pelvic joints marked as key points 612 in the second and third graphical 2D skeletons 602 and 604 serve as root joints, or reference joints, for 3D pose estimation. As disclosed herein, the 3D positions of other key points of an object can be identified relative to the pelvic joints of the object.

[0063] Return to Figure 2 , Figure 2 The 2D pose detector 216 calculates a confidence score for each keypoint identified in each image 114, 116, 118, 120, which is used to generate 2D skeleton data 240 for the image (e.g., by Figure 6 For 2D skeleton data S associated with a particular view of the image capture device, 2d , the 2D pose detector 216 generates a confidence score vector Each score Represents 2D skeleton data The joint coordinates of The confidence score represents the confidence that the 2D pose detector 216 has in identifying the location of a particular keypoint of an object in the image. For example, the 2D pose detector 216 may assign a lower confidence score to a keypoint that is occluded in the image. Assign higher confidence scores to key points in the image that are not occluded Wherein a higher confidence score indicates a higher confidence that the particular location estimated for the keypoint is accurate. For example, a keypoint may be occluded or at least partially occluded in an image if the keypoint is substantially not visible in the image due to the field of view of the image capture device that generated the image and / or because the joint is covered by, for example, the subject's clothing.

[0064] In examples where more than one object is identified in an image, the 2D pose detector 216 implements tracking algorithm(s) 242 to assign an identifier to each object identified in the image and subsequent images associated with a particular image data stream 200, 202, 204, 206. For example, the tracking algorithm(s) 242 may use bounding boxes to identify different objects. In such an example, the location of each object in the images of the respective image data streams 200, 202, 204, 206 may be tracked using a unique bounding box assigned to each object. The tracking algorithm(s) 242 may use other types of identifiers to track two or more objects in the image data. The tracking algorithm(s) 242 may be defined by user(s) input and stored in the database 208.

[0065] As disclosed herein, the 2D skeleton data and associated confidence scores generated by the 2D pose detector 216 is provided to the 3D pose calculator 218 for use in generating 3D graphical model(s) of the object(s) in the image data. Figure 4 Flowchart 400 shows the Figure 2 The example algorithm implemented by the 3D pose calculator 218 is used to generate (one or more) 3D graphics models based on the 2D skeleton data 240 generated by the 2D pose detector 216. Specifically, the 3D pose calculator 218 performs a 2D skeleton data Normalize to generate normalized 2D skeleton data (Block 402). The 3D pose detector 218 normalizes the 2D skeleton data so that the graphical representation of the 2D skeleton data for a set of synchronized images (e.g., Figure 6The 2D skeleton models 600, 602, 604, 606) have similar sizes, are substantially aligned with respect to the reference plane, etc. For example, the 3D pose detector 218 may perform the following formula to normalize the 2D skeleton data and obtain normalized 2D joint coordinates:

[0066]

[0067] In Formula 1, the variable λ is a scaling factor used to account for, for example, the distance between the root joint of the object (e.g., the pelvic joint) and the other joints of the object. The scaling factor λ can be defined based on reference data defining the distances between the pelvic joint (e.g., the root joint) and the other joints. The normalized 2D skeleton data is stored in the database 208 as normalized 2D skeleton data 244, for example, in the form of matrix or vector data.

[0068] Figure 7 The diagram shows Figure 2 The 3D pose calculator 218 is based on the 2D skeleton data 240 (e.g., Figure 6 Normalized 2D skeleton data 244 (ie, a graphical representation of the models 600, 602, 604, 606 of FIG. 6B ) generated by ). The 3D pose calculator 218 is based on the Figure 6 The 2D skeleton data S represented by the first graphic 2D skeleton 600 2d The normalized 2D skeleton data obtained Generate a first normalized 2D skeleton 700 of the object 102 based on Figure 6 The 2D skeleton data S represented by the second graphic 2D skeleton 602 2d The normalized 2D skeleton data obtained Generate a second normalized 2D skeleton 702 of the object 102 based on Figure 6 The 2D skeleton data S represented by the third graphic 2D skeleton 606 2d The normalized 2D skeleton data obtained Generate a third graphical normalized 2D skeleton 704 of the object 102 and generate a third graphical normalized 2D skeleton 704 of the object 102 based on the Figure 6 The 2D skeleton data S represented by the fourth graphic 2D skeleton 606 2d The normalized 2D skeleton data obtained Generate a fourth graphical normalized 2D skeleton 706 of the object 102. Figure 6 Compared with the graphical 2D skeletons 600, 602, 604, and 606, the normalized 2D skeleton data Figure 7 Models 700, 702, 704, 706 and Figure 6The models 600, 602, 604, and 606 are substantially similar in size. Figure 7 Normalized 2D skeleton data (e.g., matrices and / or vector(s)) of the normalized 2D graphics models 700, 702, 704, 706 may be stored as normalized 2D skeleton data 244 in Figure 2 Additionally or alternatively, in some examples, the graphical images 700 , 702 , 704 , 706 generated from the normalized 2D skeletal data 244 are stored in the database 208 .

[0069] exist Figure 2-Figure 4 In the example of FIG, normalized 2D skeleton data 244 is used to train a general recurrent neural network that generates a 3D pose prediction model that is used by 3D pose calculator 218 to transform the 2D skeleton data into (one or more) 3D graphics models or (one or more) skeletons (e.g., Figure 4 404).

[0070] Reference again Figure 2 The example system 100 includes a second computing system 246 to train a neural network to map 2D positions (e.g., (X, Y) coordinates) of joints or key points of an object to 3D positions (e.g., (X, Y, Z) coordinates). The example second computing system 246 includes a second neural network processor 248. In the examples disclosed herein, the second neural network processor 248 implements a second neural network.

[0071] Figure 2 The example second computing system 246 includes a second neural network trainer 250 . Figure 2 The example second neural network trainer 250 performs training on the neural network implemented by the second neural network processor 248. In some examples disclosed herein, the training is performed using a stochastic gradient descent algorithm. However, other methods of training neural networks may also be used in addition or in lieu thereof.

[0072] Figure 2 The example second computing system 246 includes a second training controller 252. The example second training controller 252 instructs the second neural network trainer 250 to perform training of the neural network based on the second training data 254. Figure 2 In the example of , second training data 254 used by second neural network trainer 248 to train the neural network is stored in database 256 . Figure 2The example database 256 of the illustrated example is implemented by any memory, storage device, and / or storage disk for storing data, such as flash memory, magnetic media, optical media, etc. Furthermore, the data stored in the example database 256 may be in any data format, such as binary data, comma-delimited data, tab-delimited data, Structured Query Language (SQL) structures, image data, etc. Although the database 256 is illustrated as a single element in the illustrated example, the database 256 and / or any other data storage element described herein may be implemented by any number and / or any type(s) of memories.

[0073] exist Figure 2 In the example of FIG, the training data 254 includes a previously generated 3D human pose dataset. The 3D human pose dataset is labeled with (X, Y, Z) joint positions relative to the 3D coordinate system of each image in the training data 254. For example, pelvis-centered 3D skeleton data in the 3D human pose dataset may define a pelvis joint at coordinates (0, 0, 0), where the coordinates of other joints (e.g., elbow, wrist) are offset relative to the pelvis joint.

[0074] exist Figure 2 In the example of FIG, for each 3D skeleton data centered on the pelvis in the training data 254 The training controller 252 determines the corresponding 2D skeleton data (e.g., via orthogonal projection). The training controller 252 normalizes the 2D skeleton data generated from the 3D human pose dataset to determine the scaling value As a result, the generated and The training data pairs are used for supervised learning of the neural network 248. The training controller 248 can rotate the 3D skeleton data (e.g., by random elevation and / or random azimuth) to generate additional training data.

[0075] The second neural network trainer 250 trains the neural network 248 to identify the respective depth offsets between the pelvic joint (i.e., the root joint) and other joints, wherein the depth offset represents the offset in the depth direction (Z direction) relative to the coordinate system of each image capture device. As a result of the training, a joint depth offset map is generated. where d k Denotes the depth offset from the kth joint to the pelvic joint. Joint depth offset model 258 (ie, depth offset atlas) ) are stored in database 260. Databases 256, 260 may be the same storage device or different storage devices.

[0076] Figure 2The example 3D pose calculator 218 uses the neural network trained joint depth offset model 258 to predict the 3D joint coordinates of key points and generates an initial set of (one or more) 3D graphics models or (one or more) skeletons based on the predicted 3D joint coordinates. Figure 5 Flowchart 500 illustrates 3D pose regression from normalized 2D skeleton data. Figure 5 As shown, the joint depth offset model is generated as a result of the training of the neural network 248 (Block 502). The 3D pose calculator 218 uses the normalized 2D skeleton data and joint depth offset model D to generate predicted scaled 3D skeleton data (eg, via orthographic projection.) A predicted 3D skeleton is generated for each image capture device view, which is used to generate a final 3D skeleton in the world coordinate system defined by one of the image capture device views.

[0077] 3D pose calculator 218 uses a predefined 3D skeleton template Perform the reorientation operation (block 504). Predefined 3D skeleton template Including reference bone lengths (e.g., average bone lengths). The 3D skeleton template can be defined by (one or more) user inputs and stored as skeleton template data 263. Figure 2 Specifically, the 3D pose calculator 218 makes the predicted 3D skeleton data Redirected to So that S 3d With 3D skeleton template The same bone lengths as those of the predicted 3D skeleton data generated from the image data captured by the image capture devices 104, 106, 108, 110 Same bone orientation.

[0078] As a result of the regression analysis of the 3D joint coordinates obtained from the normalized 2D skeleton data and the joint depth offset model 258, the 3D pose calculator 218 creates initial 3D skeleton data for each image in the synchronized image set. Initial 3D skeleton data Represents 2D skeletal data corresponding to the respective views from the image capture devices 104, 106, 108, 110 Results of regression analysis. Initial 3D skeleton data Initial 3D skeleton data 265 is stored in database 208 (eg, as matrix or vector data).

[0079] Figure 8The initial 3D skeleton data 265 (ie, ) from performing a general recurrent neural network (e.g., as described in relation to Figure 5 ) from the view associated with the first image capture device 104 by Figure 7 The first skeleton 700 graphically represents the normalized 2D skeleton data return Figure 1 The 3D pose of the object 102 (eg, ), where i=1). Figure 8 An example 3D skeletal model 800 is shown in the coordinate system of the first image capture device 104. In some examples, the 3D pose calculator 218 generates 2D skeletal data associated with each image capture device view. Generate a graphical representation of the initial 3D skeleton data (e.g., the same as used to generate Figure 7 In some examples, each graphical 3D model created for a particular view using the initial 3D skeleton data displays the same or substantially the same pose, but is rotated according to the view angle associated with the respective image capture device that captured the view. As described above, the initial 3D skeleton data (e.g., matrices and / or vector(s)) used to generate the graphical model 800 may be stored as initial 3D skeleton data 265 in the image capture device. Figure 2 Additionally or alternatively, in some examples, the graphical image(s) 800 generated from the initial 3D skeletal data 265 are stored in the database 208 .

[0080] As disclosed herein, the image capture devices 104, 106, 108, 110 are uncalibrated, and thus, one or more intrinsic and / or extrinsic parameters of the image capture devices 104, 106, 108, 110 are unknown to the 3D model generator 136 and / or change during the generation of the image data. Thus, in some examples, the initial 3D skeleton data 265, 800 may not fully represent the view angle associated with the corresponding image capture device 104, 106, 108, 110. Figure 2 The example 3D model generator 136 includes an image capture device parameter analyzer 264. The image capture device parameter analyzer 264 provides a means for predicting intrinsic and extrinsic parameters of the image capture devices 104, 106, 108, 110, such as the orientation and position of the second, third, and fourth image capture devices 106, 108, 110 relative to a world coordinate system defined by the coordinate system of the first image capture device 104. Figure 2The example 3D model generator 136 uses the predicted parameters of the image capture devices 104 , 106 , 108 , 110 to optimize the 3D skeleton data so that the final 3D skeleton takes into account multiple views of the object captured in the image data generated by each device 104 , 106 , 108 , 110 .

[0081] In the examples disclosed herein, the image capture device parameters may be represented by the variable Π= For each image capture device 104, 106, 108, 110, there are six parameters. Specifically, represents the Euler angle, representing the first image capture device 104 and the i-th th Rotation between image capture devices 106, 108, 110. 1i is a 3D vector, which can be converted into a 3D rotation matrix M with 3x3 elements, (e.g., T = (Tx, Ty)) is a 2D vector representing the translation in the image plane, and s i Is a 1D scalar value representing the scaling factor. In the examples disclosed herein, The variable T in i ,s i represents the intrinsic parameters of the image capture device, and R 1i represents the extrinsic parameters of the image capture device. For example, for a given 3D joint position J3 = (X, Y, Z) in the coordinate space of the first image capture device 104, the 3D joint position J3 to the i-th th The projection of the camera image plane (e.g., the image plane of the second image capture device) can be expressed as where X ′ =R 1i (X) indicates that X is R 1i The new point after rotation, and X ′ xy Represents X ′ The x,y coordinates of .

[0082] Figure 2 The example image capture device parameter analyzer 264 determines the initial value R of the image capture device based on the normalized 2D skeleton data 244 and the initial 3D skeleton data 265. 1i ,T i ,s i (For example, Figure 4 406 of the flowchart 400). For example, Figure 2 The image capture device parameter analyzer 264 uses the regressed initial 3D skeleton data and To calculate R 1iThe image capture device parameter analyzer 264 determines the optimal R by minimizing the following function 1i : The minimization may be performed using, for example, a singular value decomposition (SVD). In some examples, only coordinates corresponding to the subject's torso joints (eg, hips, shoulders) are used for minimization.

[0083] Figure 2 The image capture device parameter analyzer 264 uses the initial 3D skeleton data and normalized 2D skeleton data To determine the scalar value s i In some examples, the image capture device parameter analyzer 264 determines the optimal s by minimizing the following function i :

[0084]

[0085] In some examples, the image capture device parameter analyzer 264 translates the translation variable T i The initial estimate of is set to zero vector. The initial parameter R 1i ,T i ,s i The image capture device parameter data 266 is stored in Figure 2 in the database 208.

[0086] Figure 2 The example 3D pose calculator 218 uses the initial 3D skeleton data 265 and the initial image capture device parameter data 266 to perform an optimization that results in an estimated object (e.g., Figure 1 102) and generates a (eg, final) 3D graphics model of the object in the pose (eg, as Figure 4 As a result of the optimization, the 3D pose calculator 218 generates a value corresponding to the i-th th The 3D skeleton corresponding to the view of the image capture device (e.g., a graphics model). For example, you can use the coordinate system Figure 1 The view of the first image capture device 104 generates a 3D skeleton The optimization performed by the 3D pose calculator 218 serves to combine or fuse the initial 3D skeletons 265 , 800 generated for each view associated with the image capture device to achieve a more accurate final 3D skeleton.

[0087] Figure 2The 3D pose calculator 218 uses a 3D skeleton template (For example, ) to perform the optimization. Figure 9 A graphical representation of an example 3D skeleton template 900 or kinematic tree is shown, wherein the skeleton is shown in a static pose, represented by Figure 9 The positions of the 17 labeled key points or joints in the . These joints define the nodes in the kinematic tree. Figure 9 As shown, 3D skeleton template 900 can be defined for an individual having a height of, for example, 170 cm. The corresponding bone lengths of 3D skeleton template 900 can be generated based on average anthropometric statistics. However, if the height of the subject of interest (e.g., subject 102) is known, 3D skeleton template 900 can be adjusted based on the known body measurements. 3D skeleton template 900 is defined by user input and stored as skeleton template data 263. Figure 2 In some examples, the 3D skeleton template 900 is a graphic (e.g., as Figure 9 The image data shown) or matrix and / or vector data is stored.

[0088] In the examples disclosed herein, the 3D skeleton template 900 is used to represent the object of interest using a set of skeleton joint rotation vectors θ instead of 3D joint coordinates (e.g., Figure 1 3D skeleton of the object 102). Figure 10 A posed 3D skeleton 1000 is shown, where Figure 9 The joints of the 3D skeleton template 900 have been rotated by the skeleton joint rotation vector θ to position the 3D skeleton into a certain pose. In other words, Figure 10 The new joint position in is obtained by rotating Figure 9 In the examples disclosed herein, when the 3D pose calculator 218 performs optimization, the skeleton joint rotation vector θ is defined as the optimization parameter instead of the 3D joint position because the joint rotation vector θ takes into account the motion distribution that is independent of the variant bone length.

[0089] exist Figure 10 In the example, the skeleton joint rotation vector θ can be defined as: in An axis-angle representation of the relative rotation of bone part k relative to the parent of that particular bone part in the kinematic tree. For example, a skeleton joint rotation vector θ may define the rotation of the bone extending between joints 5 and 6 relative to the bone extending between joints 4 and 5, as Figure 9 and 10 (e.g., the rotation at joint 5). In this example, the variable θ* can be defined as the zero vector, so that S(θ * ) is equivalent to Figure 9 Skeleton template shown

[0090] Figure 2 The example 3D pose calculator 218 uses the initial 3D skeleton data predicted by the regression neural network (e.g., initial 3D skeleton data 265) to determine the corresponding initial skeleton joint rotation vector θ init , so that the 3D skeleton template The bone direction S(θ init ) and the initial 3D skeleton data The bone directions in are basically the same, where The i in _ corresponds to the view of the first image capture device 104 (ie, i=1). In this example, the 3D pose calculator 218 uses the normalized 2D skeleton data by reducing (eg, minimizing) a nonlinear least squares objective function Initial 3D skeleton data and the initial image capture device parameters π init , to solve the final (eg, optimal) 3D skeleton for the view associated with the first image capture device 104 and the final (e.g., optimal) image capture device parameters Π. Specifically, using the initial 3D pose parameters θ init and initial image capture device parameters π init , the 3D pose calculator 218 determines the optimal skeleton joint rotation vector θ and the optimal image device capture parameter Π by minimizing the following objective function.

[0091] E(θ,Π)=E joint (θ,Π)+E init (θ,Π)+w1*E prior (θ)+E temporal (θ, Π) (Formula 2);

[0092]

[0093] E prior (θ) = -log(G(θ;μ θ ,Σ)) (Formula 5);

[0094] E temporal (θ,Π)=w4*‖(S 3d (θ))-S 3d (θ t-1 ))‖2+w5*‖Π-Π t-1 ‖2 (Formula 6).

[0095] In the above formula, E joint A skeleton view of all views associated with each image capture device 104, 106, 108, 110 is defined. Figure 1 The distance between the projected joint position and the corresponding estimated joint position is measured; E init The difference between the optimized parameters and the initial parameters is measured; G(θ; μ θ ,Σ) is a Gaussian function that defines a plausible pose as a probability distribution of known poses (e.g., determined based on known 3D human pose data (e.g., training data 254)); E prior Acts as a constraint so that the optimization favors possible poses over impossible poses (e.g., favors poses where the subject's leg is bent and the subject's foot moves backward rather than forward during leg bending); E temporal represents the parameter change in the time domain; w1, w2, w3, w4, w5 are scalar weights that can be adjusted to tune the optimization. When performing the optimization, the 3D pose calculator 218 takes into account the confidence scores assigned to the predicted 2D keypoint positions by the 2D pose detector 216. For example, the optimized 3D skeleton depends less on 2D joints with lower confidence scores than on 2D joints associated with higher confidence scores. Figure 2 The post-processed 3D skeleton data 270 generated as a result of the optimization may be stored in the database 208 and used, for example, as training data 230 , 254 for a neural network.

[0096] Figure 11 Shown as by Figure 2 The example graphical model 1110 of the post-processed 3D skeleton is generated as a result of an optimization performed by the 3D pose calculator 218 of the embodiment of the present invention, using the normalized 2D skeleton data 244 (e.g., Figure 7 , 702, 704, 706 graphical representations of the models 700, 702, 704, 706), initial 3D skeleton data 265 (eg, where the initial 3D skeleton data associated with the view of the first image capture device 104 is represented by Figure 8 Model 800 graphically represented), and by Figure 9 Skeleton template of the Model 900 graphical representation Figure 11 The example final (eg, optimal) 3D graphics skeleton 1100 is in the coordinate system of the first image capture device 104 (eg, ) is shown in or projected onto the world coordinate system defined by .

[0097] The examples disclosed herein are discussed in relation to a world coordinate system defined by the coordinate system of the first image capture device 104, whereby the graphics model(s) of the post-processing 3D skeleton(s) are projected onto the world coordinate system defined by the first image capture device 104, e.g. Figure 11 However, in other examples, the coordinate system of the other image capture devices 106, 108, 110 may be selected as the world coordinate system. For example, if the coordinate system of the second image capture device 106 is selected as the world coordinate system, the rotation between the first, third, and fourth image capture devices 104, 106, 110 (e.g., determined by the image capture device parameter analyzer 264) may be used to transform the 3D pose of the object based on the rotation in the world coordinate system defined by the second image capture device 106.

[0098] Figure 2 The example 3D model generator 136 includes a communicator 272. The communicator 272 provides a means for outputting the post-processed 3D skeleton data 270 to, for example, Figure 1 The post-processed 3D skeleton data 270 may be used for applications such as biomechanical analysis of athletes, 3D character animation for movies and games, human-computer interaction analysis, and / or human behavior recognition analysis. Figure 2 The communicator 272 may output the post-processed 3D skeletal data 270 via one or more wired or wireless communication protocols.

[0099] Although Figure 2 The diagram shows the implementation Figure 1 The example method of the 3D model generator 136 is as follows: Figure 2 One or more of the elements, processes, and / or devices shown in the examples may be combined, divided, rearranged, omitted, eliminated, and / or implemented in any other manner. Additionally, the example database 208, the example image capture device controller 210, the image synchronizer 214, the example 2D pose detector 216, the example 3D pose calculator 218, the example image capture device parameter analyzer 264, the example communicator 272, and / or more generally, Figure 2The example 3D model generator 136 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, any of the example database 208, the example image capture device controller 210, the image synchronizer 214, the example 2D pose detector 216, the example 3D pose calculator 218, the example image capture device parameter analyzer 264, the example communicator 272, and / or more generally the example 3D model generator 136 may be implemented by one or more analog or digital circuits, logic circuits, programmable processor(s), programmable controller(s), graphics processing unit(s) (GPUs), digital signal processor(s) (DSPs), application specific integrated circuit(s) (ASICs), programmable logic device(s) (PLDs), and / or field programmable logic device(s) (FPLDs). When any apparatus or system claim of this patent is read to cover a pure software and / or firmware implementation, at least one of the example database 208, the example image capture device controller 210, the image synchronizer 214, the example 2D pose detector 216, the example 3D pose calculator 218, the example image capture device parameter analyzer 264, and / or the example communicator 272 is expressly defined herein to include a non-transitory computer-readable storage device or storage disk, such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc., containing the software and / or firmware. Furthermore, Figure 1 and / or Figure 2 An example 3D model generator 136 may include, in addition to Figure 2 In addition to or in place of those shown in Figure 2 , and / or may include more than one of the illustrated elements, processes, and / or devices, and / or may include more than one of the illustrated elements, processes, and devices. As used herein, the phrase "in communication with" - including variations thereof - encompasses direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or constant communication, but also includes selective communication at periodic intervals, scheduled intervals, non-periodic intervals, and / or one-time events.

[0100] Although Figure 2An example way to implement the first computing system 220 is illustrated in FIG. Figure 2 One or more of the elements, processes, and / or devices shown in the examples may be combined, divided, rearranged, omitted, eliminated, and / or implemented in any other manner. Additionally, the example neural network processor 222, the example trainer 226, the example training controller 228, the example database(s) 232, 236, and / or more generally, Figure 2 The example first computing system 220 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, the example neural network processor 222, the example trainer 226, the example training controller 228, the example database(s) 232, 236, and / or more generally any of the example first computing system 220 may be implemented by one or more analog or digital circuits, logic circuits, programmable processor(s), programmable controller(s), graphics processing unit(s) (GPUs), digital signal processor(s) (DSPs), application specific integrated circuit(s) (ASICs), programmable logic device(s) (PLDs), and / or field programmable logic device(s) (FPLDs). When any apparatus or system claim of this patent is read to cover a pure software and / or firmware implementation, at least one of the example neural network processor 222, the example trainer 226, the example training controller 228, and / or the example database(s) 232, 236 is expressly defined herein as comprising a non-transitory computer-readable storage device or storage disk, such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc., containing the software and / or firmware. In addition, the example first computing system 220 may include, in addition to Figure 2 In addition to or in place of those shown in Figure 2 , and / or may include more than one of the illustrated elements, processes, and / or devices. As used herein, the phrase "in communication with"—including variations thereof—encompasses direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or constant communication, but also includes selective communication at periodic intervals, scheduled intervals, non-periodic intervals, and / or one-time events.

[0101] Although Figure 2 An example manner of implementing the second computing system 246 is illustrated in FIG. Figure 2One or more of the elements, processes, and / or devices shown in the examples may be combined, divided, rearranged, omitted, eliminated, and / or implemented in any other manner. Additionally, the example neural network processor 248, the example trainer 250, the example training controller 252, the example database(s) 256, 260, and / or more generally, Figure 2 The example second computing system 246 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, the example neural network processor 248, the example trainer 250, the example training controller 252, the example database(s) 256, 260, and / or more generally the example second computing system 246 may be implemented by one or more analog or digital circuits, logic circuits, programmable processor(s), programmable controller(s), graphics processing unit(s) (GPUs), digital signal processor(s) (DSPs), application specific integrated circuit(s) (ASICs), programmable logic device(s) (PLDs), and / or field programmable logic device(s) (FPLDs). When any apparatus or system claim of this patent is read to cover a pure software and / or firmware implementation, at least one of the example neural network processor 248, the example trainer 250, the example training controller 252, and / or the example database(s) 256, 260 is expressly defined herein as comprising a non-transitory computer-readable storage device or storage disk, such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc., containing the software and / or firmware. In addition, the example second computing system 246 may include, in addition to Figure 2 In addition to or in place of those shown in Figure 2 , and / or may include more than one of the illustrated elements, processes, and / or devices. As used herein, the phrase "in communication with"—including variations thereof—encompasses direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or constant communication, but also includes selective communication at periodic intervals, scheduled intervals, non-periodic intervals, and / or one-time events.

[0102] exist Figure 12 The following table shows a representative method for implementing Figure 2 Flowchart of example hardware logic, machine readable instructions, hardware implemented state machines, and / or any combination thereof for an example first computing system 220. Figure 13 The following table shows a representative method for implementing Figure 2A flowchart of example hardware logic, machine readable instructions, hardware implemented state machines, and / or any combination thereof for an example second computing system 246. Figure 14 The following table shows a representative method for implementing Figure 1 and / or Figure 2 A flowchart of example hardware logic, machine-readable instructions, hardware-implemented state machines, and / or any combination thereof for an example 3D model generator 136 is provided. Figure 12-14 The machine-readable instructions may be one or more executable programs or portions of executable programs for execution by a computer processor and / or processor circuit, such as the one described below. Figure 15-17 The processors 1512, 1612, 1712 shown in the example processor platforms 1500, 1600, 1700 discussed herein may be embodied in software stored on a non-transitory computer-readable storage medium such as a CD-ROM, floppy disk, hard drive, DVD, Blu-ray disc, or memory associated with the processor(s) 1512, 1612, 1712, but the entire program and / or portions thereof may alternatively be executed by a device other than the processor(s) 1512, 1612, 1712 and / or embodied in firmware or dedicated hardware. Additionally, while reference is made to Figure 12-14 The example program is described using the flowcharts shown in , but many other methods of implementing the example first computing system 220, the example second computing system 246, and / or the example 3D model generator 136 may be used instead. For example, the order of execution of the blocks may be changed, and / or some of the described blocks may be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) that are constructed to perform the corresponding operations without executing software or firmware. The processor circuits may be distributed in different network locations and / or locally on one or more devices (e.g., a multi-core processor in a single machine, multiple processors distributed on a server rack, etc.).

[0103] The machine-readable instructions described herein may be stored in one or more formats including compressed, encrypted, segmented, compiled, executable, packaged, and the like. The machine-readable instructions described herein may be stored as data or data structures (e.g., portions of instructions, code, representations of code, and the like) that can be utilized to create, manufacture, and / or produce machine-executable instructions. For example, the machine-readable instructions may be segmented and stored on one or more storage devices and / or computing devices (e.g., servers) located at the same or different locations on a network or a collection of networks (e.g., in the cloud, in an edge device, and the like). The machine-readable instructions may require one or more of installation, modification, adaptation, updating, combination, supplementation, configuration, decryption, decompression, unpacking, distribution, reassignment, compilation, and the like so that they can be directly readable, interpretable, and / or executable by a computing device and / or other machine. For example, the machine-readable instructions may be stored as multiple parts that are individually compressed, encrypted, and stored on separate computing devices, where the parts, when decrypted, decompressed, and combined, form a set of executable instructions that implement one or more functions that together form a program such as that described herein.

[0104] In another example, machine-readable instructions may be stored in a state in which they can be read by processor circuitry, but require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc. in order to execute the instructions on a particular computing device or other device. In another example, the machine-readable instructions and / or corresponding program(s) may need to be configured (e.g., to store settings, input data, record network addresses, etc.) before the machine-readable instructions and / or corresponding program(s) can be executed in whole or in part. Thus, as used herein, a machine-readable medium may include machine-readable instructions and / or program(s) regardless of the particular format or state of the machine-readable instructions and / or program(s) when stored or otherwise at rest or in transit.

[0105] The machine-readable instructions described herein may be represented by any past, present, or future instruction language, scripting language, programming language, etc. For example, the machine-readable instructions may be represented by any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, HyperText Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0106] As described above, the present invention may be implemented using executable instructions (eg, computer and / or machine readable instructions) stored on a non-transitory computer and / or machine readable medium. Figure 12-14 The example process of a medium such as a hard drive, flash memory, read-only memory, compact disk, digital versatile disk, cache, random access memory, and / or any other storage device or storage disk in which information can be stored for any duration (e.g., for an extended period of time, permanent storage, transient storage, for temporary buffering, and / or for caching of information). As used herein, the term non-transitory computer-readable medium is expressly defined to include any type of computer-readable storage device and / or storage disk and to exclude propagating signals and to exclude transmission media.

[0107] "Include" and "comprising" (and all forms and tenses thereof) are used as opening terms in this document. Thus, whenever a claim adopts any form of "include" or "comprising" (e.g., includes, contains, has, etc.) as a preamble or used in any type of claim recitation, it is to be understood that additional elements, terms, etc. may be present without falling outside the scope of the corresponding claim or recitation. As used herein, when the phrase "at least" is used as a transition term, such as in the preamble of a claim, it is opening in the same way that the terms "include" and "comprising" are opening. The term "and / or" when used, for example, in the form of A, B, and / or C, refers to any combination or subset of A, B, and C, such as (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, (6) B and C, and (7) A and B and C. As used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A and B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A or B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. As used herein in the context of describing the execution or execution of processes, instructions, acts, activities, and / or steps, the phrase “at least one of A and B” is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein in the context of describing the execution or execution of processes, instructions, actions, activities and / or steps, the phrase "at least one of A or B" is intended to refer to implementations that include any of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B.

[0108] As used herein, singular references (e.g., "a," "a," "a first," "a second," etc.) do not exclude a plurality. As used herein, the term "a" or "an" entity refers to one or more of that entity. The terms "a," "one," "one or more," and "at least one" may be used interchangeably herein. Furthermore, although listed separately, a plurality of devices, elements, or method actions may be implemented by, for example, a single unit or processor. Furthermore, although individual features may be included in different examples or claims, they may be combined, and inclusion in different examples or claims does not imply that a combination of features is not feasible and / or advantageous.

[0109] Figure 12 is a flow chart representing example machine-readable instructions 1200 that, when executed Figure 2 When executed by the example first computing system 220 , the example first computing system 220 trains a neural network to identify locations of key points or joints of an object in 2D image data. Figure 12 The example instruction 1200 is Figure 2 When executed by the first computing system 220, a neural network and / or its model is generated, which can be distributed to other computing systems, such as Figure 2 2D pose detector 216 .

[0110] Figure 12 The example instructions 1200 begin with the training controller 228 accessing 2D object pose image data stored in the example database 232 (block 1201). The 2D object pose image data may include previously generated image data of (one or more objects) in (one or more) different poses.

[0111] The example training controller 228 labels the locations (e.g., (X, Y) coordinates) of key points in the 2D object pose image data based on, for example, image recognition techniques that enable the training controller 228 to identify joints in the image data (block 1202). The example training controller 228 generates training data 230 based on the locations of the key points in the labeled image data (block 1204).

[0112] The example training controller 228 instructs the neural network trainer 226 to perform training of the neural network 222 using the training data 230 (block 1206). Figure 12 In the example of , the training is based on supervised learning. As a result of the training, a keypoint prediction model 234 is generated (block 1208). Based on the keypoint prediction model 234, the neural network is trained to predict the 2D (X, Y) positions of keypoints of objects in the image data. The keypoint prediction model 234 can be stored in the database 236 for access by the 2D pose detector 216 of the 3D model generator 136. Figure 12 The example instructions 1200 end when no additional training (eg, retraining) is performed (blocks 1210, 1212).

[0113] Figure 13 is a flow chart representing example machine-readable instructions 1300 that, when executed Figure 2 When executed by the example second computing system 246, the example second computing system 246 trains a neural network to learn a mapping from normalized 2D skeleton data including 2D joint coordinates to 3D joint coordinates using a depth offset map, where the depth offset map indicates a depth offset from a joint to a root joint (e.g., a pelvic joint). Figure 13 The example instruction 1300 when Figure 2 When executed by the second computing system 246, a neural network and / or its model is generated, which can be distributed to other computing systems, such as Figure 2 The 3D pose calculator 218 .

[0114] Figure 13 The example begins with the training controller 252 of the second computing system 246 accessing 3D object pose image data stored in the example database 256 (block 1301). The 3D object pose image data may include previously generated 3D image data including (one or more) objects in (one or more) different poses.

[0115] The example training controller 252 determines 2D skeleton data for each ground truth pelvis-centered 3D skeleton data (e.g., where the pelvis joint is the root joint and the ground truth refers to the known joint positions relative to the pelvis joint) (block 1302). Figure 13 , the training controller 252 normalizes the 2D skeleton data to obtain the scaling factor(s). The example training controller 252 generates training data 254 based on the 2D normalized skeleton and the scaling factor(s) (block 1304).

[0116] The example training controller 252 instructs the neural network trainer 250 to perform training of the neural network 248 using the training data 254 (block 1306). Figure 13 In the example of , the training is based on supervised learning. As a result of the training, a joint depth offset model 258 is generated (block 1308). Based on the joint depth offset model 258, the neural network is trained to learn the mapping between the joint coordinates in the normalized 2D skeleton data and the depth offsets of the joints to predict the 3D joint positions. The joint depth offset model 258 can be stored in the database 256 for access by the 3D pose calculator 218 of the 3D model generator 136. Figure 13 The example instructions 1300 end when no additional training (eg, retraining) is performed (blocks 1310, 1312).

[0117] Figure 14 is a flow chart representing example machine-readable instructions 1400 that, when executed Figure 1 and / or Figure 2 When the 3D model generator 136 is executed, the 3D model generator 136 generates a 3D model based on the image captured by two or more uncalibrated image capture devices (e.g., Figure 1 The 3D graphics model (eg, skeleton) of the object in a particular pose is generated based on image data generated by the image capture devices 104, 106, 108, 110).

[0118] Figure 14 The example instructions 1400 begin with the 3D model generator 136 accessing image data 200 , 202 , 204 , 206 generated by an image capture device 104 , 106 , 108 , 110 (block 1401 ). Figure 4 The image synchronizer 214 extracts images from each image data stream 200, 202, 204, 206 and synchronizes the images from each image data stream 200, 202, 204, 206 based on time to generate a set (e.g., a first set) of synchronized images (block 1402). The image synchronizer 214 can synchronize the images based on, for example, a timestamp associated with each image.

[0119] exist Figure 14 In the example, Figure 2 The 2D pose detector 216 generates 2D skeleton data 240 for the object in the set of synchronized images (block 1404). The 2D pose detector 216 uses a key point prediction model 234 generated by a neural network (e.g., Figure 12 ) to predict the positions (e.g., (X, Y) coordinates) of the key points or joints of each object in the image. In some examples, the 2D pose detector 216 generates a 2D graphical model or skeleton 600, 602, 604, 606 defined by the predicted positions of the key points. The 2D skeleton data 240, 600, 602, 604, 606 is stored in Figure 2 in the database 208.

[0120] exist Figure 14 In the example, Figure 2 The 3D pose calculator 218 generates the 2D pose detector 216 Figure 2 and Figure 6 The 2D skeleton data 240, 600, 602, 604, 606 are normalized (block 1406). Figure 2 Normalized 2D skeleton data 244 and / or generated therefrom Figure 7 The graphical representations 700 , 702 , 704 , 706 are stored in the database 208 .

[0121] The 3D pose calculator 218 performs a neural network regression analysis to generate initial 3D skeleton data for the object (block 1408). For example, the 3D pose calculator 218 uses the normalized 2D skeleton data 244, 700, 702, 704, 706, the joint depth offset model 258 generated by the training of the neural network 248 (e.g., Figure 13 ) and a 3D skeleton template to predict the 3D joint coordinates ((X, Y, Z) coordinates) of the key points of the object in the image (for example, as shown in the flowchart of Figure 4 and Figure 5 400, 500 disclosed in FIG. 3D pose calculator 218 generates a Figure 2 and Figure 8 Initial 3D skeleton data 265,800.

[0122] Figure 14 The example instructions take into account parameters of the uncalibrated image capture device 104, 106, 108, 110 that generated the image data, such as intrinsic parameters of the device (e.g., image plane translation) and extrinsic parameters (e.g., orientation and position) (block 1410). Figure 14 In the example of , the image capture device parameter analyzer 264 estimates parameters Π={R,T,s} of the image capture devices 104, 106, 108, 110, where the parameters R and T refer to a coordinate system transformation (e.g., rotation, translation), and s is a scalar factor.

[0123] The 3D pose calculator 218 performs post-processing on the initial 3D skeleton data 265, 800 to improve the accuracy of the 3D pose represented by the 3D graphics model (block 1412). The 3D pose calculator 218 can perform post-processing by performing an optimization to identify the optimal skeleton joint rotation vectors θ and the optimal image device capture parameters π, as disclosed above in connection with equations 2-6. As a result of the optimization, the 3D pose calculator 218 generates a final 3D graphics model 1100, which is projected into a world coordinate system corresponding to the coordinate system of one of the image capture devices 104, 106, 108, 110 (e.g., the coordinate system associated with the first image capture device 104).

[0124] Figure 2 The communicator 272 transmits the post-processed 3D graphics model or skeleton to, for example, a computer system installed on a computer system via one or more wired or wireless communication protocols. Figure 1 (one or more) user applications 138 on the user device 132 (block 1414).

[0125] The example instructions of blocks 1402-1412 may be repeated for each object in the image data generated by the image capture device (block 1416). In such an example, at block 1404, the 2D pose detector 216 assigns an object tracker or identifier (e.g., a bounding box) to each object in the synchronized image to track the respective object in each image and / or in subsequent images generated by the image capture device(s) 104, 106, 108, 110.

[0126] When there is no further image data to analyze, Figure 14 The example instructions 1400 end (blocks 1418, 1420).

[0127] Figure 15 is constructed to execute Figure 12 Instructions to achieve Figure 2 The processor platform 1500 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smartphone, an iPad, etc.). TM tablet devices such as tablets), personal digital assistants (PDAs), Internet appliances, or any other type of computing device.

[0128] The illustrated example processor platform 1500 includes a processor 1512. The illustrated example processor 1512 is hardware. For example, the processor 1512 may be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements the example neural network processor 222, the example trainer 226, and the example training controller 228.

[0129] The processor 1512 of the illustrated example includes a local memory 1513 (e.g., cache). The processor 1512 of the illustrated example communicates with a main memory including a volatile memory 1514 and a non-volatile memory 1516 via a bus 1518. The volatile memory 1514 may be a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory (DRAM) Dynamic Random Access Memory, ) and / or any other type of random access memory device. The non-volatile memory 1516 can be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1514, 1516 is controlled by a memory controller.

[0130] The processor platform 1500 of the illustrated example further includes an interface circuit 1520. The interface circuit 1520 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), interface, a near field communication (NFC) interface, and / or a PCI Express interface.

[0131] In the illustrated example, one or more input devices 1522 are connected to the interface circuit 1520. The input device(s) 1522 allow a user to enter data and / or commands into the processor 1512. The input device(s) may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a trackpad, a trackball, isopoint, and / or a voice recognition system.

[0132] One or more output devices 1524 are also connected to the interface circuit 1520 of the illustrated example. Output device 1524 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT), an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. The interface circuit 1520 of the illustrated example thus typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0133] The interface circuitry 1520 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces, to facilitate data exchange with external machines (e.g., any type of computing device) via a network 1526. Communication can be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-sight wireless system, a cellular telephone system, and the like.

[0134] The processor platform 1500 of the illustrated example also includes one or more mass storage devices 1528 for storing software and / or data. Examples of such mass storage devices 1528 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0135] Figure 12The machine-executable instructions 1532 may be stored in the mass storage device 1528, in the volatile memory 1514, in the non-volatile memory 1516, and / or on a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0136] Figure 16 is constructed to execute Figure 13 Instructions to achieve Figure 2 The processor platform 1600 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smartphone, an iPad, etc.). TM tablet devices such as ), personal digital assistants (PDAs), Internet appliances, or any other type of computing device.

[0137] The illustrated example processor platform 1600 includes a processor 1612. The illustrated example processor 1612 is hardware. For example, the processor 1612 may be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements the example neural network processor 248, the example trainer 250, and the example training controller 252.

[0138] The processor 1612 of the illustrated example includes a local memory 1613 (e.g., cache). The processor 1612 of the illustrated example communicates with a main memory including a volatile memory 1614 and a non-volatile memory 1616 via a bus 1618. The volatile memory 1614 may be a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), or a non-volatile memory. Dynamic Random Access Memory (DRAM) Dynamic Random Access Memory, ) and / or any other type of random access memory device. The non-volatile memory 1616 can be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1614, 1616 is controlled by a memory controller.

[0139] The processor platform 1600 of the illustrated example further includes an interface circuit 1620. The interface circuit 1620 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), interface, a near field communication (NFC) interface, and / or a PCI Express interface.

[0140] In the illustrated example, one or more input devices 1622 are connected to the interface circuit 1620. The input device(s) 1622 allow a user to enter data and / or commands into the processor 1612. The input device(s) may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a trackpad, a trackball, isopoint, and / or a voice recognition system.

[0141] One or more output devices 1624 are also connected to the interface circuit 1620 of the illustrated example. Output device 1624 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT), an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. The interface circuit 1620 of the illustrated example thus typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0142] The interface circuitry 1620 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces, to facilitate data exchange with external machines (e.g., any type of computing device) via a network 1626. Communication can be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-sight wireless system, a cellular telephone system, and the like.

[0143] The processor platform 1600 of the illustrated example also includes one or more mass storage devices 1628 for storing software and / or data. Examples of such mass storage devices 1628 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0144] Figure 13 The machine-executable instructions 1632 may be stored in the mass storage device 1628, in the volatile memory 1614, in the non-volatile memory 1616, and / or on a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0145] Figure 17 is constructed to execute Figure 14 Instructions to achieve Figure 1 and / or Figure 2 The processor platform 1700 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cell phone, a smartphone, an iPad, etc.). TM tablet devices such as ), personal digital assistants (PDAs), Internet appliances, or any other type of computing device.

[0146] The processor platform 1700 of the illustrated example includes a processor 1712. The processor 1712 of the illustrated example is hardware. For example, the processor 1712 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements the example image capture device controller 210, the image synchronizer 214, the example 2D pose detector 216, the example 3D pose calculator 218, the example image capture device parameter analyzer 264, and the example communicator 272.

[0147] The processor 1712 of the illustrated example includes a local memory 1713 (e.g., cache). The processor 1712 of the illustrated example communicates with a main memory including a volatile memory 1714 and a non-volatile memory 1716 via a bus 1718. The volatile memory 1714 may be a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory (DRAM) Dynamic Random Access Memory, ) and / or any other type of random access memory device. The non-volatile memory 1716 can be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1714, 1716 is controlled by a memory controller.

[0148] The processor platform 1700 of the illustrated example further includes an interface circuit 1720. The interface circuit 1720 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), interface, a near field communication (NFC) interface, and / or a PCI Express interface.

[0149] In the illustrated example, one or more input devices 1722 are connected to the interface circuit 1720. The input device(s) 1722 allow a user to enter data and / or commands into the processor 1712. The input device(s) may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a trackpad, a trackball, isopoint, and / or a voice recognition system.

[0150] One or more output devices 1724 are also connected to the interface circuit 1720 of the illustrated example. Output device 1724 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT), an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. The interface circuit 1720 of the illustrated example thus typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0151] The interface circuitry 1720 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces, to facilitate data exchange with external machines (e.g., any type of computing device) via a network 1726. Communication can be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-sight wireless system, a cellular telephone system, and the like.

[0152] The processor platform 1700 of the illustrated example also includes one or more mass storage devices 1728 for storing software and / or data. Examples of such mass storage devices 1728 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0153] Figure 14 The machine-executable instructions 1732 may be stored in the mass storage device 1728, in the volatile memory 1714, in the non-volatile memory 1716, and / or on a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0154] exist Figure 18 A diagram illustrating a method for distributing information such as Figure 15 Example computer readable instructions 1532, Figure 16 Example computer readable instructions 1632 and / or Figure 17The example software distribution platform 1805 may be implemented by any computer server, data infrastructure, cloud service, etc. that can store software and transfer it to other computing devices. A third party may be a customer of the entity that owns and / or operates the software distribution platform. For example, the entity that owns and / or operates the software distribution platform may be a software (e.g., Figure 15 、 Figure 16 and / or Figure 17 The third party may be a consumer, user, retailer, OEM, etc. who purchases and / or licenses the software for use and / or resale and / or sublicense. In the illustrated example, the software distribution platform 1805 includes one or more servers and one or more storage devices. The storage device stores the various computer readable instructions 1532, 1632, 1732, which may correspond to the aforementioned Figure 12 Example computer readable instructions 1200, Figure 13 Example computer readable instructions 1300 or Figure 14 The example computer readable instructions 1400 of the example software distribution platform 1805 are provided. One or more servers of the example software distribution platform 1805 are in communication with a network 1810, which may correspond to any one or more of the Internet and / or any of the example networks 1726 described above. In some examples, as part of a commercial transaction, one or more servers respond to a request to transfer software to a requesting party. Payment for the delivery, sale, and / or license of the software may be processed by one or more servers of the software distribution platform and / or via a third-party payment entity. These servers enable purchasers and / or licensors to download computer readable instructions 1732 from the software distribution platform 1805. For example, Figure 14 Software corresponding to the example computer readable instructions 1400 may be downloaded to the example processor platform 1700, which is to execute the computer readable instructions 1732 to implement Figure 1 and / or Figure 2 In some examples, one or more servers of the software distribution platform 1805 periodically provide, transmit, and / or force updates to the software (e.g., Figure 17 1732) to ensure that improvements, patches, updates, etc. are distributed and applied to the software at the end-user device.

[0155] From the foregoing it will be apparent that example methods, apparatus, and articles of manufacture have been disclosed that provide 3D pose estimation of an object from image data generated by an image capture device without the need to perform calibration of the image capture device. The examples disclosed herein execute a neural network model to identify 2D key points or joints in the image data. The examples disclosed herein predict 3D coordinates of joints using neural network regression analysis to generate (one or more) 3D graphical models of an object in a certain pose. The (one or more) 3D graphical models are refined via optimization that takes into account parameters of the image capture device to generate (one or more) 3D graphical models that accurately represent the pose of the object captured by the multi-view image capture device. The examples disclosed herein can be implemented in environments such as sporting events where the image capture device is operated with varying zoom levels and / or varying device positions to identify the pose of an object under dynamic conditions.

[0156] This document discloses example methods, apparatuses, systems, and articles of manufacture for implementing three-dimensional pose estimation. Further examples and combinations thereof include the following:

[0157] Example 1 includes an apparatus comprising an image synchronizer for synchronizing a first image generated by a first image capture device and a second image generated by a second image capture device based on time, wherein the first image includes an object and the second image includes the object; a two-dimensional posture detector for: predicting first positions of key points of the object based on the first image and by executing a first neural network model to generate first two-dimensional data; and predicting second positions of key points of the object based on the second image and by executing the first neural network model to generate second two-dimensional data; and a three-dimensional posture calculator for generating a three-dimensional graphic model representing the posture of the object in the first image and the second image based on the first two-dimensional data and the second two-dimensional data and by executing a second neural network model.

[0158] Example 2 includes the apparatus defined in Example 1, further comprising an image capture device parameter analyzer for predicting parameters of the first image capture device, the three-dimensional pose calculator for generating the three-dimensional model based on the parameters of the first image capture device.

[0159] Example 3 includes the apparatus as defined in Example 2, wherein the parameter is associated with a rotation or translation of the first image capture device relative to a coordinate system.

[0160] Example 4 includes an apparatus as defined in Example 1 or 2, wherein the second neural network model implements a regression function.

[0161] Example 5 includes the apparatus as defined in Example 4, wherein the three-dimensional pose calculator is to predict the three-dimensional coordinates of the first joint of the object based on the regression function.

[0162] Example 6 includes the apparatus as defined in Example 1, wherein the three-dimensional pose calculator is to perform optimization to generate the three-dimensional model.

[0163] Example 7 includes an apparatus as defined in Example 1, 2, or 5, wherein at least one of the first image capture device or the second image capture device is an uncalibrated image capture device.

[0164] Example 8 includes an apparatus as defined in Example 1, wherein the three-dimensional pose calculator is used to normalize the first two-dimensional data to generate first normalized two-dimensional data and normalize the second two-dimensional data to generate second normalized two-dimensional data, and the three-dimensional pose calculator is used to generate the three-dimensional model based on the first normalized two-dimensional data and the second normalized two-dimensional data.

[0165] Example 9 includes an apparatus as defined in Example 1, wherein the three-dimensional model is projected onto a world coordinate system defined by a coordinate system of one of the first image capture device or the second image capture device.

[0166] Example 10 includes at least one non-transitory computer-readable medium comprising instructions that, when executed, cause at least one processor to perform at least the following operations: predicting two-dimensional coordinates of joints of an object in a first image generated by a first image capture device to generate first two-dimensional pose data; predicting two-dimensional coordinates of joints of the object in a second image generated by a second image capture device to generate second two-dimensional pose data; predicting parameters of the first image capture device; generating a first three-dimensional model of the object in a certain pose based on the first two-dimensional pose data and the second two-dimensional pose data; executing an optimization algorithm to generate a second three-dimensional model based on the first three-dimensional model and the parameters of the first image capture device; and outputting the second three-dimensional model for display through a user device.

[0167] Example 11 includes at least one non-transitory computer-readable medium as defined in Example 10, wherein the instructions, when executed, cause the at least one processor to perform the following operations: predict the two-dimensional coordinates of each joint of the object in the first image by executing a first neural network model.

[0168] Example 12 includes at least one non-transitory computer-readable medium as defined in Example 10 or 11, wherein the instructions, when executed, cause the at least one processor to perform the following operations: generate the first three-dimensional model by executing a second neural network model.

[0169] Example 13 includes at least one non-transitory computer-readable medium as defined in Example 12, wherein the instructions, when executed, cause the at least one processor to perform the following operations: predict three-dimensional coordinates of each joint of the object in response to execution of the second neural network model.

[0170] Example 14 includes at least one non-transitory computer-readable medium as defined in Example 10, wherein the first image capture device is a camera, and the parameter is an extrinsic parameter of the camera.

[0171] Example 15 includes at least one non-transitory computer-readable medium as defined in Example 10, 11 or 14, wherein the instructions, when executed, cause the at least one processor to perform the following operations: extract the first image from first image data generated by the first image capture device; extract the second image from second image data generated by the second image capture device; and synchronize the first image and the second image based on time.

[0172] Example 16 includes at least one non-transitory computer-readable medium as defined in Example 15, wherein the first image is a first video frame and the second image is a second video frame.

[0173] Example 17 includes at least one non-transitory computer-readable medium as defined in Example 15, wherein the object is a first object and the instructions, when executed, cause the at least one processor to perform the following operations: generate a third three-dimensional model for a second object in the first image and the second image.

[0174] Example 18 includes an apparatus comprising: a component for predicting a two-dimensional pose of an object based on first image data generated by a first image capture device and second image data generated by a second image capture device; a component for predicting parameters of the first image capture device; and a component for generating a three-dimensional graphical model of the object based on the predicted two-dimensional pose and the predicted parameters of the first image capture device.

[0175] Example 19 includes the apparatus as defined in Example 18, wherein the means for predicting the two-dimensional pose predicts the two-dimensional pose of the object by executing a first neural network model.

[0176] Example 20 includes the apparatus as defined in Example 19, wherein the means for generating is for generating the three-dimensional graphical model by executing a second neural network model.

[0177] Example 21 includes the apparatus as defined in Example 20, wherein the second neural network implements a regression function.

[0178] Example 22 includes the apparatus as defined in Example 18, wherein the means for generating determines a joint rotation vector and generates the three-dimensional graphical model based on the joint rotation vector.

[0179] Example 23 includes at least one non-transitory computer-readable medium comprising instructions that, when executed, cause at least one processor to perform at least the following operations: generate a first neural network model to predict a two-dimensional pose of an object in a first image generated by a first image capture device and in a second image generated by a second image capture device, respectively; and generate a second neural network model to predict a three-dimensional pose of the object, the predicted three-dimensional pose to be used to generate a three-dimensional graphical model of the object in the pose.

[0180] Example 24 includes at least one non-transitory computer-readable medium as defined in Example 23, wherein the second neural network model is used to implement a regression function.

[0181] Example 25 includes at least one non-transitory computer-readable medium as defined in Example 23 or 24, wherein the second neural network model is used to define an offset between a first joint of the object and a reference joint of the object.

[0182] Although certain example methods, apparatus, and articles of manufacture are disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus, and articles of manufacture fairly falling within the scope of the claims of this patent.

[0183] The following claims are hereby incorporated by reference into this Detailed Description section, with each claim standing on its own as a separate embodiment of the present disclosure.

Claims

1. At least one non-transitory computer-readable medium comprising instructions that cause at least one processor circuit to at least: providing the two-dimensional image data to a first machine learning model, the first machine learning model being trained to output feature data based on the two-dimensional image data; providing the feature data to a second machine learning model, the second machine learning model being trained based on the depth image data to output three-dimensional data based on the feature data, the three-dimensional data being associated with an object depicted in the two-dimensional image data; determining coordinates associated with points of the object based on the three-dimensional data; as well as A pose of the object is output based on the coordinates.

2. The at least one non-transitory computer-readable medium of claim 1, wherein: The gesture is associated with a translation.

3. The at least one non-transitory computer-readable medium of claim 1 , wherein: The pose is associated with a rotation.

4. The at least one non-transitory computer-readable medium of claim 1 , wherein: The second machine learning model includes a feedforward network.

5. The at least one non-transitory computer-readable medium of claim 1, wherein: The feature data includes key point data.

6. The at least one non-transitory computer-readable medium of claim 1, wherein: The instructions cause one or more processor circuits of the at least one processor circuit to obtain the two-dimensional image data from a camera.

7. The at least one non-transitory computer-readable medium of claim 1, wherein: The instructions cause one or more processor circuits of the at least one processor circuit to predict a pose of the object based on the coordinates.

8. A device comprising: Interface circuit; computer-readable instructions; as well as at least one processor circuit programmed based on the computer-readable instructions to: providing the two-dimensional image data to a first machine learning model, the first machine learning model being trained to output feature data based on the two-dimensional image data; providing the feature data to a second machine learning model, the second machine learning model being trained based on the depth image data to output three-dimensional data based on the feature data, the three-dimensional data being associated with an object depicted in the two-dimensional image data; determining coordinates associated with points of the object based on the three-dimensional data; as well as A pose of the object is output based on the coordinates.

9. The device according to claim 8, wherein The gesture is associated with a translation.

10. The device according to claim 8, wherein The pose is associated with a rotation.

11. The device according to claim 8, wherein The second machine learning model includes a feedforward network.

12. The device according to claim 8, wherein The feature data includes key point data.

13. The device according to claim 8, wherein The instructions cause one or more processor circuits of the at least one processor circuit to obtain the two-dimensional image data from a camera.

14. The device according to claim 8, wherein The instructions cause one or more processor circuits of the at least one processor circuit to predict a pose of the object based on the coordinates.

15. A system comprising: camera; an interface circuit for obtaining two-dimensional image data from the camera; computer-readable instructions; as well as at least one processor circuit programmed based on computer-readable instructions to: causing a first machine learning model to output feature data based on the two-dimensional image data; causing a second machine learning model to output three-dimensional data based on the feature data, the three-dimensional data being associated with an object depicted in the two-dimensional image data, the second machine learning model being trained based on depth image data; determining coordinates associated with points of the object based on the three-dimensional data; as well as A pose of the object is output based on the coordinates.

16. The system according to claim 15, wherein: The gesture is associated with a translation.

17. The system according to claim 15, wherein: The pose is associated with a rotation.

18. The system according to claim 15, wherein: The second machine learning model includes a feedforward network.

19. The system of claim 15, wherein: The feature data includes key point data.

20. The system of claim 15, wherein: The instructions cause one or more processor circuits of the at least one processor circuit to predict a pose of the object based on the coordinates.

Images