Systems and methods for digitizing and analyzing marksmanship skills

Abstract

The physical movement improvement system is configured to receive video data, track one or more body landmarks associated with the movement to determine movement of the body landmarks during the action, correlate the movement of the body landmarks with a score for the action, and determine movements that are detrimental to the score for the action using machine learning techniques. The system can also recommend drills to the participant to improve the score for the action.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 406,241, filed September 13, 2022, entitled "SYSTEMS AND METHODS FOR MARKSMANSHIP DIGITIZING AND ANALYZING," the contents of which are incorporated herein by reference in their entirety. [Background technology]

[0002] Target shooting is enjoyed by millions of people annually, and by many reports, the number of people who regularly practice target shooting has increased over the past decade and continues to grow. In the United States alone, it is estimated that over 52 million people regularly practice target shooting. There are many different types of recreational shooting activities, ranging from simple plinking with a handgun or rifle against paper or steel targets, to skilled long-range rifle shooting competitions, which require advanced training and skill, to fun, fast-paced pistol shooting against popping or stationary targets, or shotgun shooting against skeet, trap, sporting clays, and other games. Apart from recreational shooting, an increasing number of target shooters practice as part of their professions, such as law enforcement, the military, and security personnel.

[0003] While many participants are satisfied with their current skill level, there is an increasing number of shooters who want to improve their skills. However, in some cases, many participants do not know how to improve. The number of people at shooting ranges continues to grow, and these participants, whether for sport, recreation, self-defense, or public defense, want to improve their skills. However, without individual shooting instruction, and often even if individual instruction is provided, it can be difficult to quantify improvements and issues affecting accuracy. Furthermore, many participants do not know how to improve.

[0004] Therefore, there is a need for a system and method that can analyze participants' habits and provide feedback and recommendations for improvement. Further, there is a need for a system that can provide the aforementioned benefits in near real time using consumer devices. These and other advantages will be readily apparent from the following disclosure. Summary of the Invention

[0005] One or more computer systems may be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system that causes the system to perform the actions during operation. One or more computer programs may be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions. One general aspect includes a method for improving shooting performance. The method also includes receiving video data of a shooter, determining one or more body landmarks of the shooter, tracking the one or more body landmarks during a shot to generate shot motion data, determining a score for the shot, associating the shot motion data with the score, and generating recommendations for changing the motion data in subsequent shots. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0006] Implementations may include one or more of the following features. In this method, determining one or more body landmarks of the shooter may include generating a wireframe model by connecting the body landmarks. Associating the shot motion data with the score may include executing a classification and regression tree machine learning model to identify a causal relationship between the shot motion data and the score. The method may include determining the shooter's grip through image analysis of the shooter's video data. The method may include analyzing the shooter's grip and providing, on a display screen, a grip recommendation for changing the grip. Determining the score of the shot may include receiving target video data, performing image analysis on the received target video data, determining a hit on the target, and determining a score for the hit. Receiving video data includes capturing the video data with a mobile phone. Determining one or more body landmarks includes determining 17 body landmarks. Tracking the one or more body landmarks includes generating a bounding box around each of the one or more body landmarks. The method may include executing a machine learning model to correlate the shot motion data with the score. The machine learning model is configured to determine motion data that results in off-center hits of the target. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0007] One general aspect includes a method for improving a causal outcome of a physical movement. The method also includes receiving video data of the physical movement, determining one or more body landmarks visible in the video data of the physical movement, tracking the one or more body landmarks during the action, generating motion data based at least in part on tracking the one or more body landmarks, determining a score associated with the motion data, associating the motion data with the score, and generating a recommendation for altering the motion data in a subsequent action. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0008] Implementations may include one or more of the following features: In the method, receiving video data includes capturing the video data by a mobile computing device. The method may include executing a machine learning model to correlate the behavioral data with a score. The machine learning model is configured to determine behavioral data that results in a reduced score. The method may include predicting, by the machine learning model, a predicted score based on the behavioral data. The method may include comparing the predicted score to the score. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium. [Brief explanation of the drawings]

[0009] The accompanying drawings are part of this disclosure and are incorporated herein. The drawings illustrate examples of embodiments of the present disclosure and, together with the description and claims, serve to explain, at least in part, various principles, features, or aspects of the present disclosure. Specific embodiments of the present disclosure are described more fully below with reference to the accompanying drawings. However, various aspects of the present disclosure may be implemented in many different forms and should not be construed as limited to the implementations set forth herein. Like numbers refer to similar, but not necessarily the same or identical, elements throughout. [Figure 1A] 1 shows an image of a shooter captured by an image capture device, according to some embodiments. [Figure 1B] 1B illustrates a wireframe model of the shooter of FIG. 1A, according to some embodiments. [Figure 2] 1 illustrates motion data associated with multiple body landmarks, according to some embodiments. [Figure 3] 1 illustrates motion data associated with multiple body landmarks and automatic shot detection based on the motion data, according to some embodiments. [Figure 4] 1 illustrates analyzing motion data associated with multiple body landmarks and identifying shooting errors based on the motion data, according to some embodiments. [Figure 5] 1 illustrates analyzing motion data associated with multiple body landmarks and identifying shooting errors based on the motion data, according to some embodiments. [Figure 6] 1 illustrates analysis of motion data associated with multiple body landmarks according to some embodiments. [Figure 7] 1 illustrates the analysis of motion data associated with multiple body landmarks and identifying pose changes that result in the least improvement in performance, according to some embodiments. [Figure 8A] 1 shows an image of a shooter captured with an imaging device, according to some embodiments. [Figure 8B]8B illustrates a computer-generated wireframe model of the shooter of FIG. 8A with identified body landmarks, according to some embodiments. [Figure 9] 1 illustrates motion data associated with multiple body landmarks and identifying physical motions that lead to poor performance, according to some embodiments. [Figure 10] 10A and 10B show a diagnostic target that identifies shooter behavior based on shot patterns, according to some embodiments. [Figure 11] 1 illustrates an image of a shooter's hands and a computer-generated wireframe model associated with body landmarks that identify and analyze the shooter's grip, according to some embodiments. [Figure 12A] FIG. 1 is a diagram of a computer software program user interface for capturing video images, tracking body landmark movements, and analyzing shooter pose and motion data, according to some embodiments. [Figure 12B] FIG. 1 is a diagram of a computer software program user interface for analyzing a shooter's performance and automatically scoring the performance, according to some embodiments. [Figure 13A] 1 illustrates a system for digitizing and analyzing marksmanship, according to some embodiments. [Figure 13B] 1 illustrates a system for digitizing and analyzing marksmanship, according to some embodiments. [Figure 13C] 1 illustrates a system for digitizing and analyzing marksmanship, according to some embodiments. [Figure 14] 1 illustrates a system for digitizing and analyzing marksmanship, according to some embodiments. [Figure 15] 1 illustrates the logic of a machine learning algorithm for quantifying shot samples, according to some embodiments. [Figure 16]1 illustrates a sample decision tree machine learning model that correlates body landmark motion data with shot performance, according to some embodiments. [Figure 17] 1 illustrates an annotated decision tree machine learning model that correlates body landmark motion data with shot performance, according to some embodiments. [Figure 18] 1 illustrates a pruned decision tree machine learning model that correlates body landmark motion data with shot performance, according to some embodiments. [Figure 19A] 1 illustrates a system for capturing video data of a shooter and a target, according to some embodiments. [Figure 19B] 10 illustrates image data of a left side view of a shooter captured by an imaging device, according to some embodiments. [Figure 19C] 10 illustrates image data of a right side view of a shooter captured by an imaging device, according to some embodiments. [Figure 19D] 1 illustrates image data of an overhead view of a shooter captured by an imaging device, according to some embodiments. [Figure 19E] 1 illustrates image data of a target captured by an imaging device, according to some embodiments. [Figure 20] 1 illustrates a sample user interface of a computer application configured to receive, analyze, and score marksmanship performance, according to some embodiments. [Figure 21] 1 illustrates a sample user interface of a computer application configured to receive, analyze, and score marksmanship performance, showing shot placement and automatic scoring, according to some embodiments. [Figure 22]1 illustrates a sample user interface of a computer application configured to receive, analyze, and score a shooting skill performance, allowing selection of a body landmark and showing motion data associated with the selected body landmark, according to some embodiments. [Figure 23] Illustrated are some of the key technologies enabled by the embodiments described herein, along with the features they facilitate, according to some embodiments. [Figure 24] 10A-10C illustrate sample user interfaces of a computer application configured to receive, analyze, and score shooting skill performance, illustrating a system that uses multiple source collaboration and machine learning to detect fired shots, analyze shooting position and provide recommendations for improvement, and analyze grip position and provide recommendations for improvement, according to some embodiments. [Figure 25] 1 is a process flow for capturing operational data, analyzing the operational data, and generating recommendations for improvement, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0010] According to some embodiments, a system is described that uses computer vision and machine learning to significantly reduce the manual and labor-intensive nature of current training techniques, thus providing a system that continuously learns and adapts. According to some embodiments, the system includes a machine vision and machine learning system that tracks and estimates a participant's pose and can determine positive and negative factors that affect the quality of participation, such as pose, movement, anticipation, recoil, grip, and stance, among others. This may be primarily performed by a computer vision system that can simultaneously track several body landmarks and, in some cases, correlate the movement of one or more body landmarks with the accuracy of the shooting skill. By way of example, the system may identify and track any number of body landmarks, such as three, five, eleven, seventeen, twenty-one, twenty-five, thirty, or more body landmarks. Because the system tracks the location of landmarks, which may be in two or three dimensions, the system can correlate the movement of the landmarks with bullets fired downrange and the scoring of individual bullets. Detection of a bullet fired downrange may be determined by the movement of one or more suitable markers, such as the movement of a participant's hand or wrist (or other body marker) in response to recoil, the sound of a gunshot, a pressure wave associated with a muzzle blast, a target hit, or some other marker.

[0011] The system may further monitor the participant from one, two, three, or more viewpoints, analyze the movement of each body landmark, and further monitor the accuracy of the shot and correlate the movement of the body landmark with the accuracy. Based on the accuracy, the system may further provide an analysis of the body movements that contribute to the less-than-perfect accuracy and further suggest ways to improve the body movements to increase the accuracy.

[0012] Motion capture may be performed by one or more cameras generally pointed at the participant and one or more cameras pointed at a target. In some cases, one or more of the cameras are associated with mobile computing devices such as smartphones, tablets, laptops, personal digital assistants, and wearable devices (e.g., watches, glasses, body cams, smart hats, etc.). In some cases, the wearable devices may include sensors such as accelerometers, vibration sensors, motion sensors, or other sensors to provide motion data to the system. In some embodiments, the system tracks body marker positions over time and generates a motion plot.

[0013] Referring to FIG. 1A , one or more cameras may capture one or more views of a shooter 100. The cameras may capture video data of the shooter as he or she draws, aims, fires, reloads, and / or adjusts position. A computer system may receive the video data, analyze the video data, and create a model associated with the shooter, such as that in FIG. 1B . In some cases, the computer system may identify body landmarks and connect them to a wireframe model 102 that tracks the shooter's pose and movement of the body landmarks. In some instances, the body landmarks may include one or more of the nose, left ear, left eye, left hip, left knee 103, right ear, right eye, right hip, left ankle, left elbow, left wrist, right knee 104, right ankle, right elbow 106, right wrist 108, left shoulder, and right shoulder 110. Of course, other body landmarks are possible, but for efficiency, we will focus on these 17 body landmarks throughout this disclosure. In some embodiments, a single camera may capture two-dimensional motion data associated with one or more of the body landmarks. In some examples, two or more cameras may be used to capture three-dimensional motion data of one or more of the body landmarks.

[0014] The body landmarks may be tracked over time, such as during a shooting string (e.g., a shooting session), and the movement of one or more of the body landmarks may be tracked during this time. In some cases, two-dimensional movement is tracked in the x and y directions, corresponding to side-to-side and vertical movement. In some cases, three-dimensional movement of the body landmarks is tracked in the x, y, and z directions.

[0015] Referring to FIG. 2, a graph of selected body landmarks 200 is shown. The body landmarks are user-selectable by the user to focus on individual body landmarks or combinations of body landmarks for review. For example, the top line 202 represents the right wrist of a right-handed shooter in a horizontal direction, and the third line 203 represents the vertical direction of the right wrist over time. As can be seen, the wrist moves up and down during the motion capture process. In some cases, the system may correlate the movement of one or more body landmarks with events or stages during the shooting string.

[0016] For example, during the first stage 204, the right wrist is in a relatively low position, and the system may correlate this position and movement with a ready-to-fire state. The second stage 206 shows the right wrist moving upward in a very short interval, which may correlate with the act of drawing a pistol from a holster. The third stage 208 shows the right wrist remaining fairly stable in the vertical plane, but with sharp peaks 207a-207c in movement, which may correlate with a shot being fired from the pistol.

[0017] During the fourth stage 210, the right wrist moves downward and returns to a firing position. The system may correlate this movement with a reloading motion. In some cases, the system is trained with training data, which may be supervised learning, to correlate similar movements with various stages.

[0018] During the fifth stage 212, the shooter's right wrist initially moves upward as the shooter takes aim, then settles on the target, followed by peaks of movement 213a-213c, which may correlate with the shot being fired downrange.

[0019] In the sixth stage 214, the right wrist moves downward again to its initial position, which may correlate with holstering the pistol.

[0020] While this example focuses on the shooter's right wrist in the vertical direction, it is clear that any body landmark can be viewed and analyzed, and movements or combinations of movements can be correlated with events or actions by the shooter, including groups of body landmarks.

[0021] Referring to FIG. 3 , a close-up view of the shooting stage 300 is depicted, showing sharp peaks in the vertical movement of the right wrist 302 and the right elbow 304. The system can analyze the motion data and automatically determine when a shot was fired. The system can be configured to correlate sharp peaks in the vertical movement of the wrist and / or elbow with fired shots. As shown in FIG. 3 , each of the arrows 306 can correspond to a fired shot. Additionally, audio data can be correlated with the motion data to provide additional cues as to when a shot was fired. In some cases, audio data can be combined and / or synchronized with the motion data to provide additional details about the fired shot. The motion data can be used to infer additional information about the shooter, their habits, their posture, and other cues that can catch the shooter's attention as part of an effort to improve their accuracy.

[0022] Referring to FIG. 4 , motion data 400 is displayed for a shooter's right wrist 402 and right elbow 404. In some cases, the system may apply a trend curve 406 to the motion data, which may represent normalized motion data. Additionally, the system may make inferences and / or decisions based on the motion data 400. For example, as shown in FIG. 4 , at 708, a shooter aims a firearm at a target and attempts to hold the firearm steady; at 410, the shooter lowers their wrist and elbow, and shortly thereafter, a shot is fired 712. The system may recognize this pattern and determine that the lowering of the wrist and / or elbow immediately prior to the shot is evidence that the shooter is anticipating and preparing for the recoil of the firearm. In many cases, the shooter moves the firearm away from the target in anticipation of the recoil that will occur when the shot is fired, dramatically reducing the accuracy of the shot. Similar examples of behavioral data that may reduce a shooter's accuracy include flinching, pre-ignition pushing, trigger jerking, and closing the eyes, among others.

[0023] In some cases, the system may provide information to the shooter regarding recoil anticipation, information that may include one or more drills or practice sessions as part of an effort to improve the shooter's performance data regarding recoil anticipation. For example, the system may identify a practice regimen that may include dry firing, skipping reloads (e.g., mixing live ammunition with a magazine of dummy ammunition), or other skill-building drills.

[0024] 4 also shows that there is a drift in the shooter's posture. For example, before the first shot 412, the shooter's right wrist and right arm are at a first vertical height 414, and before the second shot 416, the shooter's right wrist and right elbow are at a second vertical height 418, higher than the first vertical height. This data indicates that the shooter did not return to the same position from the first shot to the second shot, resulting in a slightly different sight picture for the shooter, which may reduce the accuracy of subsequent shots.

[0025] 5, motion data 500 further illustrates drift. Motion data associated with a right-handed shooter's right wrist 502 and right elbow 504 not only shows recoil anticipation, indicating a downward movement of body parts immediately prior to the shot, but also shows trend lines 506 demonstrating a continuous upward drift of the shooter's wrist and elbow during the firing string. If the shooter does not return to the same position between shots, the accuracy and precision of the shots fired within the string can be dramatically reduced.

[0026] In some cases, the system may recognize drift in one or more of the body landmarks and provide this information to the shooter. In some cases, the system provides information on a display screen associated with a mobile computing device. For example, the system may be implemented on a mobile computing device associated with the shooter, and the display screen on the mobile computing device may provide information, instructions, or practice drills to the shooter to improve drift and resulting accuracy problems.

[0027] Similarly, the system may correlate the movement of body landmarks with other correctable imperfections in the shooter's position or posture. For example, referring to FIGS. 6 and 7, which illustrate body landmark motion data 600, the motion data 600 shows the movement of multiple body landmarks during the shooting string. The bottom line graph shows right ankle motion data 602, demonstrating that the shooter changed the position of his right foot. Changing position during the shooting string likely impacts the sight picture, accuracy, precision, and other metrics associated with shooting. The system may determine whether the change in foot position was positive or negative with respect to the scoring of the shot and provide recommendations to the shooter based on this change in posture. The system may look at the shooting accuracy and / or precision of shots 604a-604e fired both before and after the foot repositioning to determine whether moving the foot had a positive or negative impact on shooting performance.

[0028] In some cases, the system can correlate target scoring (e.g., shot accuracy and / or precision) with body landmarks and movements to show the shooter which positions of individual body landmarks influenced the shooter's shooting performance, for better or worse.

[0029] The system may be configured to associate specific poses, movements, and combinations with shooting performance through machine learning. In some cases, body landmark movements and movement combinations may be associated with improved shooting performance, while others may be associated with decreased shooting performance.

[0030] The system may normalize the motion data to generate normalized coordinates for each body part's position during all sessions. The score and its running average may be represented by a signal, for example, by displaying it on a user interface. The motion data and / or score may be analyzed in near real time or stored in a data file that can be saved for later analysis.

[0031] Motion data may be associated with shot pattern data, such as the x and y coordinates of each shot, and the shot coordinates may be temporally associated with the motion data occurring at the time the shot was fired. Additionally, a score may be assigned to the shot and stored with the shot data.

[0032] One or more machine learning techniques may be applied to the motion data and shot data to generate correlations between motion and shot accuracy. For example, in some embodiments, a convolutional deep neural network (CNN) may be designed to locate features in the motion and shot data collection. Other deep learning models for classification purposes may also be used to correlate motion and shot data to identify patterns that lead to improved or reduced accuracy. Transformations that correlate any set of attributes with any other set of attributes may also be used.

[0033] FIG. 8A shows a sample camera angle of a shooter 800, and FIG. 8B shows the resulting wireframe model 802 of the shooter, which allows the system to track the shooter's body movements, including selected body landmarks. As can be seen, the system can determine the shooter's pose, stance, and movements throughout the shooting string. The wireframe model may include representations of each major joint and body artifact, which may include, among other things, one or more of the shooter's nose or chin 804, right shoulder 806, right elbow 808, right wrist 810, hip 812, left femur 814, left knee 816, left lower leg 818, left ankle 820, right femur 822, right knee 824, right lower leg 826, and right ankle 828. In some cases, the system may be trained with motion data from various shooters and their past performances correlated with the motion data. In this case, the system can determine the impact of specific poses and / or movements on shooting performance. In some cases, the system may rely on performance data from a single shooter to assist that shooter in making adjustments and / or conducting further training exercises to improve the shooter's performance.

[0034] 9 shows x-axis motion data 900 associated with a shooter's head 902 and nose 904. As can be seen, the shooter's head moves backward after each shot 906a-906d during the firing string, which may be correlated with shooting performance. That is, the system may determine that the shooter is moving their head after each shot, for example to look at the target, and that they may have drifted during the firing string and not returned to the exact same spot, causing the shooter's performance to fall below a threshold.

[0035] In some cases, the system may be configured with logic to determine probable cause and effect based on either the shooter's movements and / or the target's scoring. Referring to Figures 10A and 10B, diagnostic targets 1000 for left-handed and right-handed shooters, respectively, are shown. Because the right-handed and left-handed targets are mirror images of each other, only one target will be described below.

[0036] Assuming the shooter is able to hold the pistol at the target, movements associated with the shooter may be the cause of off-center hits. If the off-center hits are regularly clustered to one side of the target, there are several possible issues that could be causing the off-center hits. For example, if a group of shots lands at the 12:00 position, the system may determine that the shooter is bending their wrist upward 1002 (e.g., riding the recoil), which often occurs in anticipation of the recoil.

[0037] If the group of shots lands at 1:30 position 1004, this may be indicative of a grip error known as heeling, where the heel of the hand pushes the pistol butt to the left in anticipation of the shot, which forces the muzzle to the right.

[0038] If a group of shots lands at the 3:00 position 1006, this may be an indication that the thumb of the shooting hand is applying too much pressure, pushing the side of the pistol to the right and forcing the muzzle to the right.

[0039] If the group of shots lands at the 4:30 position 1008, this may be an indication that the shooter is squeezing the grip too tightly while firing (e.g., lobstering). Pulling the trigger will lower the front sight, which will cause the shot to go low and to the right for a right-handed shooter.

[0040] If a group of shots lands at the 6:00 position, 1010, this may be an indication of a downward bending and forward thrust of the wrist, often an unconscious effort to control recoil and prevent muzzle lift.

[0041] If a group of shots lands at the 7:00 position 1012, this may indicate a trigger jerk or slap, which may indicate the shooter is about to pull the trigger just as the sights are on target.

[0042] If a group of shots lands at the 8:00 position 1014, this may be an indication that the shooter is squeezing the grip too tightly during the shot.

[0043] If a group of shots lands at the 9:00 position 1016, this may be an indication that the finger is too light on the trigger, which typically causes the shooter to pull the trigger at an angle when finally pulling it rearward, tending to push the gun to the left.

[0044] If a group of shots lands at the 10:30 position 1018, this may be an indication that the shooter is pushing in anticipation of the recoil and not following through well.

[0045] The system can be programmed to verify target hits during the firing string and, in combination with body landmark movements, determine whether the shooter is engaging in recoil anticipation, trigger control errors, and / or grip errors. The system can make this determination for each individual shooter and recommend practice exercises and practice strings to address their specific shooting issues.

[0046] In some cases, the system may have access to Data On Previous Engagement (DOPE) associated with the shooter, which may include previous shooting sessions, records, scores, and analysis.

[0047] Complete shooting sessions can be recorded and stored, along with real-time scoring, so the user can review them along with decisions from the machine learning system that identify and / or highlight flaws, mistakes, and changes in the shooter's posture or technique that improve or detract from the shooter's performance.

[0048] As described herein, synergies arise from synchronizing all available information sources.

[0049] Continuing with reference to Figures 10A and 10B, where the diagnostic target is shown, the basic traditional ML model maps a target shot pattern φ ∈ Φ to a shooter behavior β ∈ B, which is given by f T It is denoted as Φ→B. The behavior can be a four-dimensional (4D) phenomenon that includes three-dimensional (3D) motion data plus time. The shot pattern Φ is a two-dimensional (2D) projection of the evolving 3D phenomenon (2D plus time). The model can be based on causal hypotheses.

[0050] Experts have a much larger set of complex shooter behaviors. * ⊃B and divide it into a larger set of shot patterns Φ * which can be mapped to f XP :B * →Φ. Behavior β∈B * can be a 4D phenomenon, and the shot pattern φ∈Φ * is a 2D phenomenon. It can be assumed that this is in principle a causal model, since experts adapt the model according to their understanding of the shooter's physiological kinematics and psychology, as well as the physics of shooting.

[0051] A relatively simple iterative model for expert learning may involve predicting shot patterns from observed shooter behavior, which is XP :B * →Φ. The model can then estimate the difference between the predicted shot pattern and the actual shot pattern, which is denoted as δ(φ′,φ). The model can then generate a causal prediction model f XP This may be repeated every n shot drill.

[0052] This type of learning model is based on the shooter behavior β∈B * is treated as a 3D phenomenon, and the shot pattern φ∈Φ * is treated as a 2D phenomenon. In some cases, B * as a 4D phenomenon, or Φ * As a 3D phenomenon, the model is enhanced to treat B as a 3D phenomenon, or both. ** 3D projection (2D plus time) of 4D shooter behavior B * Represents.

[0053] The model is a decision function f G By running Φ → D (where D is a binary variable), the model can objectively distinguish good shot patterns (e.g., clustered around the bullseye) from the remaining "bad" shot patterns. The model also predicts the shooter's behavior E * can be constructed to relate to how (and why) the shot pattern φ is "bad".

[0054] In some embodiments, the sampled time signal for the shooter behavior can be reduced for a single drill as a single line in the dataset. A bounding box can be derived around the set of XY coordinates of each of the body landmarks in the drill. The bounding box can be represented as shooter behavior B. Thus, there can be a shooter behavior for each of the body landmarks during a period of time. In some cases, shooter behavior B and shot pattern Φ are represented in polar coordinates as a radius and an angle. Shooter behavior B (which in some examples is 17) is 2D with (x,y) coordinates or (r,θ) representation. Shot pattern φ∈Φ is a causal consequence of shooter behavior β∈B, and the method can treat shot pattern Φ as a proxy for shooter behavior β in the unobservable third dimension. The method treats shooter behavior β∈B and shot pattern φ∈Φ as features and treats score s∈S as a target for the ML model, which is expressed as f ML :BxΦ→S.

[0055] The model considers patterns (β,φ,s)∈Ψ that result in low and high scores. L ,Ψ H For this purpose, implicit or explicit clustering techniques may be used. In some cases, the model f ML :BxΦ→S can be constructed for multiple drills for a particular shooter and drills for multiple shooters. In some cases, a particular drill (β,φ,s) and a model f ML Given a :BxΦ→S, model quality can be assessed by comparing the actual score s∈S with the predicted score s'∈S. From these comparisons, some models may be updated sequentially.

[0056] In some embodiments, a low-scoring drill (β, φ, s) can be evaluated against Ψ to determine which of the 17 components of the observable shooter behavior β and the shot pattern φ, which serve as a proxy for unobservable shooter behavior, are likely responsible for the low score. This may be easiest for an explainable model. Unobservable behavior may be explained by the usual labels in the diagnostic target, as described above. In some cases, the shooter diagnostic application may also distinguish low scores due to aim misalignment, perhaps easiest understood as shots clustered around a center of gravity other than the target bull's-eye. In some cases, the system may determine that a reduced score was obtained and generate recommendations to improve the score. As used herein, the term "reduced score" is used to mean a score that is less than a target score. The target score may be a perfect score, a participant's best previous score, a participant's average score, or some other metric. For example, in shooting competitions, the highest score for a given shot is often "10." In this case, a reduced score is a score below "10." Similarly, in other sports, a score that is less than a desired score may be assigned. For example, a penalty kick in soccer could have a binary outcome of a missed goal resulting in a reduced score compared to a goal made. In terms of golf shots, a golfer may average a driver that hits the ball 275 yards. A reduced score could be due to a driver shot that travels 250 yards, which is less than the golfer's average driver distance, and the system could observe the behavior and determine which behavior(s) resulted in the reduced score.

[0057] According to some embodiments, the system relies on artificial intelligence (AI) and / or machine learning (ML) for two decisions: detecting shooter movement and shot landing locations, and analyzing shooter behavior for how shooter behavior leads to shot landing locations. While detecting movement has been discussed, there are multiple ways to analyze how shooter behavior leads to shot landing locations. First, an expert-based approach utilizes comparing individual shooter behavior for each drill with expert assessments of what is believed to result in a good shot. Second, a data-based approach is implemented by building a model from repetitions of shooting behaviors ("drills") by a single shooter or multiple shooters. This can be considered an AI / ML-based discovery strategy that determines which behaviors correlate with good shots.

[0058] In some cases, AI / ML may be used to automate and augment expert-based analysis in the sense that if the "ideal" behavior of a prototype is known a priori, models of these a priori known behaviors can be fitted to data from a single shooter or multiple shooters.

[0059] From that perspective, Transformer ML models can mimic automated and augmented expert-based analysis by combining pre-trained database models for inferred relationships between language fragments (similar to inferring relationships between shooter behavior and shot landing locations) with an additional stage of database adaptation.

[0060] Referring to FIG. 11 , the system determines key points related to the shooter's hand 1100. The key points may correspond to each mobile joint in the wrist, hand, and fingers, and their respective locations and positions relative to one another. The system may connect the key points to a wireframe model 1102 of the shooter's hand, allowing for accurate monitoring of pose and movement. FIG. 11 illustrates multiple key points associated with the shooter's hand that can be used to determine the grip the shooter is using. For example, by referencing the key points on the shooter's hand, the system may determine whether the shooter is using a thumbs-forward, thumbs-over, cup-and-saucer, wrist grip, trigger-guard gamer, or another style of grip. Different grips and pressure applied by the hand can impart movement to the pistol, and the system may determine that a different or modified grip would result in better performance.

[0061] The system receives signals associated with the shooter's position and movement, processes the signals, and finds errors in the shooter's stance and grip. Additionally, by combining single-frame analysis (e.g., by finding insights from the relative positions of different body parts at a particular moment in time, which is analysis over time), it is possible to identify errors due to changes in the shooter's position.

[0062] Additionally, the system can focus on the position of the hand on the pistol. A hand tracking machine learning model can be used to track the position of each bone in each finger within the video frames. The system receives signals associated with the position of each finger over time.

[0063] 12A and 12B , an application program is shown that may execute on a mobile computing device and receive image data from one or more imaging sensors associated with the mobile computing device. As with any of the embodiments described herein, methods and processes may be programmed into an application program or set of instructions that may be executed on a computing device. In some cases, the application program may execute on the mobile computing device, and audio / video capture, analysis, scoring, recommendations, training exercises, and other feedback may be performed using the mobile computing device. In some cases, the system receives video and / or audio from multiple video capture devices that capture different views of the same shooter. The system may use the multiple different views of the shooter for analysis and feedback to the shooter to improve performance.

[0064] As shown in FIG. 12A , a system running as an application on a mobile computing device 1200 captures video frames of a shooter 1202, establishes body landmarks for tracking over time, creates a wireframe model 1203 of the shooter, tracks fired shots 1204, and provides feedback to the shooter 1202. As shown, the system may identify the shooter's stance—in the illustrated example, the “Weaver” stance—track the number of shots 1204, and provide feedback 1206 regarding each shot. For example, a shot fired at 7 seconds after the start of the string indicates a center-of-mass hit. The system identifies a reload motion 1208 at 8 seconds, lasting 1.2 seconds, followed by a shot at 11 seconds, indicating the shooter anticipated recoil and the shot was off-center. In some cases, this functionality of the system may be viewed as a drill instructor, tracking shots, providing feedback, and making suggestions for improving posture, grip, stance, trigger pull, etc., to improve the shooter's performance.

[0065] 12B shows an additional screen 1210 that may be displayed by the system in a Spotter modality in which a mobile computing device may aim its video capture device at a target. In some cases, the mobile computing device may use an internal or external lens to get a better view of the target; the mobile computing device may be coupled to a spotting scope or other type of optical or electronic telephoto zoom lens to get a better target image.

[0066] The system can be toggled between different modes, such as Spotter, Drill Instructor, DOPE, Locker (which can store information about different firearms owned by the shooter), and program settings. Spotter mode may show an additional screen 1210 with a view of the target where target hits can be viewed. The system may also display other information, such as the firearm 1212a and ammunition 1212b combination used in the shot, distance to the target 1214, target type 1216, firing string time 1218, number of shots fired, number of target hits 1220, and score 1222, among others. An image of the target 1224 may also be shown, and hits 1226 may be further highlighted on the image of the target 1224. It should be understood that Spotter mode may show information in real time, including elapsed time, hits, score, and number of shots. Additionally, the system may store data associated with the firing string for later playback, review, and analysis. For example, the Drill Instructor mode may review the DOPE associated with the shooter and the firearm and ammunition combination, provide a detailed analysis of the shooter's misses with a particular firearm, and provide exercises or suggestions for improving the misses to improve the shooter's performance.

[0067] 13A, 13B, and 13C, an exemplary architecture for data capture 1300 is shown. According to some embodiments, a shooting range 1301 may be equipped with one or more sensors, such as image sensors, projectors, and computing devices. FIG. 13A shows the shooting range 1301 looking downrange from the perspective of a shooter 1302. FIG. 13B shows the shooting range 1301 from a side view, and FIG. 13C shows the shooting range 1301 from a top view.

[0068] One or more cameras 1304 may be mounted within the shooting range to capture video of the shooter from multiple angles. The cameras 1304 may be any suitable type of video capture device, including, but not limited to, a CCD camera, a thermal camera, a dual thermal camera 1305 (e.g., a capture device having both thermal and optical capture capabilities), among others. The cameras 1304 may be mounted in any suitable location within the range, such as to the sides of the shooters 1304a, 1304b, facing the front of the shooter 1304c, or overhead 1304d, among others.

[0069] In some cases, the camera 1304 may be mounted on a gantry system that provides a portable structure to support one or more cameras 1304 to capture multiple angles of the shooter.

[0070] In some embodiments, a projector 1306 may be provided to project an image onto a screen 1308. In some cases, the projector may project an arbitrary target image onto the screen or shooting wall, allowing the shooter to practice dry shots (e.g., without firing live ammunition), and the system may register hits on the projected image. For example, the image may show a target on the screen, and the shooter may dry-fire at the target. The system may register the aim point at the time the trigger is pulled and display the hit on the projected target image. Compatible devices exist that fire a laser through the barrel of a firearm when the trigger is pulled to indicate where the shot would have landed on the target. The thermal imaging camera 1304 may detect the laser hit location, and the system may be configured to display the hit on the projected target, which may then register and score the hit. In this way, shooters can practice using a shooting simulator with their own firearms without having to travel to a dedicated shooting range. The system may encourage dry-fire practice to improve poor shooting habits.

[0071] 14 illustrates a system configured for multi-source synchronization. In some embodiments, a shooter 1402 is captured from different angles by multiple video capture devices 1404. The system obtains features such as stance, pose, and movement from the shooter 1406. The system may analyze the captured audio signal 1408 to complement the video shot detection. That is, the system may analyze both the video and audio and detect shots fired by the shooter, for example, by correlating spikes in the audio waveform with a sudden muzzle lift of the firearm.

[0072] The system may be configured to detect shooting stages 1410, such as first firing string, reload, second firing string, etc., including detection of fired shots. The system may additionally identify and determine shooting errors 1412, identify errors, and drills and exercises to address shooting errors. Error detection may be iterative and further error corrections 1414 may be suggested. The system may also incorporate shot detection 1416, as described herein, including correlation of shot detection with audio data.

[0073] The system may further capture images 1418, such as video images of the target, and register hits on the target, which may be correlated with motion data before and during the trigger pull. As described herein, the system may determine shot location 1420 and ultimately determine a score 1422 of one or more of the shots fired. Thus, some embodiments of the system provide an intelligent tutroing / adaptive training solution that transforms a time-intensive, non-scalable live training environment into an automated, adaptive, virtual scenario-based training solution.

[0074] According to some embodiments, the model is based on instructions executed by one or more processors that cause the processors to perform various operations. For example, the method may include collecting (as many as possible) pause analysis files, which may be stored as comma-separated value (CSV) files.

[0075] FIG. 15 shows a shot bounding box 1500 determined by the system. The “drillparser.py” script can be configured to combine the pose analysis files. The system can then find the shot in the CSV file for each of the 17 pose landmarks, for example, and derive a bounding box 1500 oriented around the x- and y-coordinates in a 1.0-second time window up to and including the shot. The system then uses the angle φ 1502, dimensions l 1504, and w 1506 of the bounding box 1500 as elements in a decision tree model for scoring. The “allshots” approach processes data on a CSV file basis, so bounding boxes 1500 are drawn around all shots as a single row in the resulting dataset. The “oneshot” and “oneshotd” approaches can treat each shot individually, so the shot bounding box 1500 can be an infinitesimal box around a single shot; thus, each row in the dataset is one shot. The "oneshotd" method additionally orients the bounding boxes around the 17 pose landmarks in time order, while the "oneshot" method ignores time.

[0076] Thus, the system can derive an oriented bounding box around each shot, which can be oriented in the direction of the box around the body landmarks.

[0077] 3) These datasets are processed in BigML as follows:

[0078] 1. Upload to BigML as a Source object

[0079] 2. Convert Source objects to Dataset objects

[0080] 3. Split the Dataset object into Training (80%) and Test (20%) datasets.

[0081] 4. Build a tree model using elements selected from the training dataset

[0082] 5. Create BatchPredictions using the appropriate Model and Test dataset

[0083] 6. Download the Model as a JSON PML object for the next step

[0084] 4) The "modelannotator.py" script annotates the model PML file as needed for explain behavior. In some cases, this annotation consists simply of adding a "target" attribute to each node that is a list of all target (score) values ​​reachable in the tree from that node.

[0085] 5) The “itemexplainer.py” script uses the (hypothetical) pruned annotated model to “explain” which pose elements for a new drill result in an unsatisfactory score prediction.

[0086] 6) Of course, the model can be tuned by repeating model training via machine learning using instances where the predicted scores differ significantly from the actual scores.

[0087] The elements derived from the shot sample may include one or more of the following elements: shot_phi with bounding box rotation (-π≦φ≦π), shot_l—bounding box length, shot_w—bounding box width, shot_lw—bounding box area, shot_theta—bounding box center rotation (-π≦φ≦π), shot_r—bounding box center radius. A bounding box 1500 is drawn around the shot sample.

[0088] A very similar approach can be used to derive elements from body landmark samples: since body landmarks move over time, each body landmark can similarly be used to create a bounding box with length, width, area, and rotation that can be correlated to a shot element.

[0089] In some cases, the system is configured to use one or more cameras for machine vision of the shooter, determine one or more body landmarks (possibly up to 17 or more), and track the movement of each of these landmarks over time during shooting drills. The system can correlate the timed body landmark movement with scored shots to determine whether a shot is good or bad. A good shot may be relative to the DOPE for the shooter, firearm, and ammunition combination. For example, if a single shot is closer to the aim point than the shooter's average shot, it may be classified as a good shot. Furthermore, by examining several shots over time, the system can correlate shooter behavior with good or bad shots. Furthermore, by analyzing shooter behavior (e.g., body landmark movement), the system can predict whether a shot is good or bad without seeing the target outcome. For example, a good shot is considered a shot within the 9 or 10 ring on the target, while a bad shot is one outside the 9 ring. The definition of a good or bad shot may vary depending on the shooter's expertise. As an example, for a highly skilled shooter, anything outside the 10 ring may be considered a bad shot.

[0090] Finally, by correlating shooter behavior with target accuracy, the system can provide the shooter with an output regarding the behaviors that cause a decrease in accuracy. Additionally, the system can recommend one or more specific drills to the shooter to address the behaviors that cause a decrease in accuracy.

[0091] 16 illustrates some embodiments of a decision tree model 1600 for correlating body landmark factors with shot samples. A decision tree algorithm is a machine learning algorithm that uses decision trees to make predictions. It follows a tree-like model of decisions and their possible outcomes. In some cases, the algorithm works by recursively dividing the data into subsets based on the most important features at each node of the tree.

[0092] From the body landmark data, various features are extracted to represent the kinematic characteristics of body movement. These features include body landmark positions over time, bounding boxes describing the movement of body landmarks during times including before and during the shot, relative distances between landmarks, and time derivatives capturing the dynamic aspects of the movement. Feature scaling is performed to facilitate model convergence.

[0093] In some embodiments, the decision tree shown in Figure 16 serves as the core of the model. It is selected for its ability to unravel complex nonlinear relationships in the data while maintaining interpretability. The depth of the tree is optimized to minimize overfitting through techniques such as cross-validation or a predetermined maximum depth. The choice of separation criterion, whether based on Gini impurity or information gain, depends on the specific problem. Pruning methods, such as enforcing a minimum number of samples per leaf, are applied to control excessive branching.

[0094] Training the model may involve the recursive construction of a decision tree. The tree has nodes that correlate with body landmarks. For example, the left_ankle_phi node 1602 may branch to a right_wrist_lw node 1604 and a right_knee_phi node 1606, which may indicate how the rotation of the bounding box of the right ankle motion data, combined with the rotation of the bounding box area of ​​the right wrist or the bounding box associated with the right knee, affects the resulting shot landing location. Similarly, the decision tree may correlate shot landing locations with feature combinations. At each node, the training data is partitioned based on feature values ​​to optimize a selected loss function, such as mean squared error or cross-entropy. Hyperparameters can be fine-tuned via cross-validation, and various metrics are used to evaluate the model's performance.

[0095] During real-time operation, the decision tree model is continuously updated as new body landmark data becomes available for each shot fired. For each input feature vector derived from live landmark data, the model traverses the decision tree and arrives at a leaf node. The label assigned to the leaf node (indicating a positive or negative outcome) is utilized as the model's prediction. In the illustrated example, a right shoulder bounding box length between values ​​of 0.80 and 0.82 matches the predicted outcome of the shooter's performance, as shown in leaf nodes 1608 and 1610.

[0096] In the context of this model, a positive outcome indicates successful execution of a particular action or achievement of a desired shot landing location, while a negative outcome indicates an inaccurate action, deviation from the desired position, or an off-center shot landing location.

[0097] Model performance is rigorously evaluated through metrics including, but not limited to, accuracy, precision, recall, F1 score, and area under the receiver operating characteristic (ROC) curve. A confusion matrix is ​​employed to quantify the model's proficiency in classifying positive and negative outcomes.

[0098] FIG. 17 shows an annotated model 1700 showing various values ​​at several nodes. For example, during the firing string, the area of ​​the bounding box for the right wrist 1702 shows values ​​of 0.47, 0.51, 0.57, 0.59, and 0.61. This indicates that the shooter moved his right wrist within the area defined by the displayed values ​​during the firing string when shots were being fired. Following the branch below the right wrist 1702, for values ​​below the average of the values, the system can identify that the left knee 1704 moved in a specific pattern and correlate this with either good or bad shots to explore causal relationships in the combination of right wrist movement and left knee movement. Other features can be similarly annotated in the model. The annotated features can be stored in a feature vector and correlated with each shot fired for analysis of how different features combine to result in a particular shot performance.

[0099] FIG. 18 shows a pruned model 1800 that allows for a deeper dive into various features and their interrelationships with one another. For example, reviewing the right_knee_phi 1802 feature (e.g., the bounding box rotation associated with the movement of the right knee during the shooting string) can reveal values ​​of 0.8, 0.82, and 0.9. Given these values, it can be seen that a right_knee_phi value of 0.9 will yield one result at the first result node 1806. It can be seen that a right_knee_phi 1802 of 0.8 or 0.82 and a right_shoulder_l bounding box length 1804 of 0.09 will yield a second result at the second result node 1808 and the third result node 1810. In some cases, the second result node 1808 will be associated with poor shooting performance, and the third node 1810 will be associated with good shooting performance. These nonlinear causal relationships can be determined by machine learning models, allowing the system to determine which movements result in better shooting outcomes and which movements result in worse shooting outcomes by running one or more machine learning algorithms.

[0100] In some cases, an isolation forest encodes a dataset of trees such that leaf instances per leaf are identical. Node splits may be chosen randomly rather than to reduce impurity of child target values. In some cases, a node is a leaf when the training instances at the node have the same target value. In some examples, rare instances in the training dataset reach the leaf sooner than less rare instances. According to some embodiments, new anomalous instances have a shorter path relative to the tree depth for all trees in the forest, making anomaly identification more efficient.

[0101] In some cases, Classification and Regression Trees (CART) are predictive models that explain how the result variable is predicted based on other values. A CART-style decision tree is one where each fork is a split of a predictor variable and each end node contains a prediction of the result variable. This constructs a binary tree to partition the feature space into segments that are homogeneous with respect to the target variable, and is a recursive algorithm that performs binary splits on the input features based on a specific criterion to create a tree-like structure. A CART-style decision tree can be useful for predicting the result of a shot based on the motion data of one or more body landmarks. A CART-style decision tree is a multi-category classifier (e.g., N>1), while an isolation tree is a single-category classifier (e.g., "is it an anomaly or not"). In some cases, a CART-style decision tree can be made into an isolation tree by designating M<N categories as "not anomalous" and N-M as "anomalous" and pruning the branches that reach the leaf with N-M "anomalous" up to the root node. This may be similar to training an isolation tree with M<N "not anomalous" instances, which allows for impurity-reducing splits and impure "not anomalous" leaves.

[0102] FIG. 19A shows a gantry 1900 system that can provide a portable mounting structure for housing one or more imaging devices, including one or more video cameras. In some cases, the structure includes one or more support columns 1902 and one or more crossbars 1904. The gantry structure 1900 can be positioned around the shooter or, in some cases, downrange from the shooter. For example, the gantry 1900 can have the crossbar 1904 positioned approximately 1 foot (≈3 m) to 8 feet (≈2.4 m) above the shooter and between 1 foot (≈3 m) and 15 feet (≈4.5 m) in front of the shooter. In some examples, a camera is positioned on each support column and on the crossbar. Thus, in some embodiments, two, three, or more cameras can be positioned on the gantry, with some of the cameras pointed at the shooter and one or more additional cameras pointed downrange toward the target.

[0103] 19B, 19C, and 19D show various views captured by cameras mounted on the gantry 1900. A first camera 1908 may be mounted on the crossbar 1904 or support column 1902 and is positioned to capture a left side view of the shooter (FIG. 19B). The camera may be configured to capture the shooter's entire body or the shooter's upper body and head.

[0104] A second camera 1910 may be mounted on the crossbar 1904 or support 1902 and configured to capture a right side view of the shooter (FIG. 19C). The camera may be configured to capture the shooter's entire body or the shooter's upper body and head. In some cases, the first camera 1908 and the second camera 1910 utilize different fields of view such that one camera captures the shooter's entire body and the other camera captures only a portion of the shooter's body.

[0105] A third camera 1912 may be positioned on the crossbar 1904 and configured to capture an overhead view 1914 (FIG. 19D) of the shooter. By positioning the cameras to capture the shooter's movements from various angles, the system can correlate the video data from each camera and determine three-dimensional motion data of selected body landmarks.

[0106] A fourth camera may be positioned to capture video data of the target 1916 (FIG. 19E) and the impact of the shots on the target 1918. The video data from each of the cameras may be synchronized and analyzed to determine when shots were fired and to correlate the fired shots with registered hits or misses on the target.

[0107] FIG. 20 illustrates a computer program user interface 2000 usable with the systems and methods described herein. For example, the computer program may be configured to receive video data from one or more cameras, synchronize the video data, determine when shots were fired, and register and score hits or misses on the target. The user interface 2000 may include a start record button 2002 that enables the shooter to begin video capture. In some cases, the shooting string may be time, and the start record button may further start a timer. The user interface may further include a timer 2004 associated with the session. The user interface may be presented on a mobile computing device associated with the user or on a mobile computing device associated with the facility. For example, a gantry system may be installed at the facility, and a computing device associated with the facility may be connected to the gantry system, configured to receive the video data, and provide performance feedback for the embodiments described herein.

[0108] The user interface may be provided on any suitable display, such as a television, touchscreen display, tablet screen, smartphone screen, or any other visual computer interface. With further reference to FIG. 21 , the user interface 2000 may display indicia associated with the shooting string, such as video of the shooting string and movements during the shooting string 2102. The video may be displayed in a playback window, providing controls 2104 for playing, pausing, adjusting the volume, and scrubbing the video. Additionally, there may be controls for selecting different views 2106, i.e., controls that allow a viewer to select video clips captured by different cameras during the shooting string to view different views of the shooting string, either individually or in an integrated view, such as a side-by-side view. These views may be synchronized to allow viewers to view different views of the same event simultaneously.

[0109] The user interface 2000 may further show a target 2110 and may identify hits 2112 that the system has registered on the target. The user interface 2000 may further display the score of the most recent shot 2114, along with the average score for the string 2116. Of course, other information may be displayed as desired by the user, including training tips, motion data that contribute to off-center shots, or other mistakes during the shooting string.

[0110] 22 shows an additional view of a user interface 2000 that allows a user to specify a body landmark selection 2202. In addition, the user interface 2000 provides a selection for signal settings 2204 that allows the user to specify details of Y coordinate movement or X coordinate movement. In response to a user selection, the user interface 2000 may display motion data 2206 associated with that selection. This type of review and analysis allows the shooter to very specifically see the movement of individual body landmarks during the firing string, and additionally, can specifically see horizontal movement, vertical movement, or both for review.

[0111] FIG. 23 illustrates some of the features and techniques employed by embodiments of the described systems and methods. In many embodiments, the disclosed systems utilize machine vision (e.g., computer vision) 2302 to track body landmarks of a participant (e.g., the shooter) and also track target changes to provide automatic target scoring 2304. The systems and methods may also utilize audio processing 2306 to enable shot time detection 2308, which may also be combined with computer vision techniques. The disclosed systems and methods may also utilize signal processing 2310 to provide shooter position correction 2312, including pose, posture, movement, grip, trigger pull, etc. The systems and methods described herein also apply machine learning 2314 to determine the causality of off-center shots, which may include analysis of the shooter's pose, grip 2316, trigger pull, stance, and body landmark movement, among others.

[0112] 24 shows a method 2400 according to an embodiment described herein. The system may receive video data and optionally audio data. The system may be configured to detect 2402 and score shots through image processing on the video data of the target. The system may process the video data to determine fired shots and a shooter pose analysis 2404. Additionally, the system may analyze audio data for shot detection 2406.

[0113] Shot detection 2402 provides 2408 coordinates (e.g., x,y coordinates) of the shot within the target. Shooter pose analysis 2404 provides 2410 coordinates (e.g., x,y coordinates and optionally z coordinates) of body landmarks during the shooting process. Audio shot detection 2406 provides 2412 the precise time of the shot. Shooting analysis may include one or more machine learning algorithms that receive the shot and body data and, through the machine learning algorithms, detect and predict the causality of the shooter's shot performance with the firearm and ammunition combination, which may be referred to as performing shooting analysis 2414. For example, embodiments of the system may generate one or more of a session score 2416, shooting position recommendations 2418, shot misses, and grip analysis 2420, among others.

[0114] 25 illustrates a flow of a sample process 2500 for using machine vision and machine learning to track participants' body landmarks and generate recommendations for improvement. As noted herein, the systems and methods described herein can be used in any competition, such as a sporting event, that benefits from repeatability and accurate body kinematics. In addition to shooting, some such competitions include archery, golf, bowling, darts, running, swimming, pole vaulting, football, baseball, basketball, hockey, and many other types of sports. In any case, the system is configured to track participants' body landmarks and determine how to change their body movements to improve performance.

[0115] In block 2502, the system receives video data of the participants, which may be obtained from a single image capture device, or two image capture devices, or three or more image capture devices. The image capture devices may be any suitable imaging devices configured to capture continuous images of the participants, and may include any consumer or professional video camera, including cameras typically built into mobile computing devices.

[0116] In block 2504, the system determines one or more body landmarks for the participant. The body landmarks may be associated with any joint, body part, limb, or location associated with the joint, limb, or body part. In some cases, the system generates the wireframe based on one or more body landmarks and may not use all body landmarks in generating the wireframe model.

[0117] Body landmarks are tracked during performance of the activity to generate motion data at block 2506. As a non-limiting example, body landmarks may be created for a golfer's hands, wrists, arms, head, shoulders, torso, waist, knees, ankles, and feet that may be tracked during a golf club swing.

[0118] At block 2508, a score is determined and associated with the performance. As described, in activities involving a projectile, the score may be associated with the trajectory or destination of the projectile. In the case of golf, for example, the score may be based on distance, direction, or proximity to the target, or a metric associated with the participant's average or past performance. In short, any metric may be used to evaluate the quality of the performance results.

[0119] At block 2510, the score is associated with the performance, i.e., the performance may be linked to the determined score and stored for later analysis to determine trends in performance over time or to compare one performer to another.

[0120] In block 2512, the system generates recommendations for altering movements in subsequent performances to improve results. In some cases, the recommendations may involve the hands, including grips on a firearm, golf club, bat, stick, etc. The recommendations may also include changes or shifts in weight distribution. The recommendations may include movements of the hands, head, shoulders, body, legs, feet, or other body parts. Often, the recommendations include suggestions for altering the movement of one or more body landmarks to improve the performance score in subsequent attempts.

[0121] As described above, in some embodiments, the system utilizes a gantry-style arrangement that may use multiple recording devices to capture audio and video of participants and / or targets. In some cases, an imaging device may be aimed at the target, which may have a zoom lens, digital zoom capabilities, or rely on an external lens, such as a camera mounted on a spotting scope, to capture images of targets located downrange. In some cases, the system is configured to synchronize multiple sources, such as one or more video frames and / or audio data from one or more audio capture devices. In some cases, the system is configured to synchronize multiple video frames and audio data from one or more audio / video capture devices.

[0122] The system may utilize one or more machine learning models for synchronization, prediction, and validation, and may be further trained to analyze scoring data and associate it with motion data to determine correlations between specific motion data (e.g., behaviors) and scoring trends. As an example, the system may correlate a shooter's wrist pivoting downward before a shot with typically scoring outside and below the ten ring and determine that the shooter is anticipating the recoil of the firearm before the shot. The system may then provide feedback to the user with information related to the motion / score correlation, as well as one or more exercises or drills to enable the user to recognize and address behaviors that result in reduced scores. As described elsewhere herein, a similar process may be used for any motion data from any activity or sport.

[0123] In some embodiments utilizing multiple video capture devices, the system may track body motion in two or three dimensions from multiple angles. The two or three dimensional body motion data may be correlated, synchronized, and analyzed to determine two or three dimensional motion data that may be further correlated with the resulting score.

[0124] Although the described system embodiments are described in the context of a shooter firing a series of shots, it should be understood that the systems and methods described herein are applicable to capturing any type of physical motion, as well as to other sports where physical motion can lead to performance. For example, the system embodiments described herein can record the motion of a basketball free throw, a golf swing, a figure skating element, archery, soccer, a baseball swing, or a series of observable physical landmarks over time, and can be used to track, critique, and improve physical motion, such as any other sport or movement, where the motion has any observable causal consequences.

[0125] The system may include one or more processors and one or more computer-readable media that may store various modules, applications, programs, or other data. The computer-readable media may include instructions that, when executed by the one or more processors, cause the processors to perform the operations described herein for the system.

[0126] In some implementations, the processor(s) may include a central processing unit (CPU), a graphical processing unit (GPU), both a CPU and a GPU, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or additionally, the functions described herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), etc. Additionally, each of the processor(s) may possess its own local memory, which may also store program modules, program data, and / or one or more operating systems. One or more control systems, computer controllers, and remote controls may include one or more cores.

[0127] Embodiments may be provided as a computer program product including a non-transitory machine-readable storage medium having stored thereon instructions (in compressed or uncompressed format) used to program a computer (or other electronic device) to perform the processes or methods described herein. Computer-readable media may include volatile and / or non-volatile memory, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Machine-readable storage media include, but are not limited to, hard drives, floppy disks, optical disks, CD-ROMs, DVDs, read-only memory (ROM), random-access memory (RAM), EPROMs, EEPROMs, flash memory, magnetic or optical cards, solid-state memory devices, or other types of media / machine-readable media suitable for storing electronic instructions. Furthermore, embodiments may be provided as a computer program product including a transitory machine-readable signal (in compressed or uncompressed format). Examples of machine-readable signals include, but are not limited to, signals (including signals downloaded over the Internet or other networks) that can be configured to be accessed by a computer system or machine that hosts or executes a computer program, whether or not modulated using a carrier wave.

[0128] Those skilled in the art will recognize that any process or method disclosed herein can be modified in many ways. The process parameters and order of steps described and / or illustrated herein are given by way of example only and can be changed as desired. For example, although the steps illustrated and / or described herein may be shown or described in a particular order, these steps do not necessarily have to be performed in the order shown or described.

[0129] The various exemplary methods described and / or illustrated herein may omit one or more of the steps described or illustrated herein or may include additional steps in addition to those disclosed. Furthermore, the steps of any method as disclosed herein may be combined with any one or more steps of any other method as disclosed herein.

[0130] This disclosure describes exemplary embodiments and is therefore not intended to limit the scope of the embodiments of the present disclosure and the appended claims in any way. The embodiments are described above using functional components that illustrate implementations of specified functions and their relationships. The boundaries of these functional components have been arbitrarily defined herein for the convenience of description. Alternative boundaries may be defined to the extent that the specified functions and their relationships are appropriately performed.

[0131] The foregoing description of specific embodiments sufficiently clarifies the general nature of the embodiments of the present disclosure so that others, by applying the knowledge of those skilled in the art, can easily modify and / or adapt such specific embodiments for various applications without undue experimentation and without departing from the general concept of the embodiments of the present disclosure. Therefore, such adaptations and modifications are intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein. The phrases or terms used herein are for the purpose of description, not limitation, and should be interpreted by those skilled in the art in light of the teaching and guidance presented herein.

[0132] The breadth and scope of embodiments of the present disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

[0133] In particular, conditional language such as "can," "could," "might," or "may," unless expressly stated otherwise or understood otherwise within the context in which it is used, is generally intended to convey that certain implementations include certain features, elements, and / or operations, but may not include them in other implementations. Thus, such conditional language is generally not intended to imply that features, elements, and / or operations are somehow required in one or more implementations, or that one or more implementations necessarily include logic for determining whether those features, elements, and / or operations should be included or performed in any particular implementation, with or without user input or prompting.

[0134] Unless otherwise noted, as used herein, the terms "connected to" and "coupled to" (and their derivatives) should be interpreted as allowing both direct and indirect (i.e., via other elements or components) connections. Additionally, as used herein, the terms "a" or "an" should be interpreted as meaning "at least one of." Finally, for ease of use, as used herein, the terms "including" and "having" (and their derivatives) should be interpreted as open-ended and not excluding additional components.

[0135] This specification and the accompanying drawings disclose examples of systems, apparatus, devices, and techniques that may provide systems and methods for determining the acoustic signature of a fired firearm. For purposes of describing various features of the present disclosure, it is, of course, not possible to describe every conceivable combination of elements and / or methodologies, but those skilled in the art will recognize that many further combinations and permutations of the disclosed features are possible. Accordingly, various modifications can be made to the present disclosure without departing from the scope or spirit of the disclosure. Moreover, other embodiments of the present disclosure will be apparent from consideration of the specification and accompanying drawings, as well as from practice of the disclosed embodiments presented herein. The examples presented in this specification and the accompanying drawings are to be considered in all respects as illustrative and not restrictive. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

[0136] Those skilled in the art will understand that in some implementations, the functionality provided by the processes and systems described above may be provided in alternative ways, for example, divided among more software programs or routines or integrated into fewer programs or routines. Similarly, in some implementations, the illustrated processes and systems may provide more or less functionality than described, such as when other illustrated processes, respectively, lack or include such functionality, or when the amount of functionality provided is varied. Additionally, while various operations may be shown as being performed in a particular way (e.g., sequentially or simultaneously) and / or in a particular order, those skilled in the art will understand that in other implementations, these operations may be performed in other orders and in other ways. Those skilled in the art will also understand that the data structures described above may be structured differently, for example, by dividing a single data structure into multiple data structures or by combining multiple data structures into a single data structure. Similarly, in some implementations, the illustrated data structures may store more or less information than described, such as when other illustrated data structures, respectively, lack or include such information, or when the amount or type of information stored is varied. The various methods and systems shown in the figures and described herein represent example implementations. In other implementations, the methods and systems may be implemented in software, hardware, or a combination thereof. Similarly, in other implementations, the order of any method may be changed, and various elements may be added, rearranged, combined, omitted, modified, etc.

[0137] From the foregoing, it will be understood that, although specific implementations have been described herein for illustrative purposes, various modifications may be made without departing from the spirit and scope of the appended claims and the elements described therein. In addition, while certain aspects are presented below in certain claim forms, the inventors contemplate various aspects in any available claim form. For example, while only some aspects may currently be described as being embodied in a particular configuration, other aspects may likewise be so embodied. Various modifications and changes may be made, as would be apparent to one skilled in the art having the benefit of this disclosure. All such modifications and changes are intended to be encompassed, and therefore the above description should be regarded in an illustrative rather than a limiting sense.

Claims

1. 1. A method for improving shooting performance, comprising: receiving video data of the shooter; determining one or more body landmarks of the shooter; tracking the one or more body landmarks during a shot to generate shot motion data; determining a score for said shot; Associating the shot motion data with the score; generating recommendations for altering the motion data in subsequent shots; A method comprising:

2. The method of claim 1 , wherein determining one or more body landmarks of the shooter comprises generating a wireframe model by connecting the body landmarks.

3. 2. The method of claim 1, wherein associating the shot motion data with the score comprises executing a classification and regression tree machine learning model to identify a causal relationship between the shot motion data and the score.

4. The method of claim 1 , further comprising determining the shooter's grip through image analysis of the video data of the shooter.

5. 5. The method of claim 4, further comprising analyzing the grip of the shooter and providing, on a display screen, grip recommendations for changing the grip.

6. Determining the score for the shot comprises: receiving target video data; performing image analysis on the received target video data; determining a hit on the target; determining a score for said hit; The method of claim 1 , comprising:

7. The method of claim 1 , wherein receiving the video data includes capturing the video data by a mobile phone.

8. The method of claim 1 , wherein determining one or more body landmarks includes determining 17 body landmarks.

9. The method of claim 1 , wherein tracking the one or more body landmarks comprises generating a bounding box around each of the one or more body landmarks.

10. The method of claim 1 , further comprising running a machine learning model to correlate the shot motion data with the score.

11. The method of claim 10 , wherein the machine learning model is configured to determine the motion data that results in an off-center hit on a target.

12. 1. A method for improving the causal consequences of bodily movement, comprising: receiving video data of the body movement; determining one or more body landmarks visible in the video data of the body movement; tracking the one or more body landmarks during an action; generating motion data based at least in part on tracking the one or more body landmarks; determining a score associated with the motion data; associating the performance data with the score; generating recommendations for altering the operational data for subsequent actions; A method comprising:

13. The method of claim 12 , wherein receiving the video data includes capturing the video data with a mobile computing device.

14. The method of claim 12 , further comprising running a machine learning model to correlate the operational data with the score.

15. The method of claim 14 , wherein the machine learning model is configured to determine the behavioral data that results in a reduced score.

16. The method of claim 15 , further comprising predicting, by the machine learning model, a predicted score based on the operational data.

17. The method of claim 16 , further comprising comparing the predicted score to the score.

Images