Systems and methods for improving marksmanship through machine learning

Abstract

The physical movement improvement system is configured to receive video data, track one or more body landmarks associated with the movement to determine movement of the body landmarks during the action, correlate the movement of the body landmarks with a score for the action, and determine movements that are detrimental to the score for the action using machine learning techniques. The system can also recommend drills to the participant to improve the score for the action.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 406,245, entitled "SYSTEMS AND METHODS FOR MARKSMANSHIP IMPROVEMENT THROUGH MACHINE LEARNING," filed September 13, 2022, and U.S. Provisional Patent Application No. 63 / 406,208, entitled "SYSTEMS AND METHODS FOR AUTOMATED TARGET IDENTIFICATION, CLASSIFICATION, AND SCORING," filed September 13, 2022, and U.S. Provisional Patent Application No. 63 / 406,241, entitled "SYSTEMS AND METHODS FOR MARKSMANSHIP DIGITIZING AND ANALYZING," filed September 13, 2022, the contents of which are incorporated herein by reference in their entireties. [Background technology]

[0002] Target shooting is enjoyed by millions of people annually, and by many reports, the number of people who regularly practice target shooting has increased over the past decade and continues to grow. In the United States alone, it is estimated that over 52 million people regularly practice target shooting. There are many different types of recreational shooting activities, ranging from simple plinking with a handgun or rifle against paper or steel targets, to skilled long-range rifle shooting competitions, which require advanced training and skill, to fun, fast-paced pistol shooting against popping or stationary targets, or shotgun shooting against skeet, trap, sporting clays, and other games. Apart from recreational shooting, an increasing number of target shooters practice as part of their professions, such as law enforcement, the military, and security personnel.

[0003] While many participants are satisfied with their current skill level, there is an increasing number of shooters who want to improve their skills. However, in some cases, many participants do not know how to improve. The number of people at shooting ranges continues to grow, and these participants, whether for sport, recreation, self-defense, or public defense, want to improve their skills. However, without individual shooting instruction, and often even if individual instruction is provided, it can be difficult to quantify improvements and issues affecting accuracy. Furthermore, many participants do not know how to improve.

[0004] Therefore, there is a need for a system and method that can analyze participants' habits and provide feedback and recommendations for improvement. Further, there is a need for a system that can provide the aforementioned benefits in near real time using consumer devices. These and other advantages will be readily apparent from the following disclosure. Summary of the Invention

[0005] One or more computer systems may be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system that causes the system to perform the actions during operation. One or more computer programs may be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions. One general aspect includes a method for improving shooting performance. The method also includes receiving video data of a shooter, determining one or more body landmarks of the shooter, tracking the one or more body landmarks during a shot to generate shot motion data, determining a score for the shot, associating the shot motion data with the score, and generating recommendations for changing the motion data in subsequent shots. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0006] Implementations may include one or more of the following features. In this method, determining one or more body landmarks of the shooter may include generating a wireframe model by connecting the body landmarks. Associating the shot motion data with the score may include running a classification and regression tree machine learning model to identify a causal relationship between the shot motion data and the score. The method may include determining the shooter's grip through image analysis of the shooter's video data. The method may include analyzing the shooter's grip and providing, on a display screen, a grip recommendation for changing the grip. Determining the score of the shot may include receiving target video data, performing image analysis on the received target video data, determining a hit on the target, and determining a score for the hit. Receiving video data includes capturing the video data with a mobile phone. Determining one or more body landmarks includes determining 17 body landmarks. Tracking the one or more body landmarks includes generating a bounding box around each of the one or more body landmarks. The method may include running a machine learning model to correlate the shot motion data with the score. The machine learning model is configured to determine motion data that results in off-center hits of the target. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0007] One general aspect includes a method for improving a causal outcome of a physical movement. The method also includes receiving video data of the physical movement, determining one or more body landmarks visible in the video data of the physical movement, tracking the one or more body landmarks during the action, generating motion data based at least in part on tracking the one or more body landmarks, determining a score associated with the motion data, associating the motion data with the score, and generating a recommendation for altering the motion data in a subsequent action. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0008] Implementations may include one or more of the following features: In the method, receiving video data includes capturing the video data by a mobile computing device. The method may include executing a machine learning model to correlate the behavioral data with a score. The machine learning model is configured to determine behavioral data that results in a reduced score. The method may include predicting, by the machine learning model, a predicted score based on the behavioral data. The method may include comparing the predicted score to the score. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium. [Brief explanation of the drawings]

[0009] The accompanying drawings are part of this disclosure and are incorporated herein. The drawings illustrate examples of embodiments of the present disclosure and, together with the description and claims, serve to explain, at least in part, various principles, features, or aspects of the present disclosure. Specific embodiments of the present disclosure are described more fully below with reference to the accompanying drawings. However, various aspects of the present disclosure may be implemented in many different forms and should not be construed as limited to the implementations set forth herein. Like numbers refer to similar, but not necessarily the same or identical, elements throughout. [Figure 1A] 1 shows an image of a shooter captured by an image capture device, according to some embodiments. [Figure 1B] 1B illustrates a wireframe model of the shooter of FIG. 1A, according to some embodiments. [Figure 2] 1 illustrates motion data associated with multiple body landmarks, according to some embodiments. [Figure 3] 1 illustrates motion data associated with multiple body landmarks and automatic shot detection based on the motion data, according to some embodiments. [Figure 4] 1 illustrates analyzing motion data associated with multiple body landmarks and identifying shooting errors based on the motion data, according to some embodiments. [Figure 5] 1 illustrates analyzing motion data associated with multiple body landmarks and identifying shooting errors based on the motion data, according to some embodiments. [Figure 6] 1 illustrates analysis of motion data associated with multiple body landmarks according to some embodiments. [Figure 7] 1 illustrates the analysis of motion data associated with multiple body landmarks and identifying pose changes that result in the least improvement in performance, according to some embodiments. [Figure 8A] 1 shows an image of a shooter captured with an imaging device, according to some embodiments. [Figure 8B]8B illustrates a computer-generated wireframe model of the shooter of FIG. 8A with identified body landmarks, according to some embodiments. [Figure 9] 1 illustrates motion data associated with multiple body landmarks and identifying physical motions that lead to poor performance, according to some embodiments. [Figure 10] 10A and 10B show a diagnostic target that identifies shooter behavior based on shot patterns, according to some embodiments. [Figure 11] 1 illustrates an image of a shooter's hands and a computer-generated wireframe model associated with body landmarks that identify and analyze the shooter's grip, according to some embodiments. [Figure 12A] FIG. 1 is a diagram of a computer software program user interface for capturing video images, tracking body landmark movements, and analyzing shooter pose and motion data, according to some embodiments. [Figure 12B] FIG. 1 is a diagram of a computer software program user interface for analyzing a shooter's performance and automatically scoring the performance, according to some embodiments. [Figure 13A] 1 illustrates a system for digitizing and analyzing marksmanship, according to some embodiments. [Figure 13B] 1 illustrates a system for digitizing and analyzing marksmanship, according to some embodiments. [Figure 13C] 1 illustrates a system for digitizing and analyzing marksmanship, according to some embodiments. [Figure 14] 1 illustrates a system for digitizing and analyzing marksmanship, according to some embodiments. [Figure 15] 1 illustrates the logic of a machine learning algorithm for quantifying shot samples, according to some embodiments. [Figure 16]1 illustrates a sample decision tree machine learning model that correlates body landmark motion data with shot performance, according to some embodiments. [Figure 17] 1 illustrates an annotated decision tree machine learning model that correlates body landmark motion data with shot performance, according to some embodiments. [Figure 18] 1 illustrates a pruned decision tree machine learning model that correlates body landmark motion data with shot performance, according to some embodiments. [Figure 19A] 1 illustrates a system for capturing video data of a shooter and a target, according to some embodiments. [Figure 19B] 10 illustrates image data of a left side view of a shooter captured by an imaging device, according to some embodiments. [Figure 19C] 10 illustrates image data of a right side view of a shooter captured by an imaging device, according to some embodiments. [Figure 19D] 1 illustrates image data of an overhead view of a shooter captured by an imaging device, according to some embodiments. [Figure 19E] 1 illustrates image data of a target captured by an imaging device, according to some embodiments. [Figure 20] 1 illustrates a sample user interface of a computer application configured to receive, analyze, and score marksmanship performance, according to some embodiments. [Figure 21] 1 illustrates a sample user interface of a computer application configured to receive, analyze, and score marksmanship performance, showing shot placement and automatic scoring, according to some embodiments. [Figure 22]1 illustrates a sample user interface of a computer application configured to receive, analyze, and score a shooting skill performance, allowing selection of a body landmark and showing motion data associated with the selected body landmark, according to some embodiments. [Figure 23] Some of the key technologies enabled by the embodiments described herein are presented along with the features they facilitate. [Figure 24] 10A-10C illustrate sample user interfaces of a computer application configured to receive, analyze, and score shooting skill performance, illustrating a system that uses multiple source collaboration and machine learning to detect fired shots, analyze shooting position and provide recommendations for improvement, and analyze grip position and provide recommendations for improvement, according to some embodiments. [Figure 25] 1 is a process flow for capturing operational data, analyzing the operational data, and generating recommendations for improvement, according to some embodiments. [Figure 26] 1 is a process flow for correlating firearms with individual shooters, according to some embodiments. [Figure 27] 1 illustrates a sample system for determining an acoustic signature, according to some embodiments. [Figure 28] 1 illustrates a sample flowchart for pre-training a DNN, according to some embodiments. [Figure 29] 1 shows a sample flowchart for deriving spectrogram weights according to some embodiments. [Figure 30] 1 shows a sample flowchart for classifying and determining an acoustic signature. [Figure 31] 1 illustrates a system configured for automated scoring of shooting targets, according to some embodiments. [Figure 32] 1 shows a sample process flow for classifying and scoring targets, according to some embodiments. [Figure 33] 1 illustrates a sample process flow for identifying and classifying targets, according to some embodiments. [Figure 34] 1 illustrates a sample process flow for registering targets and determining scoring hits, according to some embodiments. [Figure 35] 35A, 35B, and 35C show a method for initializing a target scoring system to identify a target, according to some embodiments. [Figure 36] 1 illustrates a sample process flow for detecting impact on a target, according to some embodiments. [Figure 37] 1 illustrates a sample process flow for scoring target impacts, according to some embodiments. [Figure 38] 1 illustrates a sample user interface for automated target scoring in a software application, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0010] Body Landmarks and Pose Estimation

[0011] According to some embodiments, a system is described that uses computer vision and machine learning to significantly reduce the manual and labor-intensive nature of current training techniques, thus providing a system that continuously learns and adapts. According to some embodiments, the system includes a machine vision and machine learning system that tracks and estimates a participant's pose and can determine positive and negative factors that affect the quality of participation, such as pose, movement, anticipation, recoil, grip, and stance, among others. This may be primarily performed by a computer vision system that can simultaneously track several body landmarks and, in some cases, correlate the movement of one or more body landmarks with the accuracy of the shooting skill. By way of example, the system may identify and track any number of body landmarks, such as three, five, eleven, seventeen, twenty-one, twenty-five, thirty, or more body landmarks. Because the system tracks the location of landmarks, which may be in two or three dimensions, the system can correlate the movement of the landmarks with bullets fired downrange and the scoring of individual bullets. Detection of a bullet fired downrange may be determined by the movement of one or more suitable markers, such as the movement of a participant's hand or wrist (or other body marker) in response to recoil, the sound of a gunshot, a pressure wave associated with a muzzle blast, a target hit, or some other marker.

[0012] The system may further monitor the participant from one, two, three, or more viewpoints, analyze the movement of each body landmark, and further monitor the accuracy of the shot and correlate the movement of the body landmark with the accuracy. Based on the accuracy, the system may further provide an analysis of the body movements that contribute to the less-than-perfect accuracy and further suggest ways to improve the body movements to increase the accuracy.

[0013] Motion capture may be performed by one or more cameras generally pointed at the participant and one or more cameras pointed at a target. In some cases, one or more of the cameras are associated with mobile computing devices such as smartphones, tablets, laptops, personal digital assistants, and wearable devices (e.g., watches, glasses, body cams, smart hats, etc.). In some cases, the wearable devices may include sensors such as accelerometers, vibration sensors, motion sensors, or other sensors to provide motion data to the system. In some embodiments, the system tracks body marker positions over time and generates a motion plot.

[0014] Referring to FIG. 1A , one or more cameras may capture one or more views of a shooter 100. The cameras may capture video data of the shooter as he or she draws, aims, fires, reloads, and / or adjusts position. A computer system may receive the video data, analyze the video data, and create a model associated with the shooter, such as that in FIG. 1B . In some cases, the computer system may identify body landmarks and connect them to a wireframe model 102 that tracks the shooter's pose and movement of the body landmarks. In some instances, the body landmarks may include one or more of the nose, left ear, left eye, left hip, left knee 103, right ear, right eye, right hip, left ankle, left elbow, left wrist, right knee 104, right ankle, right elbow 106, right wrist 108, left shoulder, and right shoulder 110. Of course, other body landmarks are possible, but for efficiency, we will focus on these 17 body landmarks throughout this disclosure. In some embodiments, a single camera may capture two-dimensional motion data associated with one or more of the body landmarks. In some examples, two or more cameras may be used to capture three-dimensional motion data of one or more of the body landmarks.

[0015] The body landmarks may be tracked over time, such as during a shooting string (e.g., a shooting session), and the movement of one or more of the body landmarks may be tracked during this time. In some cases, two-dimensional movement is tracked in the x and y directions, corresponding to side-to-side and vertical movement. In some cases, three-dimensional movement of the body landmarks is tracked in the x, y, and z directions.

[0016] Referring to FIG. 2, a graph of selected body landmarks 200 is shown. The body landmarks are user-selectable by the user to focus on individual body landmarks or combinations of body landmarks for review. For example, the top line 202 represents the right wrist of a right-handed shooter in a horizontal direction, and the third line 203 represents the vertical direction of the right wrist over time. As can be seen, the wrist moves up and down during the motion capture process. In some cases, the system may correlate the movement of one or more body landmarks with events or stages during the shooting string.

[0017] For example, during the first stage 204, the right wrist is in a relatively low position, and the system may correlate this position and movement with a ready-to-fire state. The second stage 206 shows the right wrist moving upward in a very short interval, which may correlate with the act of drawing a pistol from a holster. The third stage 208 shows the right wrist remaining fairly stable in the vertical plane, but with sharp peaks 207a-207c in movement, which may correlate with a shot being fired from the pistol.

[0018] During the fourth stage 210, the right wrist moves downward and returns to a firing position. The system may correlate this movement with a reloading motion. In some cases, the system is trained with training data, which may be supervised learning, to correlate similar movements with various stages.

[0019] During the fifth stage 212, the shooter's right wrist initially moves upward as the shooter takes aim, then settles on the target, followed by peaks of movement 213a-213c, which may correlate with the shot being fired downrange.

[0020] In the sixth stage 214, the right wrist moves downward again to its initial position, which may correlate with holstering the pistol.

[0021] While this example focuses on the shooter's right wrist in the vertical direction, it is clear that any body landmark can be viewed and analyzed, and movements or combinations of movements can be correlated with events or actions by the shooter, including groups of body landmarks.

[0022] Referring to FIG. 3 , a close-up view of the shooting stage 300 is depicted, showing sharp peaks in the vertical movement of the right wrist 302 and the right elbow 304. The system can analyze the motion data and automatically determine when a shot was fired. The system can be configured to correlate sharp peaks in the vertical movement of the wrist and / or elbow with fired shots. As shown in FIG. 3 , each of the arrows 306 can correspond to a fired shot. Additionally, audio data can be correlated with the motion data to provide additional cues as to when a shot was fired. In some cases, audio data can be combined and / or synchronized with the motion data to provide additional details about the fired shot. The motion data can be used to infer additional information about the shooter, their habits, their posture, and other cues that can catch the shooter's attention as part of an effort to improve their accuracy.

[0023] Referring to FIG. 4 , motion data 400 is displayed for a shooter's right wrist 402 and right elbow 404. In some cases, the system may apply a trend curve 406 to the motion data, which may represent normalized motion data. Additionally, the system may make inferences and / or decisions based on the motion data 400. For example, as shown in FIG. 4 , at 408, a shooter aims a firearm at a target and attempts to hold the firearm steady; at 410, the shooter lowers their wrist and elbow, and shortly thereafter, a shot is fired 412. The system may recognize this pattern and determine that the movement of lowering the wrist and / or elbow immediately prior to the shot is evidence that the shooter is anticipating and preparing for the recoil of the firearm. In many cases, the shooter moves the firearm away from the target in anticipation of the recoil that will occur when the shot is fired, dramatically reducing the accuracy of the shot when the recoil is anticipated. Similar examples of behavioral data that may reduce a shooter's accuracy include flinching, pre-ignition pushing, trigger jerking, and closing the eyes, among others.

[0024] In some cases, the system may provide information to the shooter regarding recoil anticipation, information that may include one or more drills or practice sessions as part of an effort to improve the shooter's performance data regarding recoil anticipation. For example, the system may identify a practice regimen that may include dry firing, skipping reloads (e.g., mixing live ammunition with a magazine of dummy ammunition), or other skill-building drills.

[0025] 4 also shows that there is a drift in the shooter's posture. For example, before the first shot 412, the shooter's right wrist and right arm are at a first vertical height 414, and before the second shot 416, the shooter's right wrist and right elbow are at a second vertical height 418, higher than the first vertical height. This data indicates that the shooter did not return to the same position from the first shot to the second shot, resulting in a slightly different sight picture for the shooter, which may reduce the accuracy of subsequent shots.

[0026] 5, motion data 500 further illustrates drift. Motion data associated with a right-handed shooter's right wrist 502 and right elbow 504 not only shows recoil anticipation, indicating a downward movement of body parts immediately prior to the shot, but also shows trend lines 506 demonstrating a continuous upward drift of the shooter's wrist and elbow during the firing string. If the shooter does not return to the same position between shots, the accuracy and precision of the shots fired within the string can be dramatically reduced.

[0027] In some cases, the system may recognize drift in one or more of the body landmarks and provide this information to the shooter. In some cases, the system provides information on a display screen associated with a mobile computing device. For example, the system may be implemented on a mobile computing device associated with the shooter, and the display screen on the mobile computing device may provide information, instructions, or practice drills to the shooter to improve drift and resulting accuracy problems.

[0028] Similarly, the system may correlate the movement of body landmarks with other correctable imperfections in the shooter's position or posture. For example, referring to FIGS. 6 and 7, which illustrate body landmark motion data 600, the motion data 600 shows the movement of multiple body landmarks during the shooting string. The bottom line graph shows right ankle motion data 602, demonstrating that the shooter changed the position of his right foot. Changing position during the shooting string likely impacts the sight picture, accuracy, precision, and other metrics associated with shooting. The system may determine whether the change in foot position was positive or negative with respect to the scoring of the shot and provide recommendations to the shooter based on this change in posture. The system may look at the shooting accuracy and / or precision of shots 604a-604e fired both before and after the foot repositioning to determine whether moving the foot had a positive or negative impact on shooting performance.

[0029] In some cases, the system can correlate target scoring (e.g., shot accuracy and / or precision) with body landmarks and movements to show the shooter which positions of individual body landmarks influenced the shooter's shooting performance, for better or worse.

[0030] The system may be configured to associate specific poses, movements, and combinations with shooting performance through machine learning. In some cases, body landmark movements and movement combinations may be associated with improved shooting performance, while others may be associated with decreased shooting performance.

[0031] The system may normalize the motion data to generate normalized coordinates for each body part's position during all sessions. The score and its running average may be represented by a signal, for example, by displaying it on a user interface. The motion data and / or score may be analyzed in near real time or stored in a data file that can be saved for later analysis.

[0032] Motion data may be associated with shot pattern data, such as the x and y coordinates of each shot, and the shot coordinates may be temporally associated with the motion data occurring at the time the shot was fired. Additionally, a score may be assigned to the shot and stored with the shot data.

[0033] One or more machine learning techniques may be applied to the motion data and shot data to generate correlations between motion and shot accuracy. For example, in some embodiments, a convolutional deep neural network (CNN) may be designed to locate features in the motion and shot data collection. Other deep learning models for classification purposes may also be used to correlate motion and shot data to identify patterns that lead to improved or reduced accuracy. Transformations that correlate any set of attributes with any other set of attributes may also be used.

[0034] FIG. 8A shows a sample camera angle of a shooter 800, and FIG. 8B shows the resulting wireframe model 802 of the shooter, which allows the system to track the shooter's body movements, including selected body landmarks. As can be seen, the system can determine the shooter's pose, stance, and movements throughout the shooting string. The wireframe model may include representations of each major joint and body artifact, which may include, among other things, one or more of the shooter's nose or chin 804, right shoulder 806, right elbow 808, right wrist 810, hip 812, left femur 814, left knee 816, left lower leg 818, left ankle 820, right femur 822, right knee 824, right lower leg 826, and right ankle 828. In some cases, the system may be trained with motion data from various shooters and their past performances correlated with the motion data. In this case, the system can determine the impact of specific poses and / or movements on shooting performance. In some cases, the system may rely on performance data from a single shooter to assist that shooter in making adjustments and / or conducting further training exercises to improve the shooter's performance.

[0035] 9 shows x-axis motion data 900 associated with a shooter's head 902 and nose 904. As can be seen, the shooter's head moves backward after each shot 906a-906d during the firing string, which may be correlated with shooting performance. That is, the system may determine that the shooter is moving their head after each shot, for example to look at the target, and that they may have drifted during the firing string and not returned to the exact same spot, causing the shooter's performance to fall below a threshold.

[0036] In some cases, the system may be configured with logic to determine probable cause and effect based on either the shooter's movements and / or the target's scoring. Referring to Figures 10A and 10B, diagnostic targets 1000 for left-handed and right-handed shooters, respectively, are shown. Because the right-handed and left-handed targets are mirror images of each other, only one target will be described below.

[0037] Assuming the shooter is able to hold the pistol at the target, movements associated with the shooter may be the cause of off-center hits. If the off-center hits are regularly clustered to one side of the target, there are several possible issues that could be causing the off-center hits. For example, if a group of shots lands at the 12:00 position, the system may determine that the shooter is bending their wrist upward 1002 (e.g., riding the recoil), which often occurs in anticipation of the recoil.

[0038] If the group of shots lands at 1:30 position 1004, this may be indicative of a grip error known as heeling, where the heel of the hand pushes the pistol butt to the left in anticipation of the shot, which forces the muzzle to the right.

[0039] If a group of shots lands at the 3:00 position 1006, this may be an indication that the thumb of the shooting hand is applying too much pressure, pushing the side of the pistol to the right and forcing the muzzle to the right.

[0040] If the group of shots lands at the 4:30 position 1008, this may be an indication that the shooter is squeezing the grip too tightly while firing (e.g., lobstering). Pulling the trigger will lower the front sight, which will cause the shot to go low and to the right for a right-handed shooter.

[0041] If a group of shots lands at the 6:00 position, 1010, this may be an indication of a downward bending and forward thrust of the wrist, often an unconscious effort to control recoil and prevent muzzle lift.

[0042] If a group of shots lands at the 7:00 position 1012, this may indicate a trigger jerk or slap, which may indicate the shooter is about to pull the trigger just as the sights are on target.

[0043] If a group of shots lands at the 8:00 position 1014, this may be an indication that the shooter is squeezing the grip too tightly during the shot.

[0044] If a group of shots lands at the 9:00 position 1016, this may be an indication that the finger is too light on the trigger, which typically causes the shooter to pull the trigger at an angle when finally pulling it rearward, tending to push the gun to the left.

[0045] If a group of shots lands at the 10:30 position 1018, this may be an indication that the shooter is pushing in anticipation of the recoil and not following through well.

[0046] The system can be programmed to verify target hits during the firing string and, in combination with body landmark movements, determine whether the shooter is engaging in recoil anticipation, trigger control errors, and / or grip errors. The system can make this determination for each individual shooter and recommend practice exercises and practice strings to address their specific shooting issues.

[0047] In some cases, the system may have access to Data On Previous Engagement (DOPE) associated with the shooter, which may include previous shooting sessions, records, scores, and analysis.

[0048] Complete shooting sessions can be recorded and stored, along with real-time scoring, so the user can review them along with decisions from the machine learning system that identify and / or highlight flaws, mistakes, and changes in the shooter's posture or technique that improve or detract from the shooter's performance.

[0049] As described herein, synergies arise from synchronizing all available information sources.

[0050] Continuing with reference to Figures 10A and 10B, where the diagnostic target is shown, the basic traditional ML model maps a target shot pattern φ ∈ Φ to a shooter behavior β ∈ B, which is given by f T It is denoted as Φ→B. The behavior can be a four-dimensional (4D) phenomenon that includes three-dimensional (3D) motion data plus time. The shot pattern Φ is a two-dimensional (2D) projection of the evolving 3D phenomenon (2D plus time). The model can be based on causal hypotheses.

[0051] Experts have a much larger set of complex shooter behaviors. * ⊃B and divide it into a larger set of shot patterns Φ * which can be mapped to f XP :B * →Φ. Behavior β∈B * can be a 4D phenomenon, and the shot pattern φ∈Φ * is a 2D phenomenon. It can be assumed that this is in principle a causal model, since experts adapt the model according to their understanding of the shooter's physiological kinematics and psychology, as well as the physics of shooting.

[0052] A relatively simple iterative model for expert learning may involve predicting shot patterns from observed shooter behavior, which is XP :B * →Φ. The model can then estimate the difference between the predicted shot pattern and the actual shot pattern, which is denoted as δ(φ′,φ). The model can then generate a causal prediction model f XP This may be repeated every n shot drill.

[0053] This type of learning model is based on the shooter behavior β∈B * is treated as a 3D phenomenon, and the shot pattern φ∈Φ * is treated as a 2D phenomenon. In some cases, B * as a 4D phenomenon, or Φ * As a 3D phenomenon, the model is enhanced to treat B as a 3D phenomenon, or both. ** 3D projection (2D plus time) of 4D shooter behavior B * Represents.

[0054] The model is a decision function f G By running Φ → D (where D is a binary variable), the model can objectively distinguish good shot patterns (e.g., clustered around the bullseye) from the remaining "bad" shot patterns. The model also predicts the shooter's behavior E * can be constructed to relate to how (and why) the shot pattern φ is "bad".

[0055] In some embodiments, the sampled time signal for the shooter behavior can be reduced for a single drill as a single line in the dataset. A bounding box can be derived around the set of XY coordinates of each of the body landmarks in the drill. The bounding box can be represented as shooter behavior B. Thus, there can be a shooter behavior for each of the body landmarks during a period of time. In some cases, shooter behavior B and shot pattern Φ are represented in polar coordinates as a radius and an angle. Shooter behavior B (which in some examples is 17) is 2D with (x,y) coordinates or (r,θ) representation. Shot pattern φ∈Φ is a causal consequence of shooter behavior β∈B, and the method can treat shot pattern Φ as a proxy for shooter behavior β in the unobservable third dimension. The method treats shooter behavior β∈B and shot pattern φ∈Φ as features and treats score s∈S as a target for the ML model, which is expressed as f ML :BxΦ→S.

[0056] The model considers patterns (β,φ,s)∈Ψ that result in low and high scores. L ,Ψ H For this purpose, implicit or explicit clustering techniques may be used. In some cases, the model f ML :BxΦ→S can be constructed for multiple drills for a particular shooter and drills for multiple shooters. In some cases, a particular drill (β,φ,s) and a model f ML Given a :BxΦ→S, model quality can be assessed by comparing the actual score s∈S with the predicted score s'∈S. From these comparisons, some models may be updated sequentially.

[0057] In some embodiments, a low-scoring drill (β, φ, s) can be evaluated against Ψ to determine which of the 17 components of the observable shooter behavior β and the shot pattern φ, which serve as a proxy for unobservable shooter behavior, are likely responsible for the low score. This may be easiest for an explainable model. Unobservable behavior may be explained by the usual labels in the diagnostic target, as described above. In some cases, the shooter diagnostic application may also distinguish low scores due to aim misalignment, perhaps easiest understood as shots clustered around a center of gravity other than the target bull's-eye. In some cases, the system may determine that a reduced score was obtained and generate recommendations to improve the score. As used herein, the term "reduced score" is used to mean a score that is less than a target score. The target score may be a perfect score, a participant's best previous score, a participant's average score, or some other metric. For example, in shooting competitions, the highest score for a given shot is often "10." In this case, a reduced score is a score below "10." Similarly, in other sports, a score that is less than a desired score may be assigned. For example, a penalty kick in soccer could have a binary outcome of a missed goal resulting in a reduced score compared to a goal made. In terms of golf shots, a golfer may average a driver that hits the ball 275 yards. A reduced score could be due to a driver shot that travels 250 yards, which is less than the golfer's average driver distance, and the system could observe the behavior and determine which behavior(s) resulted in the reduced score.

[0058] According to some embodiments, the system relies on artificial intelligence (AI) and / or machine learning (ML) for two decisions: detecting shooter movement and shot landing locations, and analyzing shooter behavior for how shooter behavior leads to shot landing locations. While detecting movement has been discussed, there are multiple ways to analyze how shooter behavior leads to shot landing locations. First, an expert-based approach utilizes comparing individual shooter behavior for each drill with expert assessments of what is believed to result in a good shot. Second, a data-based approach is implemented by building a model from repetitions of shooting behaviors ("drills") by a single shooter or multiple shooters. This can be considered an AI / ML-based discovery strategy that determines which behaviors correlate with good shots.

[0059] In some cases, AI / ML may be used to automate and augment expert-based analysis in the sense that if the "ideal" behavior of a prototype is known a priori, models of these a priori known behaviors can be fitted to data from a single shooter or multiple shooters.

[0060] From that perspective, Transformer ML models can mimic automated and augmented expert-based analysis by combining pre-trained database models for inferred relationships between language fragments (similar to inferring relationships between shooter behavior and shot landing locations) with an additional stage of database adaptation.

[0061] Referring to FIG. 11 , the system determines key points related to the shooter's hand 1100. The key points may correspond to each mobile joint in the wrist, hand, and fingers, and their respective locations and positions relative to one another. The system may connect the key points to a wireframe model 1102 of the shooter's hand, allowing for accurate monitoring of pose and movement. FIG. 11 illustrates multiple key points associated with the shooter's hand that can be used to determine the grip the shooter is using. For example, by referencing the key points on the shooter's hand, the system may determine whether the shooter is using a thumbs-forward, thumbs-over, cup-and-saucer, wrist grip, trigger-guard gamer, or another style of grip. Different grips and pressure applied by the hand can impart movement to the pistol, and the system may determine that a different or modified grip would result in better performance.

[0062] The system receives signals associated with the shooter's position and movement, processes the signals, and finds errors in the shooter's stance and grip. Additionally, by combining single-frame analysis (e.g., by finding insights from the relative positions of different body parts at a particular moment in time, which is analysis over time), it is possible to identify errors due to changes in the shooter's position.

[0063] Additionally, the system can focus on the position of the hand on the pistol. A hand tracking machine learning model can be used to track the position of each bone in each finger within the video frames. The system receives signals associated with the position of each finger over time.

[0064] 12A and 12B , an application program is shown that may execute on a mobile computing device and receive image data from one or more imaging sensors associated with the mobile computing device. As with any of the embodiments described herein, methods and processes may be programmed into an application program or set of instructions that may be executed on a computing device. In some cases, the application program may execute on the mobile computing device, and audio / video capture, analysis, scoring, recommendations, training exercises, and other feedback may be performed using the mobile computing device. In some cases, the system receives video and / or audio from multiple video capture devices that capture different views of the same shooter. The system may use the multiple different views of the shooter for analysis and feedback to the shooter to improve performance.

[0065] As shown in FIG. 12A , a system running as an application on a mobile computing device 1200 captures video frames of a shooter 1202, establishes body landmarks for tracking over time, creates a wireframe model 1203 of the shooter, tracks fired shots 1204, and provides feedback to the shooter 1202. As shown, the system may identify the shooter's stance—in the illustrated example, the “Weaver” stance—track the number of shots 1204, and provide feedback 1206 regarding each shot. For example, a shot fired at 7 seconds after the start of the string indicates a center-of-mass hit. The system identifies a reload motion 1208 at 8 seconds, lasting 1.2 seconds, followed by a shot at 11 seconds, indicating the shooter anticipated recoil and the shot was off-center. In some cases, this functionality of the system may be viewed as a drill instructor, tracking shots, providing feedback, and making suggestions for improving posture, grip, stance, trigger pull, etc., to improve the shooter's performance.

[0066] 12B shows an additional screen 1210 that may be displayed by the system in a Spotter modality in which a mobile computing device may aim its video capture device at a target. In some cases, the mobile computing device may use an internal or external lens to get a better view of the target; the mobile computing device may be coupled to a spotting scope or other type of optical or electronic telephoto zoom lens to get a better target image.

[0067] The system can be toggled between different modes, such as Spotter, Drill Instructor, DOPE, Locker (which can store information about different firearms owned by the shooter), and program settings. Spotter mode may show an additional screen 1210 with a view of the target where target hits can be viewed. The system may also display other information, such as the firearm 1212a and ammunition 1212b combination used in the shot, distance to the target 1214, target type 1216, firing string time 1218, number of shots fired, number of target hits 1220, and score 1222, among others. An image of the target 1224 may also be shown, and hits 1226 may be further highlighted on the image of the target 1224. It should be understood that Spotter mode may show information in real time, including elapsed time, hits, score, and number of shots. Additionally, the system may store data associated with the firing string for later playback, review, and analysis. For example, the Drill Instructor mode may review the DOPE associated with the shooter and the firearm and ammunition combination, provide a detailed analysis of the shooter's misses with a particular firearm, and provide exercises or suggestions for improving the misses to improve the shooter's performance.

[0068] 13A, 13B, and 13C, an exemplary architecture for data capture 1300 is shown. According to some embodiments, a shooting range 1301 may be equipped with one or more sensors, such as image sensors, projectors, and computing devices. FIG. 13A shows the shooting range 1301 looking downrange from the perspective of a shooter 1302. FIG. 13B shows the shooting range 1301 from a side view, and FIG. 13C shows the shooting range 1301 from a top view.

[0069] One or more cameras 1304 may be mounted within the shooting range to capture video of the shooter from multiple angles. The cameras 1304 may be any suitable type of video capture device, including, but not limited to, a CCD camera, a thermal camera, a dual thermal camera 1305 (e.g., a capture device having both thermal and optical capture capabilities), among others. The cameras 1304 may be mounted in any suitable location within the range, such as to the sides of the shooters 1304a, 1304b, facing the front of the shooter 1304c, or overhead 1304d, among others.

[0070] In some cases, the camera 1304 may be mounted on a gantry system that provides a portable structure to support one or more cameras 1304 to capture multiple angles of the shooter.

[0071] In some embodiments, a projector 1306 may be provided to project an image onto a screen 1308. In some cases, the projector may project an arbitrary target image onto the screen or shooting wall, allowing the shooter to practice dry shots (e.g., without firing live ammunition), and the system may register hits on the projected image. For example, the image may show a target on the screen, and the shooter may dry-fire at the target. The system may register the aim point at the time the trigger is pulled and display the hit on the projected target image. Compatible devices exist that fire a laser through the barrel of a firearm when the trigger is pulled to indicate where the shot would have landed on the target. The thermal imaging camera 1304 may detect the laser hit location, and the system may be configured to display the hit on the projected target, which may then register and score the hit. In this way, shooters can practice using a shooting simulator with their own firearms without having to travel to a dedicated shooting range. The system may encourage dry-fire practice to improve poor shooting habits.

[0072] 14 illustrates a system configured for multi-source synchronization. In some embodiments, a shooter 1402 is captured from different angles by multiple video capture devices 1404. The system obtains features such as stance, pose, and movement from the shooter 1406. The system may analyze the captured audio signal 1408 to complement the video shot detection. That is, the system may analyze both the video and audio and detect shots fired by the shooter, for example, by correlating spikes in the audio waveform with a sudden muzzle lift of the firearm.

[0073] The system may be configured to detect shooting stages 1410, such as first firing string, reload, second firing string, etc., including detection of fired shots. The system may additionally identify and determine shooting errors 1412, identify errors, and drills and exercises to address shooting errors. Error detection may be iterative and further error corrections 1414 may be suggested. The system may also incorporate shot detection 1416, as described herein, including correlation of shot detection with audio data.

[0074] The system may further capture images 1418, such as video images of the target, and register hits on the target, which may be correlated with motion data before and during the trigger pull. As described herein, the system may determine shot location 1420 and ultimately determine a score 1422 of one or more of the shots fired. Thus, some embodiments of the system provide an intelligent tutroing / adaptive training solution that transforms a time-intensive, non-scalable live training environment into an automated, adaptive, virtual scenario-based training solution.

[0075] According to some embodiments, the model is based on instructions executed by one or more processors that cause the processors to perform various operations. For example, the method may include collecting (as many as possible) pause analysis files, which may be stored as comma-separated value (CSV) files.

[0076] FIG. 15 shows a shot bounding box 1500 determined by the system. The “drillparser.py” script can be configured to combine the pose analysis files. The system can then find the shot in the CSV file for each of the 17 pose landmarks, for example, and derive a bounding box 1500 oriented around the x- and y-coordinates in a 1.0-second time window up to and including the shot. The system then uses the angle φ 1502, dimensions l 1504, and w 1506 of the bounding box 1500 as elements in a decision tree model for scoring. The “allshots” approach processes data on a CSV file basis, so bounding boxes 1500 are drawn around all shots as a single row in the resulting dataset. The “oneshot” and “oneshotd” approaches can treat each shot individually, so the shot bounding box 1500 can be an infinitesimal box around a single shot; thus, each row in the dataset is one shot. The "oneshotd" method additionally orients the bounding box around the 17 pose landmarks in time order, while the "oneshot" method ignores time.

[0077] Thus, the system can derive an oriented bounding box around each shot, which can be oriented in the direction of the box around the body landmarks.

[0078] 3) These datasets are processed in BigML as follows:

[0079] 1. Upload to BigML as a Source object

[0080] 2. Convert Source objects to Dataset objects

[0081] 3. Split the Dataset object into Training (80%) and Test (20%) datasets.

[0082] 4. Build a tree model using elements selected from the training dataset

[0083] 5. Create BatchPredictions using the appropriate Model and Test dataset

[0084] 6. Download the Model as a JSON PML object for the next step

[0085] 4) The "modelannotator.py" script annotates the model PML file as needed for explain behavior. In some cases, this annotation consists simply of adding a "target" attribute to each node that is a list of all target (score) values ​​reachable in the tree from that node.

[0086] 5) The “itemexplainer.py” script uses the (hypothetical) pruned annotated model to “explain” which pose elements for a new drill result in an unsatisfactory score prediction.

[0087] 6) Of course, the model can be tuned by repeating model training via machine learning using instances where the predicted scores differ significantly from the actual scores.

[0088] The elements derived from the shot sample may include one or more of the following elements: shot_phi with bounding box rotation (-π≦φ≦π), shot_l—bounding box length, shot_w—bounding box width, shot_lw—bounding box area, shot_theta—bounding box center rotation (-π≦φ≦π), shot_r—bounding box center radius. A bounding box 1500 is drawn around the shot sample.

[0089] A very similar approach can be used to derive elements from body landmark samples: since body landmarks move over time, each body landmark can similarly be used to create a bounding box with length, width, area, and rotation that can be correlated to a shot element.

[0090] In some cases, the system is configured to use one or more cameras for machine vision of the shooter, determine one or more body landmarks (possibly up to 17 or more), and track the movement of each of these landmarks over time during shooting drills. The system can correlate the timed body landmark movement with scored shots to determine whether a shot is good or bad. A good shot may be relative to the DOPE for the shooter, firearm, and ammunition combination. For example, if a single shot is closer to the aim point than the shooter's average shot, it may be classified as a good shot. Furthermore, by examining several shots over time, the system can correlate shooter behavior with good or bad shots. Furthermore, by analyzing shooter behavior (e.g., body landmark movement), the system can predict whether a shot is good or bad without seeing the target outcome. For example, a good shot is considered a shot within the 9 or 10 ring on the target, while a bad shot is one outside the 9 ring. The definition of a good or bad shot may vary depending on the shooter's expertise. As an example, for a highly skilled shooter, anything outside the 10 ring may be considered a bad shot.

[0091] Finally, by correlating shooter behavior with target accuracy, the system can provide the shooter with an output regarding the behaviors that cause a decrease in accuracy. Additionally, the system can recommend one or more specific drills to the shooter to address the behaviors that cause a decrease in accuracy.

[0092] 16 illustrates some embodiments of a decision tree model 1600 for correlating body landmark factors with shot samples. A decision tree algorithm is a machine learning algorithm that uses decision trees to make predictions. It follows a tree-like model of decisions and their possible outcomes. In some cases, the algorithm works by recursively dividing the data into subsets based on the most important features at each node of the tree.

[0093] From the body landmark data, various features are extracted to represent the kinematic characteristics of body movement. These features include body landmark positions over time, bounding boxes describing the movement of body landmarks during times including before and during the shot, relative distances between landmarks, and time derivatives capturing the dynamic aspects of the movement. Feature scaling is performed to facilitate model convergence.

[0094] In some embodiments, the decision tree shown in Figure 16 serves as the core of the model. It is selected for its ability to unravel complex nonlinear relationships in the data while maintaining interpretability. The depth of the tree is optimized to minimize overfitting through techniques such as cross-validation or a predetermined maximum depth. The choice of separation criterion, whether based on Gini impurity or information gain, depends on the specific problem. Pruning methods, such as enforcing a minimum number of samples per leaf, are applied to control excessive branching.

[0095] Training the model may involve the recursive construction of a decision tree. The tree has nodes that correlate with body landmarks. For example, the left_ankle_phi node 1602 may branch to a right_wrist_lw node 1604 and a right_knee_phi node 1606, which may indicate how the rotation of the bounding box of the right ankle motion data, combined with the rotation of the bounding box area of ​​the right wrist or the bounding box associated with the right knee, affects the resulting shot landing location. Similarly, the decision tree may correlate shot landing locations with feature combinations. At each node, the training data is partitioned based on feature values ​​to optimize a selected loss function, such as mean squared error or cross-entropy. Hyperparameters can be fine-tuned via cross-validation, and various metrics are used to evaluate the model's performance.

[0096] During real-time operation, the decision tree model is continuously updated as new body landmark data becomes available for each shot fired. For each input feature vector derived from live landmark data, the model traverses the decision tree and arrives at a leaf node. The label assigned to the leaf node (indicating a positive or negative outcome) is utilized as the model's prediction. In the illustrated example, a right shoulder bounding box length between values ​​of 0.80 and 0.82 matches the predicted outcome of the shooter's performance, as shown in leaf nodes 1608 and 1610.

[0097] In the context of this model, a positive outcome indicates successful execution of a particular action or achievement of a desired shot landing location, while a negative outcome indicates an inaccurate action, deviation from the desired position, or an off-center shot landing location.

[0098] Model performance is rigorously evaluated through metrics including, but not limited to, accuracy, precision, recall, F1 score, and area under the receiver operating characteristic (ROC) curve. A confusion matrix is ​​employed to quantify the model's proficiency in classifying positive and negative outcomes.

[0099] FIG. 17 shows an annotated model 1700 showing various values ​​at several nodes. For example, during the firing string, the area of ​​the bounding box for the right wrist 1702 shows values ​​of 0.47, 0.51, 0.57, 0.59, and 0.61. This indicates that the shooter moved his right wrist within the area defined by the displayed values ​​during the firing string when shots were being fired. Following the branch below the right wrist 1702, for values ​​below the average of the values, the system can identify that the left knee 1704 moved in a specific pattern and correlate this with either good or bad shots to explore causal relationships in the combination of right wrist movement and left knee movement. Other features can be similarly annotated in the model. The annotated features can be stored in a feature vector and correlated with each shot fired for analysis of how different features combine to result in a particular shot performance.

[0100] FIG. 18 shows a pruned model 1800 that allows for a deeper dive into various features and their interrelationships with one another. For example, reviewing the right_knee_phi 1802 feature (e.g., the bounding box rotation associated with the movement of the right knee during the shooting string) can reveal values ​​of 0.8, 0.82, and 0.9. Given these values, it can be seen that a right_knee_phi value of 0.9 will yield one result at the first result node 1806. It can be seen that a right_knee_phi 1802 of 0.8 or 0.82 and a right_shoulder_l bounding box length 1804 of 0.09 will yield a second result at the second result node 1808 and the third result node 1810. In some cases, the second result node 1808 will be associated with poor shooting performance, and the third node 1810 will be associated with good shooting performance. These nonlinear causal relationships can be determined by machine learning models, allowing the system to determine which movements result in better shooting outcomes and which movements result in worse shooting outcomes by running one or more machine learning algorithms.

[0101] In some cases, an isolation forest encodes a dataset of trees such that leaf instances per leaf are identical. Node splits may be chosen randomly rather than to reduce impurity of child target values. In some cases, a node is a leaf when the training instances at the node have the same target value. In some examples, rare instances in the training dataset reach the leaf sooner than less rare instances. According to some embodiments, new anomalous instances have a shorter path relative to the tree depth for all trees in the forest, making anomaly identification more efficient.

[0102] In some cases, Classification and Regression Trees (CART) are predictive models that explain how a result variable is predicted based on other values. A CART-style decision tree is one where each fork is a split of a predictor variable and each end node contains a prediction of the result variable. This is a recursive algorithm that constructs a binary tree to partition the feature space into segments that are homogeneous with respect to the target variable, performing binary splits on the input features based on certain criteria to create a tree-like structure. A CART-style decision tree can be useful for predicting the outcome of a shot based on the motion data of one or more body landmarks. A CART-style decision tree is a multi-category classifier (e.g., N>1), while an isolation tree is a single-category classifier (e.g., "is it an anomaly or not"). In some cases, a CART-style decision tree can be made into an isolation tree by pruning the branches that reach leaves with N-M "anomalies" up to the root node, where M<N categories are designated as "not an anomaly" and N-M are designated as "anomaly". This can be similar to training an isolation tree on M<N "not an anomaly" instances, which allows for impurity-reducing splits and impure "not an anomaly" leaves.

[0103] FIG. 19A shows a gantry 1900 system that can provide a portable mounting structure for housing one or more imaging devices, including one or more video cameras. In some cases, the structure includes one or more support columns 1902 and one or more crossbars 1904. The gantry structure 1900 can be positioned around the shooter or, in some cases, downrange from the shooter. For example, the gantry 1900 can have the crossbar 1904 positioned approximately 1 foot (≈3 m) to 8 feet (≈2.4 m) above the shooter and between 1 foot (≈3 m) and 15 feet (≈4.5 m) in front of the shooter. In some examples, a camera is positioned on each support column and on the crossbar. Thus, in some embodiments, two, three, or more cameras can be positioned on the gantry, with some of the cameras pointed at the shooter and one or more additional cameras pointed downrange toward the target.

[0104] 19B, 19C, and 19D show various views captured by cameras mounted on the gantry 1900. A first camera 1908 may be mounted on the crossbar 1904 or support column 1902 and is positioned to capture a left side view of the shooter (FIG. 19B). The camera may be configured to capture the shooter's entire body or the shooter's upper body and head.

[0105] A second camera 1910 may be mounted on the crossbar 1904 or support 1902 and configured to capture a right side view of the shooter (FIG. 19C). The camera may be configured to capture the shooter's entire body or the shooter's upper body and head. In some cases, the first camera 1908 and the second camera 1910 utilize different fields of view such that one camera captures the shooter's entire body and the other camera captures only a portion of the shooter's body.

[0106] A third camera 1912 may be positioned on the crossbar 1904 and configured to capture an overhead view 1914 (FIG. 19D) of the shooter. By positioning the cameras to capture the shooter's movements from various angles, the system can correlate the video data from each camera and determine three-dimensional motion data of selected body landmarks.

[0107] A fourth camera may be positioned to capture video data of the target 1916 (FIG. 19E) and the impact of the shots on the target 1918. The video data from each of the cameras may be synchronized and analyzed to determine when shots were fired and to correlate the fired shots with registered hits or misses on the target.

[0108] FIG. 20 illustrates a computer program user interface 2000 usable with the systems and methods described herein. For example, the computer program may be configured to receive video data from one or more cameras, synchronize the video data, determine when shots were fired, and register and score hits or misses on the target. The user interface 2000 may include a start record button 2002 that enables the shooter to begin video capture. In some cases, the shooting string may be time, and the start record button may further start a timer. The user interface may further include a timer 2004 associated with the session. The user interface may be presented on a mobile computing device associated with the user or on a mobile computing device associated with the facility. For example, a gantry system may be installed at the facility, and a computing device associated with the facility may be connected to the gantry system, configured to receive the video data, and provide performance feedback for the embodiments described herein.

[0109] The user interface may be provided on any suitable display, such as a television, touchscreen display, tablet screen, smartphone screen, or any other visual computer interface. With further reference to FIG. 21 , the user interface 2000 may display indicia associated with the shooting string, such as video of the shooting string and movements during the shooting string 2102. The video may be displayed in a playback window, providing controls 2104 for playing, pausing, adjusting the volume, and scrubbing the video. Additionally, there may be controls for selecting different views 2106, i.e., controls that allow a viewer to select video clips captured by different cameras during the shooting string to view different views of the shooting string, either individually or in an integrated view, such as a side-by-side view. These views may be synchronized to allow viewers to view different views of the same event simultaneously.

[0110] The user interface 2000 may further show a target 2110 and may identify hits 2112 that the system has registered on the target. The user interface 2000 may further display the score of the most recent shot 2114, along with the average score for the string 2116. Of course, other information may be displayed as desired by the user, including training tips, motion data that contribute to off-center shots, or other mistakes during the shooting string.

[0111] 22 shows an additional view of a user interface 2000 that allows a user to specify a body landmark selection 2202. In addition, the user interface 2000 provides a selection for signal settings 2204 that allows the user to specify details of Y coordinate movement or X coordinate movement. In response to a user selection, the user interface 2000 may display motion data 2206 associated with that selection. This type of review and analysis allows the shooter to very specifically see the movement of individual body landmarks during the firing string, and additionally, can specifically see horizontal movement, vertical movement, or both for review.

[0112] FIG. 23 illustrates some of the features and techniques employed by embodiments of the described systems and methods. In many embodiments, the disclosed systems utilize machine vision (e.g., computer vision) 2302 to track body landmarks of a participant (e.g., the shooter) and also track target changes to provide automatic target scoring 2304. The systems and methods may also utilize audio processing 2306 to enable shot time detection 2308, which may also be combined with computer vision techniques. The disclosed systems and methods may also utilize signal processing 2310 to provide shooter position correction 2312, including pose, posture, movement, grip, trigger pull, etc. The systems and methods described herein also apply machine learning 2314 to determine the causality of off-center shots, which may include analysis of the shooter's pose, grip 2316, trigger pull, stance, and body landmark movement, among others.

[0113] 24 shows a method 2400 according to an embodiment described herein. The system may receive video data and optionally audio data. The system may be configured to detect 2402 and score shots through image processing on the video data of the target. The system may process the video data to determine fired shots and a shooter pose analysis 2404. Additionally, the system may analyze audio data for shot detection 2406.

[0114] Shot detection 2402 provides 2408 coordinates (e.g., x,y coordinates) of the shot within the target. Shooter pose analysis 2404 provides 2410 coordinates (e.g., x,y coordinates and optionally z coordinates) of body landmarks during the shooting process. Audio shot detection 2406 provides 2412 the precise time of the shot. Shooting analysis may include one or more machine learning algorithms that receive the shot and body data and, through the machine learning algorithms, detect and predict the causality of the shooter's shot performance with the firearm and ammunition combination, which may be referred to as performing shooting analysis 2414. For example, embodiments of the system may generate one or more of a session score 2416, shooting position recommendations 2418, shot misses, and grip analysis 2420, among others.

[0115] 25 illustrates a flow of a sample process 2500 for using machine vision and machine learning to track participants' body landmarks and generate recommendations for improvement. As noted herein, the systems and methods described herein can be used in any competition, such as a sporting event, that benefits from repeatability and accurate body kinematics. In addition to shooting, some such competitions include archery, golf, bowling, darts, running, swimming, pole vaulting, football, baseball, basketball, hockey, and many other types of sports. In any case, the system is configured to track participants' body landmarks and determine how to change their body movements to improve performance.

[0116] In block 2502, the system receives video data of the participants, which may be obtained from a single image capture device, or two image capture devices, or three or more image capture devices. The image capture devices may be any suitable imaging devices configured to capture continuous images of the participants, and may include any consumer or professional video camera, including cameras typically built into mobile computing devices.

[0117] In block 2504, the system determines one or more body landmarks for the participant. The body landmarks may be associated with any joint, body part, limb, or location associated with the joint, limb, or body part. In some cases, the system generates the wireframe based on one or more body landmarks and may not use all body landmarks in generating the wireframe model.

[0118] Body landmarks are tracked during performance of the activity to generate motion data at block 2506. As a non-limiting example, body landmarks may be created for a golfer's hands, wrists, arms, head, shoulders, torso, waist, knees, ankles, and feet that may be tracked during a golf club swing.

[0119] At block 2508, a score is determined and associated with the performance. As described, in activities involving a projectile, the score may be associated with the trajectory or destination of the projectile. In the case of golf, for example, the score may be based on distance, direction, or proximity to the target, or a metric associated with the participant's average or past performance. In short, any metric may be used to evaluate the quality of the performance results.

[0120] At block 2510, the score is associated with the performance, i.e., the performance may be linked to the determined score and stored for later analysis to determine trends in performance over time or to compare one performer to another.

[0121] In block 2512, the system generates recommendations for altering movements in subsequent performances to improve results. In some cases, the recommendations may involve the hands, including grips on a firearm, golf club, bat, stick, etc. The recommendations may also include changes or shifts in weight distribution. The recommendations may include movements of the hands, head, shoulders, body, legs, feet, or other body parts. Often, the recommendations include suggestions for altering the movement of one or more body landmarks to improve the performance score in subsequent attempts.

[0122] Acoustic Small Arms Signature

[0123] According to some embodiments, a system is described that can quickly calculate the acoustic sound signature of a shooter's firearm firing a specific ammunition in a given environment using consumer audio recording equipment (e.g., iPhone®, tablet, phone, video camera). In some cases, the system can receive an input audio or audio / visual file and calculate the firearm signature based on the sounds in the captured recording. The recording can be captured by any suitable audio and / or video capture device, such as, but not limited to, a security camera, a traffic camera, a video camera, a television camera, a mobile device recorder such as a smartphone or tablet, and other capture devices. In some cases, the capture device is a readily available consumer recording device. The acoustic signature can further include determining the make, model, silencer, ammunition type, ammunition manufacturer, and other characteristics of the firearm based on the acoustic signature from the fired firearm.

[0124] This capability can be used to detect and separate a shooter's shots from those of other shooters in a typical shooting range, which may be useful for automated scoring, etc. A version of this capability may be used to identify specific firearms and ammunition from recordings.

[0125] In some cases, the described solutions operate in near real time on a single consumer recording device, such as a mobile phone, using only a modest amount of training data. As used herein, the terms “real time” or “near real time” are broad terms and, in the context of this disclosure, relate to receiving input data, processing the input data, and outputting the results of data analysis with little or no human-perceived latency. In other words, a system such as that described herein that outputs analyzed data within less than one second is considered near real time. A system operating in real time or near real time may limit the amount of computation for machine learning, or at least training models, that the method can use to characterize a particular firearm that fires the appropriate ammunition. Additionally, there are methods that do not use ML techniques, meaning that they do not adapt (“train”) a model for a large number of samples. Some of these other methods may perform well but may also be more computationally intensive or shift the computational load to shot identification time, for example, by searching a large dictionary of signatures for “matches.”

[0126] Previous approaches to using artificial intelligence (AI) to analyze gunfire presented a simple approach of using deep neural networks (DNNs) to classify gunfire sounds by firearm and ammunition type. In these approaches, the DNNs had to be pre-trained with numerous gunfire sounds for each firearm and appropriate ammunition type. These sounds may not have been captured on the same recording device or under the same firing conditions. This potentially improves the generalization ability of the trained DNN instance to classify firearm and ammunition types regardless of the environment and recording device. However, it reduces the ability to identify the gunfire sounds of a specific firearm and ammunition type in any environment using any recording device.

[0127] Previous techniques have described a two-stage approach to classification. For example, some previous techniques use a pre-trained instance of a relatively general pre-trained DNN instance as an approximate classifier (predictor) and may use an additional trainable step on the classifier output to improve the DNN prediction. In some cases, the DNN treats finite-length time segments of the time-varying spectrogram of gunfire sounds as images and distinguishes pooled images of each firearm and ammunition pair from pooled images of other firearm and ammunition pairs. This approach has several drawbacks, including the use of a general DNN instance that is not particularly adept at classifying sounds from different environments or different ammunition.

[0128] According to some embodiments, the described system superimposes a trainable weighting mask onto an image created from a time segment of a time-varying spectrogram and provides a simple training method for adjusting the mask using a small number of instances of a shooter's small arms and ammunition fire recorded in a particular environment with a particular device. Training only the weighting mask is much less computationally intensive than training the DNN itself. Thus, according to some embodiments, the DNN may not be trained or may be trained to a much lesser extent than traditional methods, while the weighting mask is trained. This approach has several advantages.

[0129] For example, abstractly, the adjusted mask can be viewed as a signature for the combination of firearm, ammunition, environment, and recording device. This signature is not used to search a catalog of that combination's signatures of elements, but rather is used to adjust the spectrogram inputs to improve the DNN's classification performance for firearms and ammunition in a particular environment using a given recording device. The enhanced DNN classification can then be used to improve detection of the shooter's shots and distinction from those of other shooters.

[0130] This approach offers significant advantages in the computational intensity required, allows for much faster analysis, and can be used for a large number of firearms in many different environments. For example, the training data set for a particular shooter is small, so the weighting mask can be updated online using stochastic gradient descent or offline using gradient descent. In other words, the training data can be for a specific shooter using a specific firearm and ammunition combination in a given environment, resulting in a much smaller data set than if data points were aggregated from a large number of shooters in different environments.

[0131] Additionally, both algorithms may estimate the gradient of the function computed by the DNN at each iteration of the weight matrix update. While this is not computationally prohibitive, it may turn out that just the sign of the gradient or a coarsely quantized version of it is sufficient. The simplification mainly depends on whether the DNN computes a monotonically increasing or monotonically decreasing function at each input. There is a large literature investigating the properties of functions computed by DNNs. However, finding a simple answer to this question is not easy. In some cases, a DNN does this when all of its internal layers use linear or affine activation functions and the output layer uses a monotonically non-decreasing activation function.

[0132] Thus, in some embodiments, the disclosed system can quickly fingerprint a firearm and ammunition pair in a particular environment using a specific recording device. In other words, the systems and methods described herein can very quickly determine the acoustic signature of a firearm and ammunition combination in an environment. This allows the system to distinguish the analyzed acoustic signature from other firearm and ammunition combinations. This can be particularly useful, for example, in conjunction with a system performing automated target scoring, at busy shooting ranges where it is important to distinguish one shooter from another. By being able to distinguish between firearm and ammunition pairs, the scoring system can determine that a particular firearm and ammunition combination is utilized and that it is likely to coincide in time with a hit on the target, thereby reducing the number of false positives and missed shots. In some examples, the system can be executed on a mobile computing device and used at shooting ranges. If the mobile computing device has a microphone pointed in the general direction of the shooter of interest, shots fired by the shooter of interest will generally have an audio file dominated by shots fired by the shooter of interest, which can help determine whether the shots fired are from the shooter of interest. In some cases, the recording device (e.g., the mobile computing device) has a microphone pointed downrange, such as when the mobile computing device has a camera pointed at the target for automatic target scoring, in which case the volume of shots fired by the shooter of interest will be more difficult to distinguish from other shooters at the range. In these cases, the described embodiments can use training of machine learning algorithms or training of weighting masks to quickly distinguish shots fired by the intended shooter from all other shooters at the range.

[0133] In some cases, classifiers rely on feature extraction in the form of time-frequency spectrograms. Mel-frequency cepstrum coefficient (MFCC) vectors can be used as an alternative raw power spectral density vector (PSD) for the frequency representation of a spectrogram. Mel-frequency cepstrum (MFCC) was originally developed for speech processing and the way information is thought to be encoded in speech waveforms. In some cases, MFCs can be applied for firearm fingerprinting purposes. As used throughout this disclosure, "firearm fingerprint" refers to an acoustic signature, or a specific sound or sound image, that identifies a particular firearm and ammunition combination.

[0134] In some cases, MFCCs may be generated by performing a series of steps, including, but not limited to, i) windowing a signal segment and computing a fast Fourier transform (FFT) of the signal segment, ii) combining the linear FFT coefficients with MEL frequency filter bank coefficients, iii) taking the logarithm of those coefficients, and iv) computing a discrete cosine transform (DCT) of the logarithmic MEL filter bank coefficients. The FFT of a signal segment generally produces a peak at the applied frequency, as well as other peaks, typically called sidelobes, on either side of the peak frequency. The DCT represents a finite sequence of data points in terms of a sum of cosine functions oscillating at different frequencies. In some cases, fewer or more steps than those disclosed may be implemented to arrive at a firearm fingerprint based on gunfire sounds. For example, in some cases, only steps i), ii), and iii) above may be used for gunfire sounds.

[0135] According to some embodiments, the MFCC or PSD coefficients may be used as input to a DNN trained to classify the time-frequency spectrogram into several independent categories. For example, the MFCC and / or PSD coefficients may input two categories as proposed, or several categories corresponding to a (firearm, ammunition) pair, or a larger number of categories corresponding to a tuple of k>2 attributes.

[0136] In some cases, the DNN may place a new gunshot sound "near" whatever class is represented by the largest classifier output among all classifier outputs. For example, a gunshot sound may be initially classified through a nearest neighbor technique such as a k-NN algorithm. Subsequent analysis may further classify the gunshot sound.

[0137] The inner layers of the DNN may represent different sets of attributes of the input, which may also be used for gunfire classification and training.

[0138] From an optimization theory perspective, training a DNN can be thought of as essentially defining a surface with multiple local optima, and the DNN directing new inputs to the most appropriate local optima. In some embodiments, placing one or more trainable layers after a pre-trained DNN can be thought of as extracting a different set of more optimal attributes for gunfire sounds.

[0139] In some cases, placing a trainable layer, possibly including weighting coefficients, in front of the pre-trained DNN can be thought of as tuning the pre-trained DNN so that the shooter's shot of interest is the most positive example of all shots that the pre-trained DNN places in the "neighborhood" of whatever class the maximum classifier output represents among all classifier outputs. In some cases, the decision threshold can be tuned to optimize a confusion matrix for a training dataset that is used to tune the trainable input layer or another test dataset for some useful criterion.

[0140] According to some embodiments, the FFT of a windowed segment of shot audio can be computed in O(n log n) time. Computing the MFCCs has the same time order but involves computing two O(n log n) operations. In some cases, the MFCCs are not significantly better than the FFT and may be omitted.

[0141] According to some embodiments, the MEL frequency log spectrum (omitting the discrete cosine transform (DCT) to obtain the cepstrum) provides an improvement over the raw FFT, with only an extra computational cost of O(n). Therefore, in some examples, the MEL frequency log spectrum is used rather than the DCT to obtain the cepstrum.

[0142] In general, there is no fast form for computing functions computed by DNNs (composed of layers of convolutional neural networks (CNNs)), such as the FFT. The FFT exploits the regularity of the FFT kernel, which is inherently absent in the construction of any CNN. However, the examples described herein involve an FFT of the input time signal, and the pre-trained DNNs composed of CNNs can be implemented in the frequency domain.

[0143] As a non-limiting example, the following DNN-based rapidly trainable category recognizer can be implemented to rapidly determine the acoustic signature of a firearm and ammunition combination:

[0144] Let f:RM→RK represent the analog transformation of a real-valued M-dimensional input vector into a K-dimensional vector of class probabilities by a trained deep neural network. The final discrete output mapping Ψ:RK→N≦K selects the most likely of the K classes.

[0145] The system may be configured to extend the trained DNN into an enhanced binary classifier for class k. In some cases, the system may implement a rapidly trainable input stage based on the Hadamard product of a weighting matrix W and an input vector x.

number

[0146] Γ:RK×N≦K→R provides a single class probability for class k. An extended DNN may implement the following functions:

[0147]

number

[0148] An additional set of input vectors that are all instances of the same class k

number

[0149]

number

[0150] where 0<α<1 is an adaptive constant that weights the relative contribution of ζ to W. For all vectors in ζ, the target value is d(n)=1.0. Iterations are performed for each z in ζ until convergence is achieved. Here, relu[...] denotes element-wise relu() of the argument matrix.

[0151] Firing a firearm produces multiple acoustic events, such as a muzzle blast caused by gases expanding in the chamber and exiting through the barrel, and a ballistic shock wave generated by the projectile, which are most often supersonic but may also be subsonic in some cases. The acoustic events are the result of variables that produce a firearm signature, which may include, among others, the type, make, and model of the firearm, barrel length, type of ammunition, amount and identity of gunpowder, projectile weight, and projectile shape.

[0152] FIG. 26 shows a sample process 2600 for correlating firearms with a particular shooter, according to some embodiments. When a shooter visits a shooting range to practice, there may be other shooters firing firearms at the range. In some cases, a mixed range may have 10, 20, or even 30 or more shooters all firing at the same time. It may be very difficult for an audio system to register shots fired by a shooter of interest amid the clutter of firearm fire. In some cases, the system is configured to distinguish between the firearm of the shooter of interest and the firearms of other shooters at the range.

[0153] For example, in block 2602, the system receives video data of the shooter, which also includes audio data, which may be received, for example, via a multi-camera system, a dedicated microphone, or a consumer-grade audio / video capture device such as a mobile computing device.

[0154] In block 2604, the system may determine the shooter's body landmarks.

[0155] In block 2606, the system may track one or more body landmarks while the shooter fires the shot and generate motion data associated with the body landmarks during the shot.

[0156] In block 2608, the system correlates the audio data with the motion data to determine that a shot was fired by the shooter of interest. In some cases, the system may receive audio data indicative of a firearm firing, which may be represented as a spike in the sound wave file. This may be correlated with motion data, such as the shooter's wrist, indicating that recoil from the firearm displaced the shooter's hand and therefore that a shot was fired. In some cases, the system is trained to distinguish the shooter's firearm from other firearms at the range. In this case, the system may be trained to distinguish between shots fired by the shooter of interest and shots fired by other shooters at the range.

[0157] In block 2610, the system may determine a score for the shot and associate the score with the action data that led to the score.

[0158] In block 2612, the system may analyze the performance data in combination with the score to determine any mistakes the shooter made and provide suggestions for identifying mistakes and / or how to address mistakes in the future. The system may also provide training exercises to enable the shooter to address mistakes and improve their shooting performance.

[0159] Referring to FIG. 27, which illustrates an example system 2700 using online learning, audio 2702 is received and converted into an incremental spectrogram 2704. The spectrogram 2704 is framed (2706), for example, by a time window and used to determine MFCCs, for example, by determining the FFT of the windowed signal, combining the linear FFT coefficients with MEL frequency filter bank coefficients, determining the logarithm of the coefficients, and determining the DCT of the logarithmic MEL filter bank coefficients. The determined MFCCs can be input to a rapidly trainable input stage 2708 and then delivered to a DNN multi-class classifier 2710. A selector 2734 determines the classifier output k with the highest class probability to classify the shot. The rapidly trainable input stage 2708 may be referred to as a trainable weighting mask, or simply a mask. The mask can be tuned using instances of the shooter's firearm and ammunition fire as recorded in a particular environment. In some cases, training only the weighting mask is much less computationally intensive than training a DNN. The trained (e.g., tuned) weighting mask may represent the signature of a combination of firearm, ammunition, environment, and recording device. This signature is not necessarily used to search a catalog of signatures, but rather is used to tune the spectrogram input to improve the DNN's classification performance for firearms and ammunition in a particular environment using a given recording device. In some cases, the training data set for a particular shooter (e.g., a particular firearm and ammunition combination) is small, so the weighting mask can be updated using online methods, such as by using stochastic gradient descent, or offline methods, such as by using gradient descent techniques.

[0160] The inner layers of the DNN classifier 2710 may represent different sets of attributes of the input. In some cases, the DNN is trained to define a surface with multiple local optima, and the DNN can operate to guide new inputs to the most appropriate local optima. The enhanced DNN classification can then be used to improve the detection of shots from one firearm and to distinguish them from those of other shooters.

[0161] The weighting mask W of the input stage 2708 to the pre-trained DNN classifier 2710 may be adapted by an online learning loop 2712 that optimizes the weighting mask W.

[0162] Online training loop 2712 includes copies 2714, 2716, and 2718 of shot classifier input stages 2708, 2710, and 2734. When a new shot n is detected, subtractor 2720 in online training loop 2712 compares the results of the selected output k from the copies of shot classifiers 2714, 2716, and 2718 with the expected classification d(n)=1.0 to calculate a classification error term.

[0163] Block 2730 of the adaptation loop calculates the vector sign of the gradient of the DNN output with respect to the spectrogram input for the current output spectrogram x(n) from the framing block 2706.

[0164] Next, the raw incremental adjustment to the current weighting vector W i (n) is calculated by Hadamard multiplier 2732 as the product of the current shot spectrogram from framer 2706 and the code vector of classifier gradient 2730. Vector multiplier 2722 then scales the raw incremental adjustment to the weighting vector W i (n) by an arbitrary value α.

[0165] Finally, vector adder 2724 calculates preliminary updated weights by adding the current incremental adjustment to the current W i (n), which are converted into updated weight vector W i+1(n) by vector relu[] operation 126.

[0166] The weight adaptation loop 2712 described above is repeated until the weight vectors converge, e.g., |W i (n) -W i +1 (n)| ≤ δ, as indicated by the delta operator 2728. The resulting W i +1 (n) is then selected as the weight vector W(n+1) for classifying the next shot.

[0167] Offline learning can be used, such as when ζ is small. In some cases, offline learning can begin by setting W=I and updating it using gradient descent as follows:

number

[0168] In some cases, offline learning may require roughly the same amount of computation as online learning. However, online learning has the advantage over offline learning that offline learning requires access to the entire training dataset ζ at every iteration of the update function, whereas stochastic gradient descent does not. Offline learning offers the advantage of finding a local optimum, whereas online learning may only approximate it.

[0169] The current argument

number

[0170] While some systems aim to enhance DNNs to classify shots, they do so by adding layers to the output of a pre-trained network to customize it for a specific task. The pre-trained network extracts many levels of increasingly abstract features, such as from an image, and additional trainable layers are used to focus on the problem of interest. In contrast, many of the systems and methods described herein work in a much different way, achieving a more efficient, much faster, and more accurate system. The systems described herein often only add a single layer to the input of a pre-trained network. In some use cases, the camera is pointed at a target rather than focused on the shooter, and the camera and microphone pick up other shots without being able to natively determine that the gunfire came from the shooter of interest. Acoustic signatures are generated quickly and are often executed on a mobile device (e.g., a smartphone) that includes the camera and microphone. The mobile device may execute instructions (e.g., an application) that include the components and systems described herein so that classification and acoustic signature determination are performed on the mobile device. In some cases, the pre-trained network may be trained on a relatively comprehensive universe of shots. The trainable input layer may pre-distort the input data to achieve high-probability recognition by the pre-trained network of a particular firearm in the munitions and environment. This may increase the likelihood of detecting shots of interest and rejecting all others.

[0171] According to some embodiments, the systems and methods described herein converge to a useful weighting mask where the gradient of the function computed by the DNN is monotonically non-decreasing or monotonically non-increasing at each input.

[0172] Referring to FIG. 28 , which illustrates pre-training a DNN, process 2800 begins 2802, where an audio file is opened in block 2804, which may be the first audio file or the next or subsequent audio file. Using the audio file, the system captures blocks of samples framing the first and / or next shot in step 2806. In other words, each shot is windowed in time-bounded samples. In block 2808, spectrograms associated with the samples are generated and labeled with attributes.

[0173] In block 2810, the system determines whether the most recent sample is associated with the last shot. If not, the system returns to block 2806 to capture a block of samples associated with a subsequent shot. If so, the system proceeds to block 2812 to determine whether the labeled spectrogram created in block 2808 is the last file. If not, the system returns to block 2804 to open or capture the next audio file. If the system determines that the most recent file is the last file, the system proceeds to block 2814, where the labeled spectrograms are aggregated. In block 2816, a DNN is trained with the labeled spectrogram. The system stops at block 2818 with a trained DNN.

[0174] Referring to FIG. 29, which illustrates deriving spectrogram weights, process 2900 begins at 2902, where a set of shots is captured at block 2904. The shots may be captured by an audio and / or video recording device or may involve opening a file associated with one or more shots. At block 2906, the system captures a block of samples framing the first and / or next shot. In other words, each shot is windowed in time-bounded samples. At block 2908, spectrograms associated with the samples are created and labeled with attributes.

[0175] In block 2910, the system determines whether the most recent spectrogram is associated with the last shot. If not, the system returns to block 2906 to capture a block of samples associated with a subsequent shot. If so, the system proceeds to block 2912 to determine whether the labeled spectrogram created in block 2908 is the last set. If not, the system returns to block 2904 to capture the next set of shots. If the system determines that the most recent file is the last file, the system proceeds to block 2914, where the labeled spectrograms are aggregated. In block 2916, the system updates W (weighting) until the value converges. The system stops at block 2918 with the spectrogram weightings obtained.

[0176] Referring to FIG. 30, a process for classifying a shot 3000 is shown. The process begins at block 3002, where the system captures a block of samples framing the shot at block 3004. At block 3006, the system determines a spectrogram associated with the block of samples. At block 3008, the system determines a Hadamard product of the spectrogram and a weight matrix. At block 3010, the Hadamard product of the spectrogram and the weight matrix is ​​applied to a DNN. At block 3012, the system makes a binary decision as to whether the shot is associated with the signature of the firearm in question. In some cases, the system can identify the type of firearm and the ammunition fired via the firearm. For example, upon receiving an audio sample, the system can determine, without prior knowledge of the firearm in question, that the acoustic signature of the firearm in the audio sample corresponds to a 230-grain round-nosed projectile fired from a Beretta .45 ACP. At block 3014, the process stops.

[0177] The system may include one or more processors and one or more computer-readable media that may store various modules, applications, programs, or other data. The computer-readable media may include instructions that, when executed by the one or more processors, cause the processors to perform the operations described herein for the system.

[0178] In some implementations, the processor(s) may include a central processing unit (CPU), a graphical processing unit (GPU), both a CPU and a GPU, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or additionally, the functions described herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), etc. Additionally, each of the processor(s) may possess its own local memory, which may also store program modules, program data, and / or one or more operating systems. One or more control systems, computer controllers, and remote controls may include one or more cores.

[0179] Automated Target Scoring

[0180] In some cases, the described systems operate in near real time on a single consumer recording device, such as a mobile phone, using only a modest amount of training data. As used herein, the terms "real time" or "near real time" are broad terms and, in the context of this disclosure, relate to receiving input data, processing the input data, and outputting the results of data analysis with little or no human-perceived latency. In other words, a system such as that described herein that outputs analyzed data within less than one second is considered near real time. A system operating in real time or near real time may limit the amount of computation for machine learning, or at least training models, that the method can use to characterize specific target acquisition, classification, and scoring.

[0181] Previous approaches to automated target scoring have relied on acoustic triangulation, optical triangulation, and piezoelectric sensor triangulation. Acoustic triangulation has been attempted by using acoustic chamber targets that use the Mach waves of a projectile to determine the location of the projectile as it passes through the target. Acoustic triangulation automated scoring systems operate by using microphones to measure the sound waves of the projectile as it passes through the target. The sound of the projectile passing through the target from multiple audio sensors (e.g., microphones) can then be used to determine the location where the projectile passed through the target.

[0182] Optical triangulation automated scoring systems use three or more lasers, such as infrared lasers, to triangulate the projectile's position as it passes through the target. Piezoelectric sensor triangulation systems rely on an array of piezoelectric sensors on a plate that sense vibrations caused by the projectile impacting the target.

[0183] FIG. 31 illustrates a system 3100 configured to automatically identify targets, classify targets, determine target impacts, and score firing strings. The system 3100 may include a computing resource 3102, which may be a mobile computing device associated with a participant at a shooting range and may include any one or more of several mobile computing devices, such as a smartphone, a tablet computer, a laptop computer, or other suitable computing device. The computing resource 3102 typically includes one or more processors 3104 and a memory 3106 that stores one or more modules 3108. The modules 3108 may store instructions that, when executed, cause the one or more processors 3104 to perform various operations. The computing resource 3102 may further include data storage, which may be remote storage, such as a remote server or a cloud-based storage system, or local storage, or a combination thereof. Data storage can store DOPE (Data On Previous Engagement), which can allow for data tracking over time, as well as comparative data between different shooters, different firearms, different ammunition, different targets, different environments, etc.

[0184] The storage system can further enable historical trend analysis, which can be used to show a shooter's performance over time, including tracking improvements. Data storage can also be analyzed to provide performance predictions, rankings, and social features, among other benefits.

[0185] The system may incorporate one or more imaging sensors 3110, such as any suitable video camera. In some cases, the imaging sensor 3110 may be associated with a computing resource 3102. For example, in some embodiments, the computing resource 3102 may be a smartphone with a built-in camera 3110.

[0186] The camera 3110 may be aimed to capture an image of a target 3112. The target may be located at any distance from the shooter, and the camera 3110 may be aimed and / or zoomed to capture an image of the target. In some embodiments, the camera may be coupled to a lens, such as a spotting scope or camera lens, allowing the camera to capture a closer view of the target through optical or digital zoom.

[0187] The computing resources 3102 may include, among other things, instructions (e.g., modules 3108) that enable the computing device to initialize 3118 targets, detect 3114 impacts on the targets, and score 3116 impacts on the targets.

[0188] FIG. 32 shows a decision tree 3200 configured to detect, identify, and classify targets. According to some embodiments, the system does not know the type of target before the system begins searching for the target. For example, in some conventional systems, the scoring system may be pre-programmed with the target against which the shooter will aim. This allows the system to easily understand the size and shape of the target and the location and boundaries of each scoring ring or zone. In the illustrated embodiment, the system is configured to automatically determine and determine the scoring rings and zones without a priori data regarding the type of target. For example, the system may have one or more video capture devices that may be integrated into one or more mobile computing devices. As used herein, a mobile computing device may be one or more of a mobile phone, smartphone, tablet, laptop, personal digital assistant, smart glasses, body cam, wearable computing device, or any other computing device that a user may carry to a shooting range.

[0189] The mobile device may activate the camera and capture one or more frames of the target 3202. The computing device may have instructions to analyze the one or more frames and identify the target within the one or more frames by using any suitable image analysis algorithm. If a target is detected and classified in block 3204, the target is registered with the system 3206, and a scoring ring and area are determined. The system may capture additional image frames containing the target and look for differences between one frame and the next that may correlate with bullet impact on the target. The frames may be compared, and a moving average may be generated in block 3208. Moving averages are fundamental mathematical and statistical techniques applied in image analysis and machine learning for various purposes, including noise reduction, feature extraction, and trend analysis. They involve calculating the average value of pixel intensities or other data points within a moving window or kernel across an image or dataset. Moving averages can be used to extract meaningful features from an image. For example, important information can be highlighted by sliding a small window over the image and calculating the average pixel value within that window. For example, in edge detection, a moving average can highlight areas with abrupt changes in pixel intensity, helping to identify the edges or boundaries of the target and scoring area. Edge detection can also be used to identify bullet impacts on the target.

[0190] In some instances, moving averages are used in time series data analysis. As one example, moving averages can be used to establish a baseline behavior of a system to detect anomalies. Data points that deviate significantly from this baseline can be flagged as anomalies or outliers. These anomalies can be further analyzed to determine target impact.

[0191] Once the sequential moving averages are generated, they may be combined as a long-term moving average. In block 3210, the moving average image may be compared to the long-term moving average to determine differences from one frame to a subsequent frame that indicate changes to the target that are most likely associated with impact on the target.

[0192] The impacts are selected and classified in block 3212. For example, the system determines the boundaries of a scoring ring, determines the location of each impact, and associates the location of each impact with a score for the impact.

[0193] Returning to block 3204, if a target has not previously been detected and classified, for example, if the shooter is initializing the system or has replaced a target, the system determines whether a target has been detected in block 3214. If not, the system proceeds to detect the target in block 3216. If a target has been detected, the system classifies the target in block 3218, for example, by identifying the target boundaries, scoring ring boundaries, and scoring ring values.

[0194] If the system does not detect the target, the system may capture one or more additional image frames and analyze the one or more additional image frames to determine that the target is located within the field of view of the imaging device. Once the target is detected, the system may classify the target and determine its size, as well as the relative positions and sizes of the scoring rings or regions.

[0195] Figure 33 further describes the initial steps a system may take to identify and classify a target 3300 by analyzing one or more image frames. Object detection is a computer vision technique that involves identifying and locating multiple objects within an image or video stream. Unlike image classification, which determines the presence of a single object class in an entire image, object detection provides a more granular understanding by not only recognizing objects but also specifying their location through bounding boxes. In some embodiments, object detection algorithms typically output bounding boxes that enclose detected objects. These bounding boxes consist of the coordinates (x, y) of the object's upper-left corner and dimensions (width and height) that define the object's spatial extent within the image.

[0196] In block 3302, the system applies object detection to one or more images of the target to search for the target. In some embodiments, the object detection model is generic with respect to the target, allowing the system to detect any target regardless of size or shape. In block 3304, if the target is found in the same location in subsequent images (e.g., two or more images, three or more images, four or more images, etc.), the system assumes it has located the target and defines a bounding box around the target. In some cases, finding the target in the same location in subsequent images includes determining a running average of the images to determine the target's location, size, and shape.

[0197] At block 3306, the targets are optionally classified. In addition to locating objects, the system may be configured to detect objects and classify each detected object into a predetermined class or category. This allows the system to distinguish between different object types, such as circular targets, oval targets, rectangular targets, silhouette targets, etc.

[0198] A target classifier may be applied to the images within the bounding box, thus allowing the system to determine which reference target image to apply.

[0199] In block 3308, the system registers the target to the reference target image. In some cases, this involves applying a contrast adjustment to the image. This may also involve iteratively modifying the initial bounding box, such as by adjusting its corners and then projecting the adjusted bounding box onto the reference target image. The difference between the two may be applied as a score, and a hill-climbing technique may be applied to find the optimal corner, which can be correlated with the initial position of the image. Hill-climbing is an optimization algorithm used to find the local maximum (or minimum) of a given objective function. By iteratively taking small steps in the direction of higher values, the algorithm determines the highest and lowest values, which can then be used to determine the boundary of the target. In some cases, object detection is combined with semantic segmentation to provide pixel-level object masks. This allows for a more accurate understanding of object boundaries in the image, such as target boundaries and scoring ring boundaries.

[0200] In block 3310, the system initializes and registers the target and begins looking for impacts over the subsequent moving average.

[0201] FIG. 34 illustrates a process 3400 for registering a target 3300 and determining target impacts. In block 3402, the target may be re-registered, such as by performing a hill-climbing technique to search for a new set of best corners of the target. In some cases, the hill-climbing search uses mean-squared distance in a perceptual space technique. For example, mean-squared distance, also known as mean-squared perceptual error, is a metric used to measure the similarity or dissimilarity between two data points containing perceptual data such as target corners. In some cases, for each data point, associated perceptual features (in this case, target corners, edges, scoring rings, etc.) are extracted. Features can be visual descriptors that can be represented as vectors of perceptual features. These feature vectors capture relevant information about each data point in a more concise and informative form.

[0202] The mean squared distance between two points (represented as their respective feature vectors) is generated by determining the squared difference between corresponding features and calculating the average of these squared differences. The resulting mean squared distance provides a quantitative measure of the dissimilarity between two data points in perceptual space.

[0203] In block 3404, the system may apply a transformation matrix that may be used to map the set of corners to an image. In some cases, the image to which the coordinates are mapped has dimensions of 160 pixels, and in other cases, it may be less than 160 pixels.

[0204] In block 3406, the moving average image is updated, with the long-term moving average being, in some cases, about 10 seconds or more and the short-term moving average being about 0.1 seconds. In some cases, the video camera may capture more than 30 frames per second. For the short-term moving average, this equates to averaging about 3 frames to determine the short-term moving average.

[0205] In block 3408, the system determines the difference between the moving averages. For example, a long-term moving average may be associated with a static target that has not changed for 10 seconds or so and compared to a short-term moving average that reflects changes in the image. Thus, the difference between the short-term and long-term moving averages highlights changes to the image, such as a bullet hitting the target. The system may convolve any difference image with a simple impact kernel, which may be a 5x5 uniformly weighted square kernel, to find the location of the largest block in the difference image. A kernel typically refers to a convolution filter that may be used to process and modify pixel values ​​for feature extraction, etc. The square kernel may be convolved (or moved) across the image, and at each location, the kernel's value may be multiplied by the pixel values ​​in the corresponding neighborhood, and the results may be summed to generate a new pixel value in the output image. Of course, the size of the kernel may be varied to adjust the range of the neighborhood considered during the convolution, and may include any of the full set of aperture kernels with non-uniform weights, and may have any suitable size.

[0206] The convolution returns a set of potential hits on the target. The set of potential hits may be further filtered, such as by using simple statistics of a window surrounding the flagged difference. In some cases, the window is chosen to be a 16x16 window with the difference in the center of the window. Of course, other window sizes are entirely reasonable, and the pixel values ​​described herein are merely illustrative of some embodiments. The system may also apply some business rules to the windowed differences; for example, the system should not detect multiple hits in the exact same location.

[0207] In block 3410, a target impact is determined. In some cases, this is accomplished by passing the filtered set of differences to a hit classifier for scoring. If the difference score exceeds a threshold, the location is marked as a hit, and another window may be placed around the hit. In some cases, a 10x10 window is placed around the hit location in the long-term average, and the short-term average is used over 5-10 subsequent frames. This ensures that the same hit is not detected again. Thus, the difference is windowed with a first window, and if the difference exceeds a threshold score, the difference is windowed with a second window smaller than the first window. The windowed difference associated with the short-term moving average may be added to a long-term moving average of at least 5 frames, or at least 6 frames, or at least 10 frames, or at least 12 frames, or at least 15 frames or more. In some cases, if the difference score falls below the threshold, the difference is marked as a false hit, and the system does not need to evaluate and classify it again.

[0208] According to some embodiments, the system may receive audio data associated with a shot being fired and determine that a shot has been fired based on the audio data. In some cases, the audio data is correlated with a target image, and the system can convolve a differential target image in response to audio data indicating that a shot has been fired. In some cases, the system may not need to continuously convolve a differential image. In this case, the system can determine that a shot has been fired through the audio data, then update a short-term moving average and convolve the differential image to look for the shot. In some cases, the system is configured to distinguish shots fired by a user who has set their sights on a target from other shooters at the shooting range. In this way, the system can know when a shooter of interest will fire a shot, even if other active shooters are at the range.

[0209] In some cases, the audio data may be used in impact detection, such as by correlating the audio of a fired shot with impacts appearing on a target image.

[0210] 35A-35C illustrate and describe initializing the scoring system by identifying and classifying targets. In some cases, the system can automatically determine the target boundaries, but in some embodiments, user input may define the target boundaries. For example, using a human-computer interface (e.g., a touchscreen, mouse, stylus, touchpad, etc.), a human can draw a boundary around a target to help the system identify it. However, in many embodiments, the system uses machine vision to identify the target and its boundaries. FIG. 35A shows an image 3500 captured by a camera associated with the system. The image may include a target stand 3502, a target 3504, a target fixation clip 3506, and other features within the field of view. The system may determine an initial bounding box 3508 around the identified target, such as by a trained target detection model. In some embodiments, a user may define the initial bounding box, such as by drawing on a computer display using the human-computer interface. The human-computer interface may be any suitable interface, such as a touchscreen, pen, mouse, trackball, etc. The initial bounding box may not exactly match the edges and corners of the target, especially if the bounding box is user-defined. The initial bounding box and target image are sometimes referred to as an initialization frame. The initialization frame may be converted to Lab color space, which includes components for lightness, a green-to-red axis, and a blue-to-yellow axis, to produce perceptual uniformity. In some cases, the luminance channel is equalized via contrast-limited adaptive histogram equalization (CLAHE).

[0211] 35B shows a target whose coordinates have been determined as described above, often representing a quadrilateral that can be projected onto a reference target image 3510. The reference target image 3510 may also be converted to Lab color space, and a squared difference (in Lab space) may be generated between the projection and the target. A hill-climbing algorithm may be applied to the coordinates, where a possible delta is a small change to the coordinate, and a better solution may be determined by the squared perceptual difference.

[0212] 35C shows the best coordinates, such as determined by minimum difference over several random restarts of the hill climbing algorithm. These coordinates may then be used to apply an updated bounding box 3512. Thus, even if the target image is distorted, such as when the viewing angle from the camera makes the target appear as a parallelogram rather than a rectangle, the initial bounding box may be modified to match the shape of the target as presented in the image captured by the camera.

[0213] In some embodiments, the system may define the edges of a target through image analysis. However, in some cases, the edges of the target are irrelevant; only the scoring rings are important. Thus, in some cases, the system is configured to identify the scoring rings and is not concerned with the target boundary. Additionally, the system only needs to identify the scoring rings, rather than classifying the target. For example, the system may determine through one or more machine learning models that a target represents a central bullseye target with a series of scoring rings. The system may assign a score value to each ring, such as 10 points for the bullseye, 9 points for the next larger ring, and so on. Similarly, the system may identify a target with five bullseye-sized circles spaced throughout the target and assign a value of 10 points to each of these scoring rings. One or more of the multiple bullseye-sized rings may have radially spaced larger scoring rings that may be assigned a smaller value than the bullseye-sized rings. Thus, the system need only omit the step of classifying the target and focus solely on the size and location of the scoring rings.

[0214] 36 illustrates and describes hit detection and scoring of detected hits. A machine learning model may be run to determine whether the difference between image frames is likely to be a projectile hit on the target 3504. The target 3504 may be re-registered, such as by applying a hill-climbing technique using possible corner coordinates, as in the target initialization step. The differences are windowed (3602a, 3602b, 3602c) by comparing the short-term moving average difference to the long-term moving average to generate a difference image 3612 between the current target and the long-term target average.

[0215] The difference image may be convolved with an impact kernel (e.g., a windowed kernel that scans the image difference). Any point that is several standard deviations away from the long-term exponential moving average (EMA) of the maximum convolution value is flagged as a possible impact 3604a, 3604b, 3604c.

[0216] The potential hits 3604a-3604c are fed to a machine learning model (e.g., a classifier) ​​3606 that determines whether the difference is likely to be an actual hit. If the difference is above a threshold, the system marks the difference as an actual hit 3608. However, if the difference is below the threshold, the system marks the difference as a false hit 3610.

[0217] FIG. 37 illustrates and describes scoring of shots on a target 3504. Different scoring zones may be determined by the system based on computer vision, by referencing registered targets from previous shooting sessions, by retrieving stored target models from a known target database, or in some other manner. Scoring zones 3702 on the target may be represented as connected areas of one or more simple shapes (e.g., ellipses, rectangles, circles, triangles, etc.). Coordinates of detected shots 3704 may be normalized and transformed to axes implied by a reference image. In other words, the shots may be overlaid on a reference image, which can be used for the shot's coordinates. The coordinates may be Cartesian coordinates expressed as x, y values. In some cases, the coordinates may be radial coordinates, representing the shot as an angle and distance from the center of the target, etc. The system can then determine whether the shot is entirely within a single scoring zone or penetrates a scoring zone boundary, allowing the system to accurately score the shot. The system may use simple geometric shapes to determine whether a significant portion of a given impact falls within any of the simple shapes of each target zone.

[0218] In some embodiments, the system uses the coordinates of the impacts for further analysis. For example, by generating and storing the coordinates for a given firing string, grouping can be quantified and used as a measure of improvement over time. Similarly, a shooter's MOA (moment of angle) is a measure of group size in inches and arc minutes and can be determined from center to center and edge to edge. Additionally, grouping may be used to define pose, grip, or motion errors during the firing string. Grouping may be quantified, including group size, group rotation, or other metrics.

[0219] Scoring can be quantified by any suitable metric. In some cases, scoring is point-based, with points awarded for each zone on the target being added to or subtracted from an initial amount. In some cases, missed shots or extra shots fired are scored as negative or higher values, depending on the scoring type. In some cases, timed scoring is used, where total time is reflected in the score, with time increased as a penalty for misses. In some cases, group size is used to determine scoring, with extra or missed shots penalizing group size. Of course, other metrics and combinations of metrics may be determined by the system to score a particular firing string.

[0220] The system may be configured to return the coordinates of impact and the time each shot occurred for each shot in a firing string, so that the firing string can be ranged using some metric that combines position and time. For example, the time between shots may be measured, or shots after a buzzer or other initiation signal may be tracked and stored along with an accuracy metric.

[0221] In some cases, once impacts are identified, the system may draw a bounding box around one or more of the impacts. When a firing string ends, the system may draw a bounding box containing each of the shots in the group and determine a metric based on the bounding box to determine a score.

[0222] 38, which illustrates a user interface 3800 of a system developed and operable in accordance with certain of the embodiments described herein, the system may be configured to determine a bounding box 3802 that may pass through the center of the outermost impact or along the edge of the impact. The system may determine any of several metrics, such as, but not limited to, group size 3804, group overall width 3806, group height 3808, bounding box rotation angle, MOA, elevation offset 3810, windage offset 3812, and may further determine shot distance 3814, which may be manually entered or determined based on the detected projectile's time of flight.

[0223] For example, the system may be configured to register the sound of gunfire, shock waves from projectile or gunpowder explosions, movement of the firearm or shooter, or some other indicator that a shot has been fired. The system may then detect when impact on the target occurs, determine the ammunition's time of flight, and determine the target distance based on the firearm, ammunition, and / or charge. This process can be performed in near real time by a simple consumer-grade mobile computing device. In some cases, the mobile computing device may utilize the zoom capabilities of an integrated image capture device. In some cases, an external zoom lens may be used to capture image frames by the mobile computing device. For example, a mobile phone may be coupled to a spotting scope, which provides optical zoom through the spotting scope, allowing the mobile computing device to capture clearer images of targets that may be downrange. Some mobile computing devices may rely on digital zoom to capture one or more images of targets located downrange.

[0224] The system may further determine and display the number of shots fired in the current shooting string 3816, the average split time for each shot 3818, which may be useful in timed shooting competitions. The system may further show the score 3820 associated with each shot and the cumulative score 3822 for the shooting string.

[0225] Some embodiments further provide an automated scoring system capable of quickly identifying targets, classifying targets, including identifying target scoring rings, and scoring target hits at a shooting range. In some cases, the system is stored and executed on a consumer mobile computing device (e.g., an iPhone, tablet, phone, video camera). In some cases, the system includes a video camera device pointed at a target of interest, and the system is configured to identify the target, classify the target, determine shot impacts on the target, and score the target impacts. In some cases, the system is configured to prompt the shooter regarding the shooting stage. For example, the system may be configured for use in a CMP high-powered rifle competition, and the system may prompt the user that the current stage requires firing 20 shots downrange from an off-hand position within a 20-minute time frame. In some cases, the system knows the number of shots expected during the shooting stage (called a firing sequence) and may prompt the user with information related to the current shooting stage, such as the number of shots, the time frame, and the shooting location. In some cases, the shooter may input information associated with the shooting stage, such as the number of shots the system should expect, the firearm to be used, and the distance to the target, among other things. In some cases, the system is started and stopped manually, identifying shots on target only during the time the system is started.

[0226] Comprehensive Training System

[0227] As described above, in some embodiments, the system utilizes a gantry-style arrangement that may use multiple recording devices to capture audio and video of participants and / or targets. In some cases, an imaging device may be aimed at the target, which may have a zoom lens, digital zoom capabilities, or rely on an external lens, such as a camera mounted on a spotting scope, to capture images of targets located downrange. In some cases, the system is configured to synchronize multiple sources, such as one or more video frames and / or audio data from one or more audio capture devices. In some cases, the system is configured to synchronize multiple video frames and audio data from one or more audio / video capture devices.

[0228] The described system therefore provides a comprehensive firearms training solution that enables shooters, even when in a mixed range, to track body movements, identify errors in technique, correlate scores with errors in technique, automatically score targets, and distinguish shots fired from a participant's firearm from those from other firearms.

[0229] The system may utilize one or more machine learning models for synchronization, prediction, and validation, and may be further trained to analyze scoring data and associate it with motion data to determine correlations between specific motion data (e.g., behaviors) and scoring trends. As an example, the system may correlate a shooter's wrist pivoting downward before a shot with typically scoring outside and below the ten-ring and determine that the shooter is anticipating firearm recoil before the shot. The system may then provide feedback to the user with information related to the motion / score correlation, as well as one or more exercises or drills to enable the user to recognize and address behaviors that result in reduced scores. The system may be further trained to distinguish firearm fires associated with a shooter of interest, even when in a mixed range with multiple shooters. As described elsewhere herein, a similar process may be used for any motion data from any activity or sport.

[0230] In some embodiments utilizing multiple video capture devices, the system may track body motion in two or three dimensions from multiple angles. The two or three dimensional body motion data may be correlated, synchronized, and analyzed to determine two or three dimensional motion data that may be further correlated with the resulting score.

[0231] Although the described system embodiments are described in the context of a shooter firing a series of shots, it should be understood that the systems and methods described herein are applicable to capturing any type of physical motion, as well as to other sports in which physical motion can lead to performance metrics. For example, the system embodiments described herein can record the motion of a basketball free throw, a golf swing, a figure skating element, archery, soccer, a baseball swing, or a series of observable physical landmarks over time, and can be used to track, critique, and improve physical motion, such as any other sport or movement, if the motion has any observable causal consequences.

[0232] The system may include one or more processors and one or more computer-readable media that may store various modules, applications, programs, or other data. The computer-readable media may include instructions that, when executed by the one or more processors, cause the processors to perform the operations described herein for the system.

[0233] In some implementations, the processor(s) may include a central processing unit (CPU), a graphical processing unit (GPU), both a CPU and a GPU, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or additionally, the functions described herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), etc. Additionally, each of the processor(s) may possess its own local memory, which may also store program modules, program data, and / or one or more operating systems. One or more control systems, computer controllers, and remote controls may include one or more cores.

[0234] Embodiments may be provided as a computer program product including a non-transitory machine-readable storage medium having stored thereon instructions (in compressed or uncompressed format) used to program a computer (or other electronic device) to perform the processes or methods described herein. Computer-readable media may include volatile and / or non-volatile memory, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Machine-readable storage media include, but are not limited to, hard drives, floppy disks, optical disks, CD-ROMs, DVDs, read-only memory (ROM), random-access memory (RAM), EPROMs, EEPROMs, flash memory, magnetic or optical cards, solid-state memory devices, or other types of media / machine-readable media suitable for storing electronic instructions. Furthermore, embodiments may be provided as a computer program product including a transitory machine-readable signal (in compressed or uncompressed format). Examples of machine-readable signals include, but are not limited to, signals (including signals downloaded over the Internet or other networks) that can be configured to be accessed by a computer system or machine that hosts or executes a computer program, whether or not modulated using a carrier wave.

[0235] Those skilled in the art will recognize that any process or method disclosed herein can be modified in many ways. The process parameters and order of steps described and / or illustrated herein are given by way of example only and can be changed as desired. For example, although the steps illustrated and / or described herein may be shown or described in a particular order, these steps do not necessarily have to be performed in the order shown or described.

[0236] The various exemplary methods described and / or illustrated herein may omit one or more of the steps described or illustrated herein or may include additional steps in addition to those disclosed. Furthermore, the steps of any method as disclosed herein may be combined with any one or more steps of any other method as disclosed herein.

[0237] This disclosure describes exemplary embodiments and is therefore not intended to limit the scope of the embodiments of the present disclosure and the appended claims in any way. The embodiments are described above using functional components that illustrate implementations of specified functions and their relationships. The boundaries of these functional components have been arbitrarily defined herein for the convenience of description. Alternative boundaries may be defined to the extent that the specified functions and their relationships are appropriately performed.

[0238] The foregoing description of specific embodiments sufficiently clarifies the general nature of the embodiments of the present disclosure so that others, by applying the knowledge of those skilled in the art, can easily modify and / or adapt such specific embodiments for various applications without undue experimentation and without departing from the general concept of the embodiments of the present disclosure. Therefore, such adaptations and modifications are intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein. The phrases or terms used herein are for the purpose of description, not limitation, and should be interpreted by those skilled in the art in light of the teaching and guidance presented herein.

[0239] The breadth and scope of embodiments of the present disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

[0240] In particular, conditional language such as "can," "could," "might," or "may," unless expressly stated otherwise or understood otherwise within the context in which it is used, is generally intended to convey that certain implementations include certain features, elements, and / or operations, but may not include them in other implementations. Thus, such conditional language is generally not intended to imply that features, elements, and / or operations are somehow required in one or more implementations, or that one or more implementations necessarily include logic for determining whether those features, elements, and / or operations should be included or performed in any particular implementation, with or without user input or prompting.

[0241] Unless otherwise noted, as used herein, the terms "connected to" and "coupled to" (and their derivatives) should be interpreted as allowing both direct and indirect (i.e., via other elements or components) connections. Additionally, as used herein, the terms "a" or "an" should be interpreted as meaning "at least one of." Finally, for ease of use, as used herein, the terms "including" and "having" (and their derivatives) should be interpreted as open-ended and not excluding additional components.

[0242] This specification and the accompanying drawings disclose examples of systems, apparatus, devices, and techniques that may provide systems and methods for determining the acoustic signature of a fired firearm. For purposes of describing various features of the present disclosure, it is, of course, not possible to describe every conceivable combination of elements and / or methodologies, but those skilled in the art will recognize that many further combinations and permutations of the disclosed features are possible. Accordingly, various modifications can be made to the present disclosure without departing from the scope or spirit of the disclosure. Moreover, other embodiments of the present disclosure will be apparent from consideration of the specification and accompanying drawings, as well as from practice of the disclosed embodiments presented herein. The examples presented in this specification and the accompanying drawings are to be considered in all respects as illustrative and not restrictive. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

[0243] Those skilled in the art will understand that in some implementations, the functionality provided by the processes and systems described above may be provided in alternative ways, for example, divided among more software programs or routines or integrated into fewer programs or routines. Similarly, in some implementations, the illustrated processes and systems may provide more or less functionality than described, such as when other illustrated processes, respectively, lack or include such functionality, or when the amount of functionality provided is varied. Additionally, while various operations may be shown as being performed in a particular way (e.g., sequentially or simultaneously) and / or in a particular order, those skilled in the art will understand that in other implementations, these operations may be performed in other orders and in other ways. Those skilled in the art will also understand that the data structures described above may be structured differently, for example, by dividing a single data structure into multiple data structures or by combining multiple data structures into a single data structure. Similarly, in some implementations, the illustrated data structures may store more or less information than described, such as when other illustrated data structures, respectively, lack or include such information, or when the amount or type of information stored is varied. The various methods and systems shown in the figures and described herein represent example implementations. In other implementations, the methods and systems may be implemented in software, hardware, or a combination thereof. Similarly, in other implementations, the order of any method may be changed, and various elements may be added, rearranged, combined, omitted, modified, etc.

[0244] From the foregoing, it will be understood that, although specific implementations have been described herein for illustrative purposes, various modifications may be made without departing from the spirit and scope of the appended claims and the elements described therein. In addition, while certain aspects are presented below in certain claim forms, the inventors contemplate various aspects in any available claim form. For example, while only some aspects may currently be described as being embodied in a particular configuration, other aspects may likewise be so embodied. Various modifications and changes may be made, as would be apparent to one skilled in the art having the benefit of this disclosure. All such modifications and changes are intended to be encompassed, and therefore the above description should be regarded in an illustrative rather than a limiting sense.

Claims

1. 1. A method for improving shooting performance, comprising: receiving video data of the shooter; determining one or more body landmarks of the shooter; tracking the one or more body landmarks during a shot to generate shot motion data; determining a score for said shot; Associating the shot motion data with the score; generating recommendations for altering the motion data in subsequent shots; A method comprising:

2. The method of claim 1 , wherein determining one or more body landmarks of the shooter comprises generating a wireframe model by connecting the body landmarks.

3. 2. The method of claim 1, wherein associating the shot motion data with the score comprises executing a classification and regression tree machine learning model to identify a causal relationship between the shot motion data and the score.

4. The method of claim 1 , further comprising determining the shooter's grip through image analysis of the video data of the shooter.

5. 5. The method of claim 4, further comprising analyzing the grip of the shooter and providing, on a display screen, grip recommendations for changing the grip.

6. Determining the score for the shot comprises: receiving target video data; performing image analysis on the received target video data; determining a hit on the target; determining a score for said hit; The method of claim 1 , comprising:

7. The method of claim 1 , wherein receiving the video data includes capturing the video data by a mobile phone.

8. The method of claim 1 , wherein determining one or more body landmarks includes determining 17 body landmarks.

9. The method of claim 1 , wherein tracking the one or more body landmarks comprises generating a bounding box around each of the one or more body landmarks.

10. The method of claim 1 , further comprising running a machine learning model to correlate the shot motion data with the score.

11. The method of claim 10 , wherein the machine learning model is configured to determine the motion data that results in an off-center hit on a target.

12. 1. A method for improving the causal consequences of bodily movement, comprising: receiving video data of the body movement; determining one or more body landmarks visible in the video data of the body movement; tracking the one or more body landmarks during an action; generating motion data based at least in part on tracking the one or more body landmarks; determining a score associated with the motion data; associating the performance data with the score; generating recommendations for altering the operational data for subsequent actions; A method comprising:

13. The method of claim 12 , wherein receiving the video data includes capturing the video data with a mobile computing device.

14. The method of claim 12 , further comprising running a machine learning model to correlate the operational data with the score.

15. The method of claim 14 , wherein the machine learning model is configured to determine the behavioral data that results in a reduced score.

16. The method of claim 15 , further comprising predicting, by the machine learning model, a predicted score based on the operational data.

17. The method of claim 16 , further comprising comparing the predicted score to the score.

Images