Long and short soldier training trajectory analysis method and system based on three-dimensional reconstruction

Abstract

The invention provides a long and short soldier training trajectory analysis method and system based on three-dimensional reconstruction. A training scene is decoupled into a static background and a dynamic foreground containing trainees and weapons; a parameterized six-degree-of-freedom B spline function is introduced to accurately model the rigid body motion of the weapon; meanwhile, a deformable human body model is adopted to represent non-rigid deformation of a human body; space points in the dynamic foreground at the t moment are converted from an observation space to a unified standard space, and coordinates of the standard space are input into a second nerve radiation field; sampling along the light of the camera by using a volume rendering technology, querying color and density in combination with the two neural radiation fields, and generating a rendered image through integration; after luminosity loss of a rendered image and a real image is minimized and joint optimization convergence is carried out, an optimal weapon B spline function is extracted, zero-order, first-order and second-order derivatives are calculated, and a three-dimensional motion track, a posture, a linear velocity, an angular velocity and an acceleration of a weapon in the whole training process are output.

Description

Technical Field

[0001] This application belongs to the field of trajectory analysis, and in particular relates to a method and system for trajectory analysis of long and short weapons training based on three-dimensional reconstruction. Background Technology

[0002] In fields such as martial arts, sports competitions, and military training, acquiring key motion parameters of weapons in three-dimensional space, such as posture, velocity, and acceleration, can provide objective and quantitative data support for optimizing athletes' or soldiers' movements, evaluating techniques, and formulating tactics. Optical motion capture systems reconstruct high-precision three-dimensional motion trajectories by attaching reflective markers to weapons and trainees' bodies and using multiple high-speed cameras for tracking. However, this marker-based method has significant drawbacks: the equipment is expensive, requires deployment in specialized venues, and the marker attachment process is cumbersome, prone to displacement or detachment during vigorous movement, interfering with the training process itself. Traditional computer vision techniques perform markerless motion analysis using ordinary videos from single or multiple perspectives, but these methods struggle to meet the high-precision requirements for weapon trajectory analysis when dealing with complex occlusion problems, rapid movements, and accurately reconstructing three-dimensional postures. NeRF, through a multilayer perceptron, implicitly learns the geometric and appearance information of a scene, enabling the synthesis of highly realistic new perspective images from sparse multi-view images. To handle dynamic scenes, traditional NeRF methods have been extended, for example, by introducing time as input or constructing a deformation field from the observation space to the gauge space to model the motion of objects. These dynamic NeRF methods have achieved significant results in the reconstruction and synthesis of new perspectives in dynamic scenes such as people and vehicles. They typically decompose the scene into a static background and a dynamic foreground, and model the deformation of the dynamic foreground uniformly. Existing dynamic NeRF methods mainly aim to achieve high-quality visual rendering, often treating dynamic foregrounds such as the trainee and the weapon in their hand as a whole, and modeling them as a fuzzy, unified non-rigid deformation. This makes it difficult to effectively decouple and explicitly express the rigid motion of the weapon from the non-rigid motion of the human body. Therefore, it is difficult to directly and accurately extract the weapon as an independent rigid body, including its continuous six-degree-of-freedom motion trajectory, angular velocity, and other high-order motion parameters, thus limiting its application in professional motion analysis. Summary of the Invention

[0003] To address the aforementioned problems, in the first aspect of this application, a method for analyzing the training trajectories of long and short weapons based on 3D reconstruction is proposed, comprising the following steps: Acquire multi-view synchronous video sequences and camera intrinsic and extrinsic parameters for the weapon training process; decompose the training scene into a static background field and a dynamic foreground field, wherein the dynamic foreground field includes the trainee and the weapon, and respectively use a first neural radiation field to represent the static background field and a second neural radiation field to represent the dynamic foreground field. A parameterized six-DOF B-spline function is constructed to describe the rigid body motion of a weapon, and a deformable human body model is constructed to characterize the non-rigid deformation of the human body. For any spatial point in the dynamic foreground field at time t, the observation space is transformed to a unified normed space using a mapping parameterized by the B-spline function and the deformable human body model, and the coordinates of the spatial point in the normed space are used as features input to the second neural radiation field. The camera light rays are sampled using volume rendering technology. For each sampling point, the color and density are queried and synthesized by combining the first neural radiation field and the second neural radiation field. The rendered image is obtained by integrating along the light rays. The photometric rendering loss between the rendered image and the real observed image is constructed, and the first neural radiation field and the second neural radiation field, the weapon six-degree-of-freedom B-spline function and the deformable human body model are jointly optimized. After the joint optimization converges, the optimal six-degree-of-freedom B-spline function of the weapon is extracted, and the zeroth, first and second derivatives are calculated to obtain the three-dimensional motion trajectory, attitude, linear velocity, angular velocity and acceleration of the weapon throughout the training process.

[0004] In a second aspect of this application, a training trajectory analysis system for long and short weapons based on three-dimensional reconstruction is proposed, comprising the following modules: The scene decomposition module is used to acquire multi-view synchronous video sequences and camera internal and external parameters for the weapon training process; the training scene is decomposed into a static background field and a dynamic foreground field, wherein the dynamic foreground field includes the trainee and the weapon, and the static background field is represented by a first neural radiation field and the dynamic foreground field is represented by a second neural radiation field. The mapping module is used to construct a parameterized six-DOF B-spline function to describe the rigid body motion of a weapon, and a deformable human body model to characterize the non-rigid deformation of the human body. For any spatial point in the dynamic foreground field at time t, the mapping parameterized by the B-spline function and the deformable human body model is used to transform the observation space to a unified normed space, and the coordinates of the spatial point in the normed space are used as features input to the second neural radiation field. The optimization module is used to sample along the camera rays using volume rendering technology. For each sampling point, it combines the first neural radiation field and the second neural radiation field to query and synthesize color and density, and integrates along the rays to obtain the rendered image. It constructs the photometric rendering loss between the rendered image and the real observed image, and jointly optimizes the first neural radiation field and the second neural radiation field, the weapon's six-degree-of-freedom B-spline function, and the deformable human body model. The analysis module is used to extract the optimal six-degree-of-freedom B-spline function of the weapon after the joint optimization has converged, and to calculate the zeroth, first and second derivatives to obtain the three-dimensional motion trajectory, attitude, linear velocity, angular velocity and acceleration of the weapon throughout the training process.

[0005] In an optional embodiment, acquiring the multi-view synchronized video sequence and camera intrinsic and extrinsic parameters for the weapon training process includes: Eight to twelve high-speed industrial cameras with a frame rate of no less than 100fps were deployed in a ring around the training area, and the intrinsic and extrinsic parameters of all cameras were calculated using a checkerboard calibration board and a joint beam adjustment algorithm. A hardware synchronization controller ensures that the start timestamp error of all camera exposures is less than 1 millisecond.

[0006] In an optional embodiment, decomposing the training scene into a static background field and a dynamic foreground field includes: Pixel-level masks of the trainee and weapon in each frame of the image are extracted using an instance segmentation network. During the joint optimization process, the photometric loss for optimizing the first neural radiation field is calculated based on the pixel region outside the mask, and the photometric loss for optimizing the second neural radiation field is calculated based on the pixel region covered by the mask.

[0007] In an optional embodiment, the transformation from the observation space to a unified canonical space using a mapping jointly parameterized by the B-spline function and the deformable human body model includes: For any sampling point in the observation space at time t, the sampling point is transformed to a unified standard space. Specifically, the inverse transformation of the deformable human body model is applied to the points belonging to the trainee, and the inverse matrix of the transformation corresponding to the six-degree-of-freedom B-spline function is applied to the points belonging to the weapon. Within this unified, standardized space, trainees and weapons together constitute a static, standardized scenario.

[0008] In an optional embodiment, the joint optimization includes: Construct the total loss function ,in To compensate for the photometric error between the rendered image and the real image To render the cumulative opacity of the light rays and the binary cross-entropy loss between the foreground mask, For the parameter regularization term of the deformable human body model, This is a motion smoothing regularization term applied to the B-spline function to penalize excessive velocity or acceleration.

[0009] In an optional embodiment, the construction of the parameterized six-DOF B-spline function for describing the rigid body motion of the weapon includes: A cubic B-spline function with a series of SE(3) group elements as control points is constructed. The continuous motion trajectory of the weapon in the time dimension is fitted by optimizing the control points. The SE(3) group elements encode the weapon's three-dimensional rotation matrix and translation vector.

[0010] In an optional embodiment, obtaining the weapon's three-dimensional motion trajectory, attitude, linear velocity, angular velocity, and acceleration throughout the training process includes: From the optimized and converged B-spline function, the SE(3) matrix at any time t is obtained by interpolation, which serves as the three-dimensional attitude and position of the weapon at that time. The first and second time derivatives of the B-spline function are obtained and mapped from the Lie algebra SE(3) space to the Euclidean space. The linear velocity, angular velocity, linear acceleration and angular acceleration of the weapon are obtained analytically.

[0011] This invention eliminates the need for attaching any markers to weapons or trainees, utilizing only ordinary multi-view videos for analysis. It decouples the rigid motion of the weapon from the non-rigid deformation of the human body, providing an explicit and continuous mathematical description of the weapon's motion through a parameterized six-DOF B-spline function. This description is then jointly optimized with a deformable human body model within a neural radiation field framework. This not only accurately reconstructs the weapon's three-dimensional trajectory and posture throughout the entire training process but also allows for the direct calculation of higher-order kinematic parameters such as linear velocity, angular velocity, and acceleration by differentiating the B-spline function. This provides training evaluation with refined and quantitative data support that is difficult to achieve with traditional markerless methods. Attached Figure Description

[0012] Figure 1 A flowchart of a specific embodiment; Figure 2 This is a schematic diagram showing the distribution of high-speed cameras; Figure 3 This is a schematic diagram illustrating the separation of a static background and a dynamic foreground. Figure 4 A schematic diagram of mapping to a uniform standard space; Figure 5 This is a schematic diagram for joint optimization. Detailed Implementation

[0013] To facilitate understanding of the technical solution of this application, the application will be further described below with reference to the accompanying drawings.

[0014] The terms "first" and "second," etc., used in the specification, claims, and drawings of this application are used only to distinguish different objects and not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0015] The term "embodiment" as used herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0016] In this application, "at least one (item)" refers to one or more, "more than one" refers to two or more, "at least two (items)" refers to two or three or more, and "and / or" is used to describe the relationship between related objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. "Or" indicates that there can be two relationships, such as only A exists and only B exists; when A and B are not mutually exclusive, it can also mean that there are three relationships, such as only A exists, only B exists, and both A and B exist simultaneously. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items. For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c".

[0017] In one embodiment, this application proposes a method for analyzing the training trajectories of long and short weapons based on three-dimensional reconstruction, such as... Figure 1 As shown, it includes the following steps: S1. Acquire multi-view synchronous video sequences and camera internal and external parameters for the weapon training process; decompose the training scene into a static background field and a dynamic foreground field, wherein the dynamic foreground field includes the trainee and the weapon, and respectively use a first neural radiation field to represent the static background field and a second neural radiation field to represent the dynamic foreground field. Deploy multiple high-speed cameras, such as 4 to 8, around the training area. Figure 2 As shown, hardware synchronization or post-processing visual alignment is used to ensure that all video frames are strictly aligned in time; the initial few frames of static images are processed using open-source Structure-from-Motion toolkits such as COLMAP to automatically calibrate the external parameters such as position and orientation of each camera, as well as the internal parameters such as focal length and distortion.

[0018] Each frame of the image can be processed using instance segmentation networks such as Mask R-CNN to generate foreground masks for the trainer and weapons, thereby dividing the image pixels into static background and dynamic foreground. Alternatively, object detection models such as the YOLO series can be used to decompose the training scene, and the decomposition results are as follows: Figure 3As shown; the first neural radiation field is a multilayer perceptron model, which takes three-dimensional spatial coordinates as input and outputs the color and volume density of the point; the second neural radiation field is also a multilayer perceptron model, which takes three-dimensional coordinates in normal space as input and outputs the color and volume density of the point in normal state.

[0019] S2, construct a parameterized six-DOF B-spline function to describe the rigid body motion of a weapon, and a deformable human body model to characterize the non-rigid deformation of the human body; for any spatial point in the dynamic foreground field at time t, use the mapping parameterized by the B-spline function and the deformable human body model to transform the observation space to a unified normed space, and input the coordinates of the spatial point in the normed space as features into the second neural radiation field. The weapon's six degrees of freedom motion, namely three-dimensional translation and three-dimensional rotation, is modeled by a time-varying fourth-order B-spline function, and the control points of this function are the parameters to be optimized. The non-rigid deformation of the human body is represented using the industry-standard SMPL model, and its posture and body shape parameters are also variables to be optimized. For any spatial point in the foreground region at time t, it is determined whether it belongs to the weapon or the human body, for example, through target recognition or detection. If it belongs to the weapon, the inverse transformation of the rigid body transformation defined by the B-spline function at time t is applied to map it to the weapon's canonical model space; if it belongs to the human body, the inverse linear hybrid skinning transformation of the non-rigid deformation defined by the SMPL model at time t is applied to map it to the human body's standard A-pose canonical space, thus obtaining the coordinates of the point in the unified canonical space, such as... Figure 4 As shown.

[0020] S3. Sampling is performed along the camera light rays using volume rendering technology. For each sampling point, color and density are queried and synthesized by combining the first neural radiation field and the second neural radiation field. The rendered image is obtained by integrating along the light rays. The photometric rendering loss between the rendered image and the real observed image is constructed, and the first neural radiation field and the second neural radiation field, the weapon's six-degree-of-freedom B-spline function, and the deformable human body model are jointly optimized. For any pixel in the image, layered random sampling is performed along its corresponding camera ray to obtain a series of 3D sampling points. For each sampling point, if it is located in the background region, its color and density are directly obtained by querying the first neural radiation field. If it is located in the foreground region, the aforementioned dynamic mapping transformation to the normal space is performed first, and the color and density are obtained by querying the second neural radiation field. Then, using the standard volume rendering integral formula, the color and density of all sampling points are accumulated along the ray to calculate the final rendered color of the pixel. The L2 norm, i.e., the mean square error, between the rendered colors of all pixels and the colors of the real video image is calculated as the photometric loss. The Adam optimizer is used to simultaneously update the weights of the two neural radiation field networks, the control points of the B-spline function, and all parameters of the SMPL model through the backpropagation algorithm. Figure 5 As shown.

[0021] S4. After the joint optimization converges, extract the optimal six-degree-of-freedom B-spline function of the weapon and calculate the zeroth, first and second derivatives to obtain the three-dimensional motion trajectory, attitude, linear velocity, angular velocity and acceleration of the weapon throughout the training process.

[0022] Optimization convergence means that the control points of the B-spline function have reached the optimal solution. The optimal B-spline function is evaluated by dense sampling on the time axis. Its zeroth derivative value is the three-dimensional spatial position and rotational attitude of the weapon at each moment, forming the motion trajectory. Using the analytic differentiability of the B-spline function, its first derivative can be calculated to obtain the linear velocity vector of the weapon's center of mass and the angular velocity vector of the attitude change. Similarly, by calculating its second derivative, the linear acceleration of the weapon's center of mass and the angular acceleration of the attitude change can be obtained.

[0023] In an optional embodiment, acquiring the multi-view synchronized video sequence and camera intrinsic and extrinsic parameters for the weapon training process includes: Eight to twelve high-speed industrial cameras with a frame rate of no less than 100fps were deployed in a ring around the training area, and the intrinsic and extrinsic parameters of all cameras were calculated using a checkerboard calibration board and a joint beam adjustment algorithm. A hardware synchronization controller ensures that the start timestamp error of all camera exposures is less than 1 millisecond.

[0024] To fully capture the details of the trainee's high-speed weapon-wielding movements, such as sword slashes or spear thrusts, this embodiment deploys 10 high-speed industrial cameras evenly around the circumference of a circular area with a radius of 5 meters. These cameras are set to a frame rate of 200fps, significantly higher than the 30fps of conventional video, enabling clear recording of each instantaneous movement and avoiding motion blur. This circular deployment ensures that regardless of the trainee's orientation, their movements are captured unobstructed by at least three cameras, providing ample visual data for subsequent 3D reconstruction.

[0025] Before data acquisition, the spatial position and optical characteristics of all cameras need to be precisely calibrated. Staff moved a 9x6 checkerboard calibration board across the field, allowing all cameras to photograph the calibration board from different angles. These images were used to initially calculate the intrinsic parameters of each camera, such as focal length and distortion coefficient, as well as the extrinsic parameters, namely the camera's position and orientation relative to the world coordinate system. Subsequently, a joint-beam adjustment algorithm was used to globally optimize the intrinsic and extrinsic parameters of all cameras, reducing the average reprojection error of 3D points under all camera views to below 0.5 pixels, establishing a high-precision unified 3D space. To achieve precise time alignment of multi-view images, a hardware synchronization controller sent trigger signals to all cameras, ensuring that the difference in the start time of each camera's shutter exposure was less than 0.5 milliseconds. Sub-millisecond synchronization accuracy is crucial for analyzing weapon terminal velocity exceeding 20 meters per second, preventing tearing or distortion of the 3D reconstruction model due to time asynchrony.

[0026] In an optional embodiment, decomposing the training scene into a static background field and a dynamic foreground field includes: Pixel-level masks of the trainee and weapon in each frame of the image are extracted using an instance segmentation network. During the joint optimization process, the photometric loss for optimizing the first neural radiation field is calculated based on the pixel region outside the mask, and the photometric loss for optimizing the second neural radiation field is calculated based on the pixel region covered by the mask.

[0027] To improve reconstruction efficiency and quality, the complex training scene is decoupled into two parts for modeling. The first part is the static background field, i.e., the training ground itself, such as the ground, walls, and fixed training equipment. The second part is the dynamic foreground field, which includes the trainee and their weapon. To achieve this separation, a pre-trained instance segmentation network, such as YOLACT or Mask R-CNN, is used to process each frame of the input image. This instance segmentation network can output a pixel-level accurate mask, clearly defining which pixels belong to the trainee, which belong to the weapon, and which belong to the background.

[0028] In the subsequent neural radiation field training phase, when optimizing the first neural radiation field representing the static background, its photometric loss function only calculates the pixel area outside the mask, that is, it only compares the rendered background color with the background color in the real image. Conversely, when optimizing the second neural radiation field representing the dynamic foreground, the photometric loss function only focuses on the pixel area covered by the mask. In this way, the geometric and textural information of the static background is not disturbed by moving people and weapons, and the modeling of the dynamic foreground does not need to consider complex background changes, allowing both neural radiation fields to learn a purer and more accurate scene representation, and the final synthesized video is clearer and more realistic.

[0029] In an optional embodiment, the transformation from the observation space to a unified canonical space using a mapping jointly parameterized by the B-spline function and the deformable human body model includes: For any sampling point in the observation space at time t, the sampling point is transformed to a unified standard space. Specifically, the inverse transformation of the deformable human body model is applied to the points belonging to the trainee, and the inverse matrix of the transformation corresponding to the six-degree-of-freedom B-spline function is applied to the points belonging to the weapon. Within this unified, standardized space, trainees and weapons together constitute a static, standardized scenario.

[0030] The canonical space is a static three-dimensional reference frame in which the trainee maintains a standard static posture, such as a T-pose, while the weapon is in a fixed initial position and posture. The complex time-varying four-dimensional problem—three-dimensional space plus time—is decomposed into a simple static three-dimensional geometry learning problem and a dynamic motion learning problem. At any time t, when querying the color and density of a three-dimensional point P in the observation space, this point P is first mapped back to the canonical space. If the segmentation information indicates that point P belongs to the trainee, the inverse transformation corresponding to the pose parameters of the deformable human model at time t is applied. For example, if at time t the model shows the right arm raised 30 degrees forward, the inverse transformation will rotate the point on the right arm in the observation space 30 degrees backward, returning it to the canonical T-pose position. If point P belongs to the weapon, the inverse matrix of the pose transformation matrix calculated by the six-degree-of-freedom B-spline function at time t is applied. For example, if the weapon translates 2 meters and rotates 45 degrees at time t, the inverse matrix transformation will move it back to the origin position in the canonical space. Through this mapping, the dynamic foreground at all times is unified into the same static, standardized scene for learning, which greatly simplifies the modeling difficulty of neural radiation fields.

[0031] In an optional embodiment, the joint optimization includes: Construct the total loss function ,in To compensate for the photometric error between the rendered image and the real image To render the cumulative opacity of the light rays and the binary cross-entropy loss between the foreground mask, For the parameter regularization term of the deformable human body model, This is a motion smoothing regularization term applied to the B-spline function to penalize excessive velocity or acceleration.

[0032] To optimize the final reconstruction result in terms of visual, geometric, and physical properties, a joint loss function with multiple sub-items is employed. The total loss L is a weighted sum of four key components, each targeting a specific optimization objective. The weighting coefficients α, β, and γ are preset hyperparameters, such as 0.1, 0.01, and 0.01 respectively, used to balance the relative importance of different loss terms.

[0033] First item This is the photometric loss, calculated by comparing the pixel colors of the rendered image with those of the actual camera-captured image, for example using the L2 norm, to ensure visual fidelity in the reconstructed scene. (Second term) It's a mask. It uses binary cross-entropy to compare the cumulative opacity rendered along each ray with the foreground mask, ensuring that the rendered foreground outline is precisely aligned with the outline extracted by the instance segmentation network, thus avoiding the problem of blurred model edges. (Third item) This is the human posture regularization term. It imposes constraints on the posture parameters of deformable human models, penalizing postures that do not conform to the laws of human kinematics, such as excessive joint bending or limb clipping, thus ensuring the naturalness of human movements. (Fourth term) It is a weapon motion smoothing regularization term. By penalizing the higher-order derivatives of the B-spline function, i.e. excessive velocity and acceleration, it ensures that the weapon's trajectory is physically smooth and continuous, eliminating unrealistic situations such as instantaneous movement or violent shaking.

[0034] More specifically, in one embodiment, the photometric loss is achieved by... The calculation yields a result where R is a set of rays randomly sampled from all camera views in one optimization iteration, and r is a ray in set R. The true pixel color of the ray r. The color of the rendered pixel; if the pixel is outside the mask, Rendered by the first neural radiation field; otherwise, rendered by both the first and second neural radiation fields. Mask loss is used. The calculation yielded that, It is a set that contains only the light rays passing through the foreground region. This is the true mask value corresponding to the ray r. If the pixel belongs to the foreground... If it belongs to the background, ; The cumulative opacity of the rays rendered on the model represents the probability that the rendered soft mask or rays will hit the object. The regularization term for the deformable human model uses... The calculation yielded that, The pose prior loss penalizes unnatural joint angles, preferably implemented through a pre-trained variational autoencoder (VAE) or Gaussian mixture model (GMM). To compensate for the prior loss on body size and penalize unrealistic body proportions, the body size parameters of the human model are preferably selected from low-dimensional coefficients obtained through principal component analysis. Weapon motion smoothness regularization term. The calculation formula is S(t) is the B-spline function of SE(3) for the six-degree-of-freedom motion of the weapon. S(t) outputs an SE(3) matrix at any time t, representing the weapon's attitude (rotation matrix) and position (translation vector). and These are the first and second derivatives of the B-spline function with respect to time t, respectively. In the formula above, Represents the square of the L2 norm, for example and They are all RGB three-channel color vectors, for example (R,G,B), and the difference between them... It is a difference vector (ΔR, ΔG, ΔB). norm is , Then it is .

[0035] In an optional embodiment, the construction of the parameterized six-DOF B-spline function for describing the rigid body motion of the weapon includes: A cubic B-spline function with a series of SE(3) group elements as control points is constructed. The continuous motion trajectory of the weapon in the time dimension is fitted by optimizing the control points. The SE(3) group elements encode the weapon's three-dimensional rotation matrix and translation vector.

[0036] To accurately represent the continuous motion of a rigid body like a weapon during training, a continuous time function was constructed. A cubic B-spline function was chosen because of its excellent property of continuous second derivative; the trajectory it describes is smooth in both velocity and acceleration, consistent with the motion of objects in the real world.

[0037] The B-spline function is a series of elements on a special Euclidean group SE(3). Each SE(3) group element is a 4x4 matrix that can simultaneously and compactly encode the weapon's three-dimensional rotation information (a 3x3 rotation matrix) and three-dimensional translation information (a 3x1 translation vector). For example, for a sword dance movement lasting 10 seconds, it may only require setting 20 SE(3) control points. By optimizing the positions of these 20 control points, the B-spline function can generate a complete, smooth, six-degree-of-freedom motion trajectory that covers the entire duration and is queryable at any millisecond. Compared to optimizing thousands of independent poses, this greatly reduces computational complexity and the risk of overfitting.

[0038] In an optional embodiment, obtaining the weapon's three-dimensional motion trajectory, attitude, linear velocity, angular velocity, and acceleration throughout the training process includes: From the optimized and converged B-spline function, the SE(3) matrix at any time t is obtained by interpolation, which serves as the three-dimensional attitude and position of the weapon at that time. The first and second time derivatives of the B-spline function are obtained and mapped from the Lie algebra SE(3) space to the Euclidean space. The linear velocity, angular velocity, linear acceleration and angular acceleration of the weapon are obtained analytically.

[0039] Once the joint optimization of the entire system converges, a final B-spline function that accurately describes the weapon's motion is obtained. This function allows for detailed kinematic analysis. To obtain the weapon's precise three-dimensional attitude and position at any given time t, such as 2.51 seconds, simply substitute t=2.51 into the B-spline function for interpolation to obtain an SE(3) matrix. The rotation part of this matrix gives the weapon's orientation at that moment, while the translation part gives its coordinates in three-dimensional space. By continuously querying the pose at different times, the complete motion trajectory of the weapon throughout the entire training process can be plotted.

[0040] Since the B-spline function is analytically differentiable, its first and second derivatives in time can be directly calculated to obtain high-precision velocity and acceleration information. Calculating the first derivative of the function yields a result in the Lie algebra SE(3) space, representing the instantaneous velocity. Through a standard mapping relationship, it can be converted into a six-dimensional vector, where the first three dimensions are angular velocities in radians per second, and the last three dimensions are linear velocities in meters per second. Similarly, by calculating the second derivative of the function and mapping it, the angular and linear accelerations of the weapon can be obtained. For example, this method can accurately calculate the peak velocity of the spear tip accelerating from rest to 15 meters per second within 0.1 seconds when the spear is thrust out, as well as the acceleration change curve throughout the process, avoiding the numerical errors and noise caused by differential calculations based on discrete frames. As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0041] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0042] The method and electronic device for providing product object information provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for analyzing the training trajectories of long and short weapons based on three-dimensional reconstruction, characterized in that, Includes the following steps: Acquire multi-view synchronous video sequences and camera internal and external parameters for weapons training processes; The training scene is decomposed into a static background field and a dynamic foreground field. The dynamic foreground field includes the trainee and the weapon. The static background field is represented by a first neural radiation field and the dynamic foreground field is represented by a second neural radiation field. A parameterized six-DOF B-spline function is constructed to describe the rigid body motion of a weapon, and a deformable human body model is constructed to characterize the non-rigid deformation of the human body. For any spatial point in the dynamic foreground field at time t, the observation space is transformed to a unified normed space using a mapping parameterized by the B-spline function and the deformable human body model, and the coordinates of the spatial point in the normed space are used as features input to the second neural radiation field. The camera light rays are sampled using volume rendering technology. For each sampling point, the color and density are queried and synthesized by combining the first neural radiation field and the second neural radiation field. The rendered image is obtained by integrating along the light rays. The photometric rendering loss between the rendered image and the real observed image is constructed, and the first neural radiation field and the second neural radiation field, the weapon six-degree-of-freedom B-spline function and the deformable human body model are jointly optimized. After the joint optimization converges, the optimal six-degree-of-freedom B-spline function of the weapon is extracted, and the zeroth, first and second derivatives are calculated to obtain the three-dimensional motion trajectory, attitude, linear velocity, angular velocity and acceleration of the weapon throughout the training process.

2. The method according to claim 1, characterized in that, The acquisition of multi-view synchronized video sequences and camera intrinsic and extrinsic parameters for the weapons training process includes: Eight to twelve high-speed industrial cameras with a frame rate of no less than 100fps were deployed in a ring around the training area, and the intrinsic and extrinsic parameters of all cameras were calculated using a checkerboard calibration board and a joint beam adjustment algorithm. A hardware synchronization controller ensures that the start timestamp error of all camera exposures is less than 1 millisecond.

3. The method according to claim 1, characterized in that, The process of decomposing the training scene into a static background field and a dynamic foreground field includes: Pixel-level masks of the trainee and weapon in each frame of the image are extracted using an instance segmentation network. During the joint optimization process, the photometric loss for optimizing the first neural radiation field is calculated based on the pixel region outside the mask, and the photometric loss for optimizing the second neural radiation field is calculated based on the pixel region covered by the mask.

4. The method according to claim 1, characterized in that, The transformation from the observation space to a unified normed space using a mapping parameterized by the B-spline function and the deformable human body model includes: For any sampling point in the observation space at time t, the sampling point is transformed to a unified standard space. Specifically, the inverse transformation of the deformable human body model is applied to the points belonging to the trainee, and the inverse matrix of the transformation corresponding to the six-degree-of-freedom B-spline function is applied to the points belonging to the weapon. Within this unified, standardized space, trainees and weapons together constitute a static, standardized scenario.

5. The method according to claim 1, characterized in that, The joint optimization includes: Construct the total loss function ,in To compensate for the photometric error between the rendered image and the real image To render the cumulative opacity of the light rays and the binary cross-entropy loss between the foreground mask, For the parameter regularization term of the deformable human body model, This is a motion smoothing regularization term applied to the B-spline function to penalize excessive velocity or acceleration.

6. The method according to claim 1, characterized in that, The construction of the parameterized six-DOF B-spline function for describing the rigid body motion of a weapon includes: A cubic B-spline function with a series of SE(3) group elements as control points is constructed. The continuous motion trajectory of the weapon in the time dimension is fitted by optimizing the control points. The SE(3) group elements encode the weapon's three-dimensional rotation matrix and translation vector.

7. The method according to claim 1, characterized in that, The acquisition of the weapon's three-dimensional motion trajectory, attitude, linear velocity, angular velocity, and acceleration throughout the training process includes: From the optimized and converged B-spline function, the SE(3) matrix at any time t is obtained by interpolation, which serves as the three-dimensional attitude and position of the weapon at that time. The first and second time derivatives of the B-spline function are obtained and mapped from the Lie algebra SE(3) space to the Euclidean space. The linear velocity, angular velocity, linear acceleration and angular acceleration of the weapon are obtained analytically.

8. A training trajectory analysis system for long and short weapons based on three-dimensional reconstruction, characterized in that, Includes the following modules: The scene decomposition module is used to acquire multi-view synchronized video sequences and camera intrinsic and extrinsic parameters for the weapon training process; The training scene is decomposed into a static background field and a dynamic foreground field. The dynamic foreground field includes the trainee and the weapon. The static background field is represented by a first neural radiation field and the dynamic foreground field is represented by a second neural radiation field. The mapping module is used to construct a parameterized six-DOF B-spline function to describe the rigid body motion of a weapon, and a deformable human body model to characterize the non-rigid deformation of the human body. For any spatial point in the dynamic foreground field at time t, the mapping parameterized by the B-spline function and the deformable human body model is used to transform the observation space to a unified normed space, and the coordinates of the spatial point in the normed space are used as features input to the second neural radiation field. The optimization module is used to sample along the camera rays using volume rendering technology. For each sampling point, it combines the first neural radiation field and the second neural radiation field to query and synthesize color and density, and integrates along the rays to obtain the rendered image. It constructs the photometric rendering loss between the rendered image and the real observed image, and jointly optimizes the first neural radiation field and the second neural radiation field, the weapon's six-degree-of-freedom B-spline function, and the deformable human body model. The analysis module is used to extract the optimal six-degree-of-freedom B-spline function of the weapon after the joint optimization has converged, and to calculate the zeroth, first and second derivatives to obtain the three-dimensional motion trajectory, attitude, linear velocity, angular velocity and acceleration of the weapon throughout the training process.

9. The system according to claim 8, characterized in that, The acquisition of multi-view synchronized video sequences and camera intrinsic and extrinsic parameters for the weapons training process includes: Eight to twelve high-speed industrial cameras with a frame rate of no less than 100fps were deployed in a ring around the training area, and the intrinsic and extrinsic parameters of all cameras were calculated using a checkerboard calibration board and a joint beam adjustment algorithm. A hardware synchronization controller ensures that the start timestamp error of all camera exposures is less than 1 millisecond.

10. The system according to claim 8, characterized in that, The process of decomposing the training scene into a static background field and a dynamic foreground field includes: Pixel-level masks of the trainee and weapon in each frame of the image are extracted using an instance segmentation network. During the joint optimization process, the photometric loss for optimizing the first neural radiation field is calculated based on the pixel region outside the mask, and the photometric loss for optimizing the second neural radiation field is calculated based on the pixel region covered by the mask.

Images