A real-time athlete running trajectory recording method based on multi-view splicing video

Abstract

The invention discloses a real-time athlete running trajectory recording method based on multi-eye splicing video, and specifically relates to the field of trajectory recording technology. The method obtains vibration information, thermal expansion information, illumination fluctuation information, and driver program response delay information of a camera, and outputs an internal and external parameter calibration optimization index according to the vibration deviation coefficient, thermal expansion deformation coefficient, illumination fluctuation coefficient, and driver program response delay coefficient. The method dynamically adjusts the initial internal and external parameters of the camera according to the internal and external parameter calibration optimization index, effectively copes with the camera parameter changes caused by factors such as vibration, ambient temperature changes, and illumination fluctuations, ensures the accuracy of spliced ​​images and motion trajectories, seamlessly splices images of multiple cameras through multi-eye splicing technology, generates a panoramic view covering the entire venue, detects and marks the enclosing frame of each athlete, and adopts a trajectory feature vector smoothing processing method based on five frames to generate a more stable and realistic bird's-eye view of the athlete's motion trajectory.

Description

Technical Field

[0001] The present invention relates to the technical field of track recording, and more specifically, to a real-time athlete running track recording method based on multi-view spliced ​​videos. Background Art

[0002] With the rapid development of intelligent video surveillance technology and computer vision technology, the athlete running trajectory recording method based on multi-view splicing video has been widely used in sports event analysis, tactical research, athlete training and other fields. This method installs multiple synchronous cameras on the sports field and uses the images taken by the cameras for real-time splicing to accurately track and record the athlete's movement trajectory. Adjacent cameras are calibrated relative to each other (internal and external parameters) to achieve image splicing and spatial reconstruction, thereby ensuring that the entire sports field can be fully covered and the athlete's position and movement in the field can be accurately obtained.

[0003] However, the system based on multi-eye stitching video faces a series of challenges in practical application. One of the most important issues is that the internal and external parameters of the camera may change due to various factors, thus affecting the accuracy of the stitched image and the accuracy of the athlete's trajectory. The internal and external parameters of the camera include internal parameters (such as focal length, principal point, distortion coefficient, etc.) and external parameters (such as camera pose, i.e. the position and direction of the camera in space). The accuracy of these parameters is directly related to the accuracy of image stitching and spatial reconstruction, while the vibration offset of the camera, the deformation of the lens, the aging of the optical components, the response delay of the driver, and the fluctuation of the lighting environment may cause the internal and external parameters to no longer be applicable, thus affecting the reliability of the athlete's trajectory recording.

[0004] In traditional multi-camera stitching systems, calibration is usually performed in advance and it is assumed that the camera parameters remain unchanged during the entire operation. However, in dynamic and complex application environments, such as camera shake on a sports field, lens focus adjustment, and ambient light changes, the pre-calibrated internal and external parameters will be affected. Therefore, how to monitor and adjust the camera's internal and external parameters in real time so that they always remain consistent with the actual scene has become a key issue in ensuring the accuracy of the multi-camera stitching video system. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a real-time athlete running trajectory recording method based on multi-view spliced ​​video to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A method for recording a real-time athlete's running trajectory based on multi-view spliced ​​videos comprises the following steps:

[0008] Step S1, setting multiple cameras based on the focal length of the selected camera and the size of the venue to ensure that the multi-camera covers the entire venue, and performing initial calibration of the internal and external parameters of the set cameras;

[0009] Step S2, obtain the camera's vibration information, thermal expansion information, illumination fluctuation information, and driver response delay information through the sensor network, and calculate the vibration deviation coefficient Zdp, thermal expansion deformation coefficient Rpz, illumination fluctuation coefficient Gzb, and driver response delay coefficient Ycx. Based on the above four information coefficients, an internal and external parameter calibration optimization model is constructed, and an internal and external parameter calibration optimization index nwcs is output to optimize the internal and external parameters of the initial camera calibration. The model formula is as follows In the formula, w 1 、w 2 、w 3 、w 4 They represent the preset proportional coefficients of vibration deviation coefficient, thermal expansion deformation coefficient, illumination fluctuation coefficient, and driver response delay coefficient, respectively, and w 1 、w 2 、w 3 、w 4 All are greater than 0;

[0010] Step S3, stitching the pictures taken by the multi-camera to obtain a panoramic view of the field, inputting the stitched panoramic view into the target detection neural network to detect the athletes in the picture and mark the bounding box of each athlete, and extracting the trajectory feature vector of each athlete as the unique identification of the athlete;

[0011] Step S4, obtaining trajectory feature vectors of a total of five frames, namely, the current frame and two frames before and after, from continuous video frames, and performing smoothing processing on the current frame, and generating a bird's-eye view of the athlete's motion trajectory based on the smoothed trajectory feature vectors.

[0012] In a preferred embodiment, the logic for obtaining the vibration deviation coefficient is as follows:

[0013] The acceleration data of the camera in three-dimensional space is obtained by the acceleration sensor a(t)=[a x (t),a y (t),a z (t)], where a x (t), a y (t), a z (t) represents the acceleration in the X-axis, Y-axis, and Z-axis directions respectively; the instantaneous acceleration amplitude of the vibration atotal(t) is calculated as follows Calculate the offset dtotal(t) caused by vibration. The expression is as follows Where t2-t1 represents the operating time of the camera; calculate the vibration deviation coefficient Zdp, the expression is as follows Where dmax represents the maximum deviation allowed by the camera during vibration.

[0014] In a preferred embodiment, the logic for obtaining the thermal expansion coefficient is as follows:

[0015] By deploying an ambient temperature sensor to obtain the temperature of the environment in which the camera is located, the thermal expansion deformation ΔLpart of the camera components is calculated. The expression is as follows: ΔLpart = L0*αpart*ΔT, where L0 represents the initial volume of the component, αpart represents the thermal expansion coefficient of the component, and ΔT is the temperature change based on the reference temperature;

[0016] Calculate the thermal expansion coefficient Rpz, the expression is as follows Where ΔLpart i represents the thermal expansion deformation of the i-th component of the camera, δ i Represents the preset proportional coefficient of the i-th component, i∈{1,2,...,I}, where I is a positive integer.

[0017] In a preferred embodiment, the logic for obtaining the illumination fluctuation coefficient is as follows:

[0018] The light intensity value of the camera lens is obtained through the light intensity sensor, and the light intensity change rate ΔL at adjacent moments is calculated t , the expression is as follows ΔL t =|L t+1 -L t |, where L t Represents the light intensity value of the camera lens at time t, L t+1 Represents the light intensity value of the camera lens at time t+1;

[0019] Calculate the mean μillum of the rate of change of light intensity, expressed as follows Where t∈{1,2,...,T}, T is a positive integer;

[0020] Calculate the standard deviation of the rate of change of light intensity σillum, the expression is as follows

[0021] Calculate the light fluctuation coefficient Gzb, the expression is as follows

[0022] In a preferred embodiment, the logic for obtaining the driver response delay coefficient is as follows:

[0023] The time interval between the camera receiving the control command and the start of image capture is obtained from the log records provided by the camera driver, and is used as the driver response time. The driver response time data set XY = {ΔTwin j}={ΔTwin 1 ,ΔTwin 2 ,...,ΔTwin J}, where ΔTwin j represents the driver response time obtained by the jth sampling, j∈{1,2,...,J}, J is a positive integer;

[0024] The driver response time is compared with the preset driver response time threshold. If the driver response time is greater than the driver response time threshold, it is added to the driver response delay data set YV = {ΔTwin m}={ΔTwin 1 ,ΔTwin 2 ,...,ΔTwin M}, where ΔTwin m represents the driver response time that is greater than the driver response time threshold for the mth time, m∈{1,2,...,M}, where M is a positive integer;

[0025] Calculate the mean μΔTwin of the driver response delay dataset, expressed as follows Where m∈{1,2,...,M}, M is a positive integer;

[0026] Calculate the standard deviation σΔTwin of the driver response delay dataset as follows

[0027] Calculate the driver response delay distribution f(ΔTwin m ) reconstructs the driver response delay dataset, and the expression is as follows

[0028] Calculate the driver response delay coefficient Ycx, the expression is as follows Where TCT represents the total available time of the CPU, TT represents the total number of tasks of the driver, and TP represents the priority of the shooting task.

[0029] In a preferred embodiment, the internal and external parameter calibration optimization index is compared with a preset internal and external parameter calibration optimization index threshold, and the deviation of the camera internal and external parameter calibration is classified to determine whether optimization is required, as follows:

[0030] If the internal and external parameter calibration optimization index is greater than the internal and external parameter calibration optimization index threshold, it is necessary to optimize the internal and external parameters of the initial camera calibration; if the internal and external parameter calibration optimization index is less than or equal to the internal and external parameter calibration optimization index threshold, it is not necessary to optimize the internal and external parameters of the initial camera calibration.

[0031] In a preferred embodiment, when the internal and external parameters of the initial calibration of the camera need to be optimized, the genetic expression programming algorithm is used to tune the internal and external parameters of the camera, as follows:

[0032] Step K1, gene encoding, the leaf nodes of each gene expression tree correspond to the internal and external parameters of the camera, and the non-leaf nodes are operators;

[0033] Step K2, initialize the population and generate an initial population, in which each individual is a gene expression tree, representing different internal and external parameter variables, and converts them into the form of gene expression by randomly initializing the internal and external parameter values ​​of the camera;

[0034] Step K3, evaluate the quality of each individual based on the fitness function. The larger the value of the fitness function, the better the optimization effect of the individual, that is, the better the optimization effect of the internal and external parameters of the camera. The function expression of the fitness function is as follows: Where fitness represents the output value of the fitness function, nwcs represents the internal and external parameter calibration optimization index, P g represents the coordinates of the real image point g obtained in the actual shooting scene, represents the coordinates of the predicted image g calculated by the initial internal and external parameters, g∈{1,2,...,G}, G is a positive integer;

[0035] Step K4, performing selection, crossover and mutation operations on gene expression individuals;

[0036] Step K5, iterative evolution: determine whether the iteration time reaches the preset iteration time threshold. If the iteration time is equal to the iteration time threshold, the iteration process ends and the optimal solution individual is output. If the iteration time is less than the iteration time threshold, proceed to step K3.

[0037] Step K6, decode the optimal solution individual through in-order traversal to obtain the camera's internal and external parameter tuning variables.

[0038] Technical effects and advantages of the present invention:

[0039] The present invention obtains the vibration information, thermal expansion information, illumination fluctuation information, and driver program response delay information of the camera through a sensor network, and constructs an internal and external parameter calibration optimization model according to the vibration deviation coefficient, thermal expansion deformation coefficient, illumination fluctuation coefficient, and driver program response delay coefficient, outputs an internal and external parameter calibration optimization index, and dynamically adjusts the initial internal and external parameters of the camera according to the internal and external parameter calibration optimization index, effectively responds to the camera parameter changes caused by factors such as vibration, ambient temperature changes, and illumination fluctuations, ensures the accuracy of spliced ​​images and motion trajectories, seamlessly splices the images of multiple cameras through multi-eye splicing technology, generates a panoramic view covering the entire venue, and then inputs it into an efficient target detection neural network, detects and marks the encircling frame of each athlete in real time, and extracts the athlete's trajectory feature vector accurately and uniquely, providing a strong foundation for subsequent motion trajectory analysis, adopts a trajectory feature vector smoothing processing method based on five frames, eliminates the influence of factors such as camera shake, false detection or image noise on the motion trajectory, and generates a more stable and real athlete motion trajectory overhead view. The smoothing process can improve the accuracy of the trajectory, make the athlete's motion path smoother and more coherent, thereby providing high-quality data support for subsequent analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings;

[0041] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Example: Figure 1 The present invention provides a method for recording a real-time athlete's running trajectory based on multi-eye spliced ​​video, comprising the following steps:

[0044] Step S1, setting multiple cameras based on the focal length of the selected camera and the size of the venue to ensure that the multi-camera covers the entire venue, and performing initial calibration of the internal and external parameters of the set cameras;

[0045] Step S2, obtain the camera's vibration information, thermal expansion information, illumination fluctuation information, and driver response delay information through the sensor network, and calculate the vibration deviation coefficient Zdp, thermal expansion deformation coefficient Rpz, illumination fluctuation coefficient Gzb, and driver response delay coefficient Ycx. Based on the above four information coefficients, an internal and external parameter calibration optimization model is constructed, and an internal and external parameter calibration optimization index nwcs is output to optimize the internal and external parameters of the initial camera calibration. The model formula is as follows In the formula, w 1 、w 2 、w 3 、w 4 They represent the preset proportional coefficients of vibration deviation coefficient, thermal expansion deformation coefficient, illumination fluctuation coefficient, and driver response delay coefficient, respectively, and w 1 、w 2 、w 3 、w 4 All are greater than 0;

[0046] Step S3, stitching the pictures taken by the multi-camera to obtain a panoramic view of the field, inputting the stitched panoramic view into the target detection neural network to detect the athletes in the picture and mark the bounding box of each athlete, and extracting the trajectory feature vector of each athlete as the unique identification of the athlete;

[0047] Step S4, obtaining trajectory feature vectors of a total of five frames, namely, a current frame and two frames before and after the current frame, from continuous video frames, and performing smoothing processing on the current frame, and generating a bird's-eye view of the athlete's motion trajectory based on the smoothed trajectory feature vectors;

[0048] Step S1, setting multiple cameras based on the focal length of the selected camera and the size of the venue to ensure that the multi-camera covers the entire venue, and performing initial calibration of the internal and external parameters of the set cameras;

[0049] The present invention firstly reasonably configures multiple cameras according to the size of the venue and the focal length requirements of the selected cameras to ensure that the coverage of the camera's field of view can fully cover the entire sports venue. The purpose of configuring multiple cameras is to eliminate the dead angles in the sports venue to the greatest extent through the collaborative work and image stitching between the cameras, so as to achieve all-round and blind spot-free athlete trajectory capture; the size of the venue directly affects the number of cameras required and the focal length of each camera. For larger venues, the field of view of a single camera may not be enough to cover the entire area, so it is necessary to increase the number of cameras and select lenses with longer focal lengths to ensure that each camera can cover a certain range of areas. The focal length of the camera determines the field of view of the shooting. The longer the focal length, the smaller the field of view of the camera, but the clearer the details. In order to ensure that the entire venue is covered, multiple cameras usually need to be arranged at different positions of the venue, such as the edge, corner or center line of the venue, etc., to ensure that there is a certain overlapping field of view between the cameras, so that adjacent cameras can share information in the same scene, thereby achieving effective stitching and calibration;

[0050] In the initial calibration of the internal and external parameters of the set camera, the checkerboard calibration method is used to calibrate the internal parameters of the camera, and the external parameters are initially calibrated by setting a common calibration object in the camera field of view and matching the feature points of the calibration objects in the field of view of multiple cameras;

[0051] It should be noted that when there are players on the field, the players themselves in the intersection area between cameras can be used as calibration objects;

[0052] Step S2, obtaining the camera's vibration information, thermal expansion information, illumination fluctuation information, and driver program response delay information through the sensor network, constructing an internal and external parameter calibration optimization model, outputting an internal and external parameter calibration optimization index, and optimizing the internal and external parameters of the initial calibration of the camera;

[0053] Calculate the vibration deviation coefficient Zdp based on the camera's vibration information;

[0054] The vibration deviation coefficient is used to measure the degree of deviation of the camera's position due to vibration. In a dynamic environment, such as a sports venue, the camera's vibration may come from multiple factors, such as venue vibration, equipment movement, athlete running, spectator interference, etc. The vibration deviation coefficient can help the system dynamically evaluate the stability of the camera and make corrections during the real-time calibration process;

[0055] The vibration deviation coefficient can be used as an important basis for optimization calibration. Its main functions are reflected in the following aspects:

[0056] Real-time feedback of calibration error: When the vibration deviation coefficient exceeds the preset threshold, it means that the camera calibration may have deviated. At this time, the internal and external parameters of the camera can be corrected through the optimization algorithm to ensure the accuracy of the athlete's trajectory recording.

[0057] Dynamically adjust camera parameters: Based on the vibration deviation coefficient, the system can dynamically adjust the camera's focal length, posture and other internal parameters, and perform local calibration optimization to reduce the impact of vibration on camera image quality.

[0058] Athlete trajectory accuracy control: The vibration deviation coefficient will affect the positioning accuracy of the camera. Therefore, by controlling the vibration deviation coefficient, it can be ensured that the system can still accurately track the athlete's position when the vibration is large.

[0059] Therefore, by calculating the vibration deviation coefficient, the impact of vibration on the camera position can be quantified, providing a more accurate correction basis for camera calibration, thereby ensuring that the multi-eye stitching video system continues to provide high-precision athlete trajectory records in dynamic sports scenes.

[0060] The logic for obtaining the vibration deviation coefficient is as follows:

[0061] The acceleration data of the camera in three-dimensional space is obtained by the acceleration sensor a(t)=[a x (t),a y (t),a z (t)], where a x (t), a y (t), a z (t) represents the acceleration in the X-axis, Y-axis, and Z-axis directions respectively; the instantaneous acceleration amplitude of the vibration atotal(t) is calculated as follows Calculate the offset dtotal(t) caused by vibration. The expression is as follows Where t2-t1 represents the operating time of the camera; calculate the vibration deviation coefficient Zdp, the expression is as follows Where dmax represents the maximum deviation allowed by the camera during vibration;

[0062] Calculate the thermal expansion deformation coefficient Rpz based on the thermal expansion information of the camera;

[0063] The thermal expansion coefficient is used to measure the impact of ambient temperature changes on the deformation of camera materials. Especially in the real-time athlete running trajectory recording system of multi-eye stitching video, temperature fluctuations may cause the camera's lens, sensor, body and other components to expand or contract, thereby affecting the camera's internal and external parameters, resulting in calibration errors, and ultimately affecting the accurate recording of the athlete's trajectory.

[0064] The thermal expansion coefficient can provide an important basis for optimizing the internal and external parameters of the camera. In real-time trajectory recording, temperature changes may cause slight deformation of the camera lens or sensor, which in turn affects the camera's field of view and shooting accuracy. Therefore, the thermal expansion coefficient can be applied to the following aspects:

[0065] Dynamic calibration optimization: Based on the calculated thermal expansion coefficient, the system can adjust the camera’s internal and external parameters in real time, especially in environments with large temperature fluctuations, to avoid image distortion or stitching errors caused by expansion effects.

[0066] Temperature monitoring: By monitoring the camera’s thermal expansion coefficient in real time, the system can respond to changes in ambient temperature, providing early warning of possible calibration deviations and ensuring the accuracy of trajectory recording.

[0067] Long-term stability testing: The thermal expansion coefficient can also be used to evaluate the thermal effects that may occur during long-term use of the camera, helping with equipment maintenance and replacement.

[0068] Therefore, by calculating and optimizing the thermal expansion coefficient, the negative impact of ambient temperature changes on camera performance can be effectively reduced, ensuring the accuracy and stability of the multi-eye stitching system in complex environments.

[0069] The logic for obtaining the thermal expansion coefficient is as follows:

[0070] By deploying an ambient temperature sensor to obtain the temperature of the environment in which the camera is located, the thermal expansion deformation ΔLpart of the camera components is calculated. The expression is as follows: ΔLpart = L0*αpart*ΔT, where L0 represents the initial volume of the component, αpart represents the thermal expansion coefficient of the component, and ΔT is the temperature change based on the reference temperature;

[0071] Calculate the thermal expansion coefficient Rpz, the expression is as follows Where ΔLpart i represents the thermal expansion deformation of the i-th component of the camera, δ i represents the preset proportional coefficient of the i-th component, i∈{1,2,...,I}, where I is a positive integer;

[0072] It should be noted that the components of a camera include lenses, sensor boards, and bodies, and the materials of different components have their own specific thermal expansion coefficients. The thermal expansion coefficients of materials can usually be found in the material science database. For example, the expansion coefficients of materials such as metals, plastics, glass, and ceramics are different; δ i Set it according to the actual situation. For example, adopt the expert empowerment method, that is, invite experts in related fields to determine the preset proportion coefficients of various indicators through professional opinion surveys and comprehensive evaluations;

[0073] Calculate the illumination fluctuation coefficient Gzb based on the illumination fluctuation information of the camera;

[0074] The light fluctuation coefficient is used to measure the impact of the fluctuation of ambient light intensity on the camera's internal and external parameters. Especially in dynamic scenes, changes in lighting conditions may affect the camera's exposure settings, image contrast, and detail capture capabilities, thereby affecting the accuracy of image stitching, target detection, and athlete trajectory recording. Therefore, real-time monitoring and calculation of the light fluctuation coefficient is crucial.

[0075] The illumination fluctuation coefficient can provide important information for optimizing the camera's internal and external parameters. The specific applications are as follows:

[0076] Automatic exposure adjustment: Based on the light fluctuation coefficient, the system can determine the light stability of the current environment. If the fluctuation is large, the system can adjust the camera's exposure settings (such as shutter speed, gain, etc.) as needed to reduce the impact of light fluctuations.

[0077] Internal and external parameter optimization: Light fluctuations affect the brightness, contrast, and clarity of camera images, which may cause distortion during stitching or reduce the accuracy of target detection. The light fluctuation coefficient can be used to adjust the parameters in the stitching algorithm, optimize the image synthesis effect, and avoid stitching errors caused by exposure changes.

[0078] Image quality assessment: The illumination fluctuation coefficient can also be used as one of the evaluation indicators of image quality. If the illumination fluctuation is too large, it may cause uneven image brightness, affect image quality, and thus affect the extraction and display of athlete trajectories.

[0079] Dynamic correction: In the multi-camera stitching system, the illumination fluctuation coefficient can be used as a basis for real-time adjustment of the camera's internal and external parameters. When the illumination fluctuation coefficient reaches a certain threshold, the system can trigger a dynamic correction process to recalibrate the camera's position and posture to ensure the accuracy of image stitching and the accuracy of the athlete's trajectory.

[0080] Therefore, by real-time monitoring of lighting changes and calculating the lighting fluctuation coefficient, we can provide a basis for optimizing the internal and external parameters of the camera, ensure that the camera system can work stably under different lighting conditions, and ensure the accuracy of the athlete's running trajectory. By calculating and optimizing the lighting fluctuation coefficient, we can improve the performance of the multi-eye splicing video system in complex lighting environments and ensure the accuracy and availability of the motion trajectory.

[0081] The logic for obtaining the light fluctuation coefficient is as follows:

[0082] The light intensity value of the camera lens is obtained through the light intensity sensor, and the light intensity change rate ΔL at adjacent moments is calculated t , the expression is as follows ΔL t =|L t+1 -L t |, where L t Represents the light intensity value of the camera lens at time t, Lt+1 Represents the light intensity value of the camera lens at time t+1;

[0083] Calculate the mean μillum of the rate of change of light intensity, expressed as follows Where t∈{1,2,...,T}, T is a positive integer;

[0084] Calculate the standard deviation of the rate of change of light intensity σillum, the expression is as follows

[0085] Calculate the light fluctuation coefficient Gzb, the expression is as follows

[0086] Calculate the driver response delay coefficient Ycx according to the driver response delay information of the camera;

[0087] The driver response delay coefficient is used to measure the delay from the camera driver receiving the control command to the actual start of image acquisition or data processing. The driver response delay may be caused by many factors, such as hardware interface delay, operating system scheduling, program internal processing, etc. These delays may have an adverse effect on applications with high real-time requirements (such as multi-eye splicing video, athlete running track recording, etc.). Therefore, calculating and optimizing the driver response delay coefficient is crucial to improving the real-time performance and accuracy of the system.

[0088] The driver response delay coefficient can provide important information for optimizing the camera's internal and external parameters. The specific applications are as follows:

[0089] Optimize system real-time performance: By calculating the driver response delay coefficient, the real-time performance of the system can be evaluated. If the coefficient is large, the camera driver may need to be optimized to reduce delay fluctuations and ensure real-time recording of the athlete's trajectory.

[0090] Dynamically adjust the control strategy: In the case of large delays, the system may need to dynamically adjust the control strategy, for example, by reducing the frequency of image acquisition or optimizing other delay parameters to compensate for the impact of response delay on system real-time performance.

[0091] Time synchronization when calibrating internal and external parameters: Driver response delay may cause time synchronization problems during image acquisition. By calculating the driver response delay coefficient, the system can adjust the time synchronization between multiple cameras to ensure the timing consistency of image acquisition data, thereby improving the stitching quality and the accuracy of the athlete's trajectory.

[0092] Multi-eye stitching and target detection optimization: If the driver response delay coefficient is large, the inconsistency of image acquisition time may affect the stitching effect. By real-time monitoring of the delay coefficient, the system can dynamically adjust during the stitching process, optimize the image fusion algorithm, and reduce the impact of delay on athlete trajectory extraction.

[0093] The logic for obtaining the driver response delay coefficient is as follows:

[0094] The time interval between the camera receiving the control command and the start of image capture is obtained from the log records provided by the camera driver, and is used as the driver response time. The driver response time data set XY = {ΔTwin j}={ΔTwin 1 ,ΔTwin 2 ,...,ΔTwin J}, where ΔTwin j represents the driver response time obtained by the jth sampling, j∈{1,2,...,J}, J is a positive integer;

[0095] The driver response time is compared with the preset driver response time threshold. If the driver response time is greater than the driver response time threshold, it is added to the driver response delay data set YV = {ΔTwin m}={ΔTwin 1 ,ΔTwin 2 ,...,ΔTwin M}, where ΔTwin m represents the driver response time that is greater than the driver response time threshold for the mth time, m∈{1,2,...,M}, where M is a positive integer;

[0096] Calculate the mean μΔTwin of the driver response delay dataset, expressed as follows Where m∈{1,2,...,M}, M is a positive integer;

[0097] Calculate the standard deviation σΔTwin of the driver response delay dataset as follows

[0098] Calculate the driver response delay distribution f(ΔTwin m ) reconstructs the driver response delay dataset, and the expression is as follows

[0099] Calculate the driver response delay coefficient Ycx, the expression is as follows Where TCT represents the total available time of the CPU, TT represents the total number of tasks of the driver, and TP represents the priority of the shooting task;

[0100] It should be noted that the priority of the shooting task is represented by the order of the shooting task in the task sorting queue;

[0101] According to the vibration deviation coefficient, thermal expansion deformation coefficient, illumination fluctuation coefficient, and driver response delay coefficient, an internal and external parameter calibration optimization model is constructed to output the internal and external parameter calibration optimization index nwcs. The formula based on the model is as follows In the formula, w 1 、w 2 、w 3 、w 4 They represent the preset proportional coefficients of vibration deviation coefficient, thermal expansion deformation coefficient, illumination fluctuation coefficient, and driver response delay coefficient, respectively, and w 1 、w 2 、w 3 、w 4 All are greater than 0;

[0102] It should be noted that before constructing the internal and external parameter calibration optimization model, it is necessary to ensure that the vibration deviation coefficient, thermal expansion deformation coefficient, light fluctuation coefficient, and driver response delay coefficient are all normalized; 1 、w 2 、w 3 、w 4 Set it according to the actual situation. For example, adopt the expert empowerment method, that is, invite experts in related fields to determine the preset proportion coefficients of various indicators through professional opinion surveys and comprehensive evaluations;

[0103] From the above calculation expression, it can be seen that the larger the vibration deviation coefficient, the larger the thermal expansion deformation coefficient, the larger the illumination fluctuation coefficient, and the larger the driver response delay coefficient, the larger the internal and external parameter calibration optimization index, and the more serious the factors affecting the accuracy of the internal and external parameter calibration (vibration, thermal expansion, illumination fluctuation, and driver response delay) are, which may lead to a large deviation in the camera's internal and external parameter calibration, and adjustment and optimization are required. On the contrary, the smaller the vibration deviation coefficient, the smaller the thermal expansion deformation coefficient, the smaller the illumination fluctuation coefficient, and the smaller the driver response delay coefficient, the smaller the internal and external parameter calibration optimization index, which means that the factors affecting the camera's internal and external parameters are small, and the current calibration accuracy is high, and no much adjustment is required;

[0104] Compare the internal and external calibration optimization index with the preset internal and external calibration optimization index threshold, and classify the deviation of the camera's internal and external calibration to determine whether optimization is required, as follows:

[0105] If the internal and external parameter calibration optimization index is greater than the internal and external parameter calibration optimization index threshold, it indicates that the initial calibrated internal and external parameters are no longer applicable, and the deviation of the camera's internal and external parameter calibration is large, and the camera's initial calibrated internal and external parameters need to be optimized; if the internal and external parameter calibration optimization index is less than or equal to the internal and external parameter calibration optimization index threshold, it indicates that the initial calibrated internal and external parameters and the camera's operating status are good and can meet the current shooting requirements, and there is no need to optimize the camera's initial calibrated internal and external parameters;

[0106] When it is necessary to optimize the internal and external parameters of the initial camera calibration, the genetic expression programming algorithm is used to tune the internal and external parameters of the camera, as follows:

[0107] Step K1, gene encoding, the leaf nodes of each gene expression tree correspond to the internal and external parameters of the camera (such as focal length, principal point coordinates, rotation matrix and translation vector, etc.), and the non-leaf nodes are operators (such as addition, multiplication, etc.);

[0108] Step K2, initialize the population and generate an initial population, in which each individual is a gene expression tree, representing different internal and external parameter variables, and converts them into the form of gene expression by randomly initializing the internal and external parameter values ​​of the camera;

[0109] Step K3, evaluate the quality of each individual based on the fitness function. The larger the value of the fitness function, the better the optimization effect of the individual, that is, the better the optimization effect of the internal and external parameters of the camera. The function expression of the fitness function is as follows: Where fitness represents the output value of the fitness function, nwcs represents the internal and external parameter calibration optimization index, P g represents the coordinates of the real image point g obtained in the actual shooting scene, represents the coordinates of the predicted image g calculated by the initial internal and external parameters, g∈{1,2,...,G}, G is a positive integer;

[0110] Step K4, perform selection, crossover and mutation operations on gene expression individuals. The selection operation is the core of gene expression programming, and its purpose is to select individuals with high fitness from the current population for reproduction. Common selection methods include roulette selection, tournament selection, etc.; crossover operation is an important operation in the algorithm, which simulates the gene exchange process. Crossover operation steps: Randomly select two parent individuals from the population. Randomly select the crossover point and exchange the subtrees of the parent gene expression tree. Generate two new offspring individuals. The purpose of crossover is to produce offspring individuals that may have higher fitness by combining the advantages of the parents; mutation operation is to increase the diversity of the population through small gene mutations. Mutation operation steps: Randomly select an individual. Randomly change the operator or constant of a node in the gene expression of the individual. Generate new offspring individuals. Mutation operation helps to explore the solution space and prevent the algorithm from falling into the local optimal solution;

[0111] Step K5, iterative evolution: determine whether the iteration time reaches the preset iteration time threshold. If the iteration time is equal to the iteration time threshold, the iteration process ends and the optimal solution individual is output. If the iteration time is less than the iteration time threshold, proceed to step K3.

[0112] It should be noted that the iteration time threshold can be set according to actual conditions. For example, the iteration time threshold is set to 1s, that is, when the iteration time is equal to 1s, the iteration process ends;

[0113] Step K6, decoding the optimal solution individual through in-order traversal to obtain the camera's internal and external parameter tuning variables;

[0114] Step S3, stitching the pictures taken by the multi-camera to obtain a panoramic view of the field, inputting the stitched panoramic view into the target detection neural network to detect the athletes in the picture and mark the bounding box of each athlete, and extracting the trajectory feature vector of each athlete as the unique identification of the athlete;

[0115] Unify the images of each camera into a global coordinate system based on the internal and external parameters of the camera. Use image stitching algorithms, such as panorama stitching or image transformation matrix (homography). By calculating the transformation matrix between images, multiple camera images are seamlessly stitched into a complete field panorama. The stitched panorama will cover the entire field, ensuring that each area of ​​the image contains relevant athlete information, providing sufficient viewing angle for target detection.

[0116] Input the stitched panorama into the trained object detection neural network. Commonly used object detection networks include YOLO (You Only Look Once), SSD (Single Shot Multibox Detector) or Faster R-CNN (Region Convolutional Neural Network);

[0117] In this embodiment, the YOLOv8 model is used for target detection. It is an efficient and accurate target detection model that can quickly identify athletes in real-time scenarios;

[0118] The YOLOv8 model transforms object detection into a regression problem by gridding the image.

[0119] Each bounding box output by the model contains: the location coordinates of the athlete (the upper left and lower right corners of the box), the athlete category (classification of athlete and non-athlete);

[0120] For each frame of the panorama, the YOLOv8 model generates a bounding box for each athlete, including the size, position, and category of the athlete.

[0121] After detecting the bounding box of the athlete, the trajectory feature vector of each athlete is extracted based on the Re-ID network. The trajectory feature vector includes appearance features, the coordinates of the center point of the bounding box, and the motion posture features.

[0122] The trajectory feature vector of each athlete is used as the unique identifier of the athlete;

[0123] Step S4, obtaining trajectory feature vectors of a total of five frames, namely, a current frame and two frames before and after the current frame, from continuous video frames, and performing smoothing processing on the current frame, and generating a bird's-eye view of the athlete's motion trajectory based on the smoothed trajectory feature vectors;

[0124] From the continuous video frames, we obtain the trajectory feature vector set of the current frame and the two frames before and after, a total of five frames, denoted as V t-2 、V t-1 、V t 、V t+1 、V t+2 , where V t Represents the trajectory feature vector of the current frame;

[0125] Calculate the average of the trajectory feature vectors of the five frames before and after to obtain the smoothed trajectory feature vector of the current frame:

[0126] Generate a bird's-eye view of the athlete's motion trajectory based on the smoothed trajectory feature vector;

[0127] The present invention obtains the vibration information, thermal expansion information, illumination fluctuation information, and driver program response delay information of the camera through a sensor network, and constructs an internal and external parameter calibration optimization model according to the vibration deviation coefficient, thermal expansion deformation coefficient, illumination fluctuation coefficient, and driver program response delay coefficient, outputs an internal and external parameter calibration optimization index, and dynamically adjusts the initial internal and external parameters of the camera according to the internal and external parameter calibration optimization index, effectively responds to the camera parameter changes caused by factors such as vibration, ambient temperature changes, and illumination fluctuations, ensures the accuracy of spliced ​​images and motion trajectories, seamlessly splices the images of multiple cameras through multi-eye splicing technology, generates a panoramic view covering the entire venue, and then inputs it into an efficient target detection neural network, detects and marks the encircling frame of each athlete in real time, and extracts the athlete's trajectory feature vector accurately and uniquely, providing a strong foundation for subsequent motion trajectory analysis, adopts a trajectory feature vector smoothing processing method based on five frames, eliminates the influence of factors such as camera shake, false detection or image noise on the motion trajectory, and generates a more stable and real athlete motion trajectory overhead view. The smoothing process can improve the accuracy of the trajectory, make the athlete's motion path smoother and more coherent, thereby providing high-quality data support for subsequent analysis.

[0128] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0129] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0130] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for recording a real-time athlete's running trajectory based on multi-view spliced ​​video, characterized in that: The steps include: Step S1, setting multiple cameras based on the focal length of the selected camera and the size of the venue to ensure that the multi-camera covers the entire venue, and performing initial calibration of the internal and external parameters of the set cameras; Step S2, obtain the camera's vibration information, thermal expansion information, illumination fluctuation information, and driver response delay information through the sensor network, and calculate the vibration deviation coefficient Zdp, thermal expansion deformation coefficient Rpz, illumination fluctuation coefficient Gzb, and driver response delay coefficient Ycx. Based on the above four information coefficients, an internal and external parameter calibration optimization model is constructed, and an internal and external parameter calibration optimization index nwcs is output to optimize the internal and external parameters of the initial camera calibration. The model formula is as follows Wherein, w1, w2, w3, and w4 represent preset proportional coefficients of vibration deviation coefficient, thermal expansion deformation coefficient, light fluctuation coefficient, and driver response delay coefficient, respectively, and w1, w2, w3, and w4 are all greater than 0; Step S3, stitching the pictures taken by the multi-camera to obtain a panoramic view of the field, inputting the stitched panoramic view into the target detection neural network to detect the athletes in the picture and mark the bounding box of each athlete, and extracting the trajectory feature vector of each athlete as the unique identification of the athlete; Step S4, obtaining trajectory feature vectors of a total of five frames, namely, a current frame and two frames before and after the current frame, from continuous video frames, and performing smoothing processing on the current frame, and generating a bird's-eye view of the athlete's motion trajectory based on the smoothed trajectory feature vectors; Compare the internal and external calibration optimization index with the preset internal and external calibration optimization index threshold, and classify the deviation of the camera's internal and external calibration to determine whether optimization is required, as follows: If the internal and external parameter calibration optimization index is greater than the internal and external parameter calibration optimization index threshold, it is necessary to optimize the internal and external parameters of the camera's initial calibration; if the internal and external parameter calibration optimization index is less than or equal to the internal and external parameter calibration optimization index threshold, it is not necessary to optimize the internal and external parameters of the camera's initial calibration; When it is necessary to optimize the internal and external parameters of the initial camera calibration, the genetic expression programming algorithm is used to tune the internal and external parameters of the camera, as follows: Step K1, gene encoding, the leaf nodes of each gene expression tree correspond to the internal and external parameters of the camera, and the non-leaf nodes are operators; Step K2, initialize the population and generate an initial population, in which each individual is a gene expression tree, representing different internal and external parameter variables, and converts them into the form of gene expression by randomly initializing the internal and external parameter values ​​of the camera; Step K3, evaluate the quality of each individual based on the fitness function. The larger the value of the fitness function, the better the optimization effect of the individual, that is, the better the optimization effect of the internal and external parameters of the camera. The function expression of the fitness function is as follows: Where fitness represents the output value of the fitness function, nwcs represents the internal and external parameter calibration optimization index, P g represents the coordinates of the real image point g obtained in the actual shooting scene, represents the coordinates of the predicted image g calculated by the initial internal and external parameters, g∈{1,2,...,G}, G is a positive integer; Step K4, performing selection, crossover and mutation operations on gene expression individuals; Step K5, iterative evolution: determine whether the iteration time reaches the preset iteration time threshold. If the iteration time is equal to the iteration time threshold, the iteration process ends and the optimal solution individual is output. If the iteration time is less than the iteration time threshold, proceed to step K3. Step K6, decode the optimal solution individual through in-order traversal to obtain the camera's internal and external parameter tuning variables.

2. A method for recording a real-time athlete's running trajectory based on multi-eye spliced ​​video according to claim 1, characterized in that: The logic for obtaining the vibration deviation coefficient is as follows: The acceleration data of the camera in three-dimensional space is obtained by the acceleration sensor a(t)=[a x (t),a y (t),a z (t)], where a x (t), a y (t), a z (t) represents the acceleration in the X-axis, Y-axis, and Z-axis directions respectively; the instantaneous acceleration amplitude of the vibration atotal(t) is calculated as follows Calculate the offset dtotal(t) caused by vibration. The expression is as follows Where t2-t1 represents the operating time of the camera; Calculate the vibration deviation coefficient Zdp, the expression is as follows Where dmax represents the maximum deviation allowed by the camera during vibration.

3. A method for recording a real-time athlete's running trajectory based on multi-eye spliced ​​video according to claim 1, characterized in that: The logic for obtaining the thermal expansion coefficient is as follows: By deploying an ambient temperature sensor to obtain the temperature of the environment in which the camera is located, the thermal expansion deformation ΔLpart of the camera components is calculated. The expression is as follows: ΔLpart = L0*αpart*ΔT, where L0 represents the initial volume of the component, αpart represents the thermal expansion coefficient of the component, and ΔT is the temperature change based on the reference temperature; Calculate the thermal expansion coefficient Rpz, the expression is as follows Where ΔLpart i represents the thermal expansion deformation of the i-th component of the camera, δ i Represents the preset proportional coefficient of the i-th component, i∈{1,2,...,I}, where I is a positive integer.

4. A method for recording a real-time athlete's running trajectory based on multi-eye spliced ​​video according to claim 1, characterized in that: The logic for obtaining the light fluctuation coefficient is as follows: The light intensity value of the camera lens is obtained through the light intensity sensor, and the light intensity change rate ΔL at adjacent moments is calculated t , the expression is as follows ΔL t =|L t+1 -L t |, where L t Represents the light intensity value of the camera lens at time t, L t+1 Represents the light intensity value of the camera lens at time t+1; Calculate the mean μillum of the rate of change of light intensity, expressed as follows Where t∈{1,2,...,T}, T is a positive integer; Calculate the standard deviation of the rate of change of light intensity σillum, the expression is as follows Calculate the light fluctuation coefficient Gzb, the expression is as follows 5. A method for recording a real-time athlete's running trajectory based on multi-eye spliced ​​video according to claim 1, characterized in that: The logic for obtaining the driver response delay coefficient is as follows: The time interval between the camera receiving the control command and the start of image capture is obtained from the log records provided by the camera driver, and is used as the driver response time. The driver response time data set XY = {ΔTwin j }={ΔTwin1,ΔTwin2,...,ΔTwin J }, where ΔTwin j represents the driver response time obtained by the jth sampling, j∈{1,2,...,J}, J is a positive integer; The driver response time is compared with the preset driver response time threshold. If the driver response time is greater than the driver response time threshold, it is added to the driver response delay data set YV = {ΔTwin m }={ΔTwin1,ΔTwin2,...,ΔTwin M }, where ΔTwin m represents the driver response time that is greater than the driver response time threshold for the mth time, m∈{1,2,...,M}, where M is a positive integer; Calculate the mean μΔTwin of the driver response delay dataset, expressed as follows Where m∈{1,2,...,M}, M is a positive integer; Calculate the standard deviation σΔTwin of the driver response delay dataset as follows Calculate the driver response delay distribution f(ΔTwin m ) reconstructs the driver response delay dataset, and the expression is as follows Calculate the driver response delay coefficient Ycx, the expression is as follows Where TCT represents the total available time of the CPU, TT represents the total number of tasks of the driver, and TP represents the priority of the shooting task.

Images