Image recognition processing method and system fusing time information

Abstract

The application relates to the technical field of image recognition and discloses an image recognition processing method and system fusing time information, which directly stamps by hardware synchronization triggering and exposure time, ensures that an image sequence has uniform frame intervals and a microsecond-level precision exposure starting time stamp from the source, and provides accurate timing data basis for subsequent processing. On this basis, time intervals and exposure time are converted into time encoding deeply coupled with a time Transformer model, so that the attention to timing information can be adaptively adjusted according to scene content, thereby greatly improving the recognition accuracy in a complex dynamic scene. In addition, the space-time representation strictly aligned with the model learning and the physical time scale not only improves the physical authenticity of motion estimation and prediction, but also provides reliable support for directly measuring physical quantities such as speed and acceleration in an image sequence.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more specifically to an image recognition processing method and system that integrates time information. Background Technology

[0002] In visual tasks such as target recognition, tracking, and behavior analysis based on image sequences, it is necessary not only to identify the visual features of the target but also to accurately understand its motion state and spatiotemporal relationships. Traditional image acquisition systems often use software triggering, where the host program sends acquisition commands and a system timestamp is added after the image data transmission is complete. The timestamp recorded in this method is not the actual moment the image sensor begins exposure, but rather the moment the data arrives at the processing unit. There are unpredictable and fluctuating communication and system scheduling delays between the two. Furthermore, the rhythm of software triggering is easily affected by operating system load, leading to uneven acquisition frame intervals and time jitter.

[0003] The temporal inaccuracies caused by the aforementioned situations pose a fundamental challenge to subsequent image recognition processing. Whether it's classic state estimation based on Kalman filtering or temporal modeling based on deep learning models, they typically implicitly assume uniform temporal sequence or accurate timestamps. When the actual physical temporal sequence of the input data does not match the model assumptions, it leads to problems such as motion estimation bias, tracking drift, and misjudgment of temporal relationships, severely limiting application performance in demanding scenarios such as high-precision measurement, high-speed process analysis, and multi-sensor fusion. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide an image recognition processing method and system that integrates time information, so as to solve the technical problem that the image recognition results of the model are inaccurate due to the inaccuracy of the timing of the image sequence input to the model and the inconsistency with the model assumptions.

[0005] The first aspect of this invention discloses an image recognition processing method that integrates time information, the method comprising:

[0006] Obtain the image sequence and its associated timestamp; the timestamp associated with each frame of the image sequence is the exposure start timestamp, and the time intervals between frames of the image sequence are uniform;

[0007] Based on the exposure start timestamp, the precise time interval ΔT between adjacent frames in the image sequence is determined;

[0008] Based on the precise time interval ΔT and the exposure start timestamp, a time-coded feature is generated;

[0009] The temporal coding features are fused with the visual features extracted from the image sequence to obtain the fused features;

[0010] The deep neural network model outputs image recognition results based on the fused features.

[0011] Furthermore, the process of generating time-coded features includes:

[0012] Based on the exposure start timestamp of each frame of the image, a periodic time position encoding vector is generated using sine and cosine functions;

[0013] The frequency basis parameters of the sine and cosine functions are configured according to the precise time interval ΔT. After configuration, the frequency spectrum of the time position encoding vector changes over the entire time range of the image sequence is inversely proportional to ΔT.

[0014] Furthermore, the process of generating time-coded features also includes:

[0015] A relative time embedding vector is generated based on the precise time interval ΔT between multiple consecutive frames and stored using a learnable lookup table. The learnable lookup table is indexed by the number of inter-frame displacement steps, which is obtained by dividing the time difference by the precise time interval ΔT and rounding it down.

[0016] Furthermore, the deep neural network model is a spatiotemporal Transformer model; the generation process of the fused features includes:

[0017] The temporal location encoding vector is added element by element to the image patch embedding vector to form a spatiotemporal embedding sequence;

[0018] The spatiotemporal embedding sequence is input into the multilayer encoder of the spatiotemporal Transformer model. A temporal bias matrix is ​​added to the dot product of the query vector and the key vector in the self-attention mechanism of the multilayer encoder. The temporal bias matrix is ​​constructed based on the relative temporal embedding vector and is used to characterize the temporal correlation strength between different image patches.

[0019] The process of generating the image patch embedding vector includes:

[0020] Before fusing the temporal coding features with the visual features extracted from the image sequence, each frame image is divided into multiple image blocks, and each image block is converted into an embedding vector through linear projection to obtain the image block embedding vector.

[0021] Furthermore, before constructing the time bias matrix, the method further includes:

[0022] Based on the spatiotemporal embedding sequence, a dynamic gating coefficient is generated at each query position in the sequence;

[0023] After generating the dynamic gating coefficients, the construction process of the time bias matrix specifically includes:

[0024] Construct the initial time bias matrix based on the relative time embedding vector;

[0025] The initial time bias matrix is ​​modulated in the row direction using the dynamic gating coefficients to obtain the modulated time bias matrix.

[0026] Furthermore, the process of generating the dynamic gating coefficient specifically includes:

[0027] For each query location, perform linear projection and compression operations on the spatiotemporal embedding vector to obtain intermediate values ​​in scalar form;

[0028] The intermediate value is input into the Sigmoid activation function, and the output value is mapped to the (0,1) interval as the dynamic gating coefficient for the query position.

[0029] Furthermore, the linear projection operation is performed through a linear projection layer; the weight parameters of the linear projection layer are associated with the learnable lookup table in the following way:

[0030] During the training of the spatiotemporal Transformer model, the model parameters are updated based on constraints including a joint regularization loss function, which encourages the changes in the weight parameters to be consistent with the distribution of the changes in the vectors in the learnable lookup table.

[0031] Furthermore, the method also includes multi-target tracking operations, specifically including:

[0032] The multi-target recognition results output by the deep neural network model are input into multiple parallel Kalman filters;

[0033] Based on the precise time interval ΔT, the state transition matrix of each Kalman filter is determined; the motion state prediction and data association of the corresponding target are performed using each Kalman filter.

[0034] Furthermore, obtaining the image sequence and associated timestamps includes:

[0035] Receives a data stream from an image acquisition system configured to:

[0036] Image exposure is performed in response to a periodic hardware trigger signal issued by a timer synchronized with a high-precision clock, and the current moment from the high-precision clock is recorded as the exposure start timestamp at the start of the exposure.

[0037] A second aspect of the present invention discloses an image recognition processing system that integrates time information, the system comprising:

[0038] The data acquisition module is used to acquire image sequences and associated timestamps; the timestamp associated with each frame of the image sequence is the exposure start timestamp, and the inter-frame time intervals of the image sequence are uniform.

[0039] An interval determination module is used to determine the precise time interval ΔT between adjacent frames in the image sequence based on the exposure start timestamp;

[0040] The image recognition module is used to generate time-coded features based on the precise time interval ΔT and the exposure start timestamp;

[0041] The temporal coding features are fused with the visual features extracted from the image sequence to obtain the fused features;

[0042] The deep neural network model outputs image recognition results based on the fused features.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] This invention ensures uniform frame intervals and microsecond-level precision in the exposure start timestamp of image sequences from the source by directly stamping the exposure time with hardware synchronization triggering, providing a precise temporal data foundation for subsequent processing. Furthermore, by converting the time interval and exposure time into time codes deeply coupled to the time-transformer model, the focus on temporal information can be adaptively adjusted according to the scene content, significantly improving recognition accuracy in complex dynamic scenes. In addition, the spatiotemporal representation, strictly aligned with the physical time scale through model learning, not only improves the physical realism of motion estimation and prediction but also provides reliable support for directly measuring physical quantities such as velocity and acceleration in image sequences. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0046] Figure 1 This is a flowchart illustrating an image recognition processing method that integrates time information, as disclosed in an embodiment of the present invention. Detailed Implementation

[0047] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0048] Example 1

[0049] The first aspect of this invention discloses an image recognition processing method that integrates time information; please refer to [link / reference]. Figure 1 , Figure 1 This is a flowchart illustrating an image recognition processing method that integrates time information, as disclosed in an embodiment of the present invention. The method includes:

[0050] Obtain the image sequence and its associated timestamp; the timestamp associated with each frame of the image sequence is the exposure start timestamp, and the time intervals between frames of the image sequence are uniform;

[0051] Based on the exposure start timestamp, the precise time interval ΔT between adjacent frames in the image sequence is determined;

[0052] Based on the precise time interval ΔT and the exposure start timestamp, a time-coded feature is generated;

[0053] The temporal coding features are fused with the visual features extracted from the image sequence to obtain the fused features;

[0054] The deep neural network model outputs image recognition results based on the fused features.

[0055] Furthermore, image sequences and associated timestamps are obtained by receiving data streams from an image acquisition system, wherein the image acquisition system is configured as follows:

[0056] Image exposure is performed in response to a periodic hardware trigger signal issued by a timer synchronized with a high-precision clock, and the current moment from the high-precision clock is recorded as the exposure start timestamp at the start of the exposure.

[0057] It is understood that in this embodiment of the invention, the image sequence and associated timestamps originate from an image acquisition system with hardware synchronization at its core. This system provides a unified microsecond-level time reference for all components through a high-precision real-time clock module. The programmable time trigger module synchronizes with this reference and generates hardware trigger signals strictly according to a preset period. The image acquisition module synchronizes with the same time reference and, at the physical moment its sensor responds to the hardware trigger signal to initiate exposure, immediately latches the current time reference value and assigns it to the frame data as the exposure start timestamp. This operation ensures that the timestamp physically coincides with the exposure start point, eliminating the uncertainty caused by software delays. Simultaneously, the periodic triggering guaranteed by the hardware timer ensures that the acquisition time interval between frames remains uniform from the source. Therefore, the essence of the system output is a series of visual sample sequences uniformly arranged on the timeline, with each point bearing a precise physical time marker, providing a deterministic and consistent timing foundation for subsequent processing that is difficult to achieve with traditional software triggering methods.

[0058] Based on the unique data characteristics provided by the aforementioned image acquisition system, the subsequent image recognition and processing can be built upon a precise physical time dimension. Specifically, the exposure start timestamp provides each frame with coordinates on an absolute time axis, enabling the model to perceive the specific physical moment when the event occurred. The precise time interval ΔT defines the basic scale between these coordinate points; as a constant and known physical quantity, it provides a benchmark for measuring the speed of motion and change.

[0059] Furthermore, in this embodiment of the invention, during image recognition using a deep neural network model, the purpose of using the time-coding features generated by the exposure start timestamp and the precise time interval ΔT is to inject quantifiable and computable physical time perception into the model, thereby aligning and calibrating the model's internal temporal dynamics with the continuous temporal flow of the external physical world. The encoding generated based on the exposure start timestamp provides the model with a deterministic anchor point on the absolute time axis for each frame of visual observation, enabling the model to establish a stable mapping between the input sequence and continuous physical time. The encoding generated based on the precise time interval ΔT defines the physical reference scale for the relationship between frames, allowing the model's internal representation of motion, change, and duration to break free from the limitations of traditional video analysis based on fuzzy ordinal relationships, and instead be built upon a precise physical time difference with ΔT as the basic unit. The combined effect of these two encodings enables the model, especially the spatiotemporal Transformer model, to not only calculate the relevance of visual content in core mechanisms such as self-attention, but also to accurately assess and weight the strength of the physical temporal correlation between observation data at different times. This method of deeply integrating physical time semantics in representation and computation fundamentally improves the model's ability to understand dynamic scenes, enabling it to make more physically accurate estimates and predictions of the target's motion state, thereby achieving a performance breakthrough in tasks requiring high-precision temporal understanding.

[0060] Furthermore, the process of generating temporally encoded features includes:

[0061] Based on the exposure start timestamp of each frame of the image, a periodic time position encoding vector is generated using sine and cosine functions;

[0062] The frequency basis parameters of the sine and cosine functions are configured according to the precise time interval ΔT. After configuration, the frequency spectrum of the time position encoding vector changes over the entire time range of the image sequence is inversely proportional to ΔT.

[0063] Specifically, in this embodiment of the invention, the process of generating the time-location encoding vector involves converting the precise exposure start timestamp of each frame of the image—a continuous time value—into a multi-dimensional vector representation so that the deep neural network model can utilize it effectively. Since the original timestamp values ​​have a large range and lack periodicity, directly inputting them into the model would be difficult to learn. Therefore, a combination of sine and cosine functions is used to map the timestamp into a fixed-dimensional vector with periodicity and smooth variation. For each frame, its exposure start timestamp is input into a set of pre-defined sine and cosine functions with different frequencies of change, thereby generating a multi-dimensional vector. Each dimension of the vector is composed of sine or cosine function values ​​of a specific frequency, collectively representing the periodic characteristics of that time point across multiple frequency scales.

[0064] To match the encoding with the physical acquisition rhythm, this invention dynamically configures the frequency basis parameters of the sine and cosine functions based on a precise time interval ΔT. When ΔT is small, it indicates a high image acquisition frequency and short time intervals between adjacent frames. In this case, the temporal encoding needs to exhibit sufficient variation within a short time span to distinguish dense sampling moments. Therefore, the frequency basis parameters are adjusted to a higher level, allowing the generated temporal location encoding vector to complete more periodic oscillations within the same physical time. Conversely, when ΔT is large, the acquisition frequency is low and the time intervals between adjacent frames are long. Therefore, the frequency basis parameters need to be reduced to slow down the rate of encoding change and avoid disrupting the continuity due to excessive changes in the encoding values ​​of adjacent frames. Through this configuration, the overall frequency spectrum of the final generated temporal location encoding vector exhibits an inverse relationship with ΔT over the entire time range covered by the image sequence. This adaptive encoding mechanism ensures that regardless of the actual acquisition speed, the vector representation of temporal information maintains a stable rhythm of change that is coordinated with the physical sampling rate, thus providing an efficient time-aware foundation for the model.

[0065] Furthermore, the process of generating temporally encoded features also includes:

[0066] A relative time embedding vector is generated based on the precise time interval ΔT between multiple consecutive frames and stored using a learnable lookup table. The learnable lookup table is indexed by the number of inter-frame displacement steps, which is obtained by dividing the time difference by the precise time interval ΔT and rounding it down.

[0067] Specifically, in this embodiment of the invention, the purpose of generating the relative time embedding vector is to provide the model with a precise representation of the temporal relationship between frames. When the model needs to obtain the temporal relationship between any two frames, it calculates the actual physical time difference between these two exposure start timestamps. Since the inter-frame time interval of the image sequence is uniform and known to be ΔT, this time difference can essentially be expressed as an integer multiple of ΔT. Therefore, dividing the physical time difference by ΔT and rounding it down yields a pure integer index representing the number of inter-frame displacement steps, such as one step, two steps, or ten steps.

[0068] Subsequently, this integer index is set as the lookup key for the learnable lookup table. This lookup table is initialized before model training and optimized along with other model parameters during training. For each possible displacement step (e.g., from 0 steps to a preset maximum step number), the table stores a corresponding fixed-dimensional vector, i.e., a relative time embedding vector. By querying the index, the vector representation of that specific time relationship can be efficiently obtained. The purpose of this setup is to utilize a known, uniform ΔT to discretize the originally continuous, estimation-required time relationship into a regular, deterministic step relationship, thereby making the learning of the time relationship representation stable and efficient. Compared to traditional methods that require laborious learning of fuzzy time relationships from non-uniform interval sequences, this invention, through hardware-guaranteed ΔT, makes a lookup table-based regularization embedding mechanism possible, greatly simplifying the modeling difficulty of temporal logic and improving its generalization ability and accuracy.

[0069] Furthermore, the deep neural network model is a spatiotemporal Transformer model; the generation process of the fused features includes:

[0070] The temporal location encoding vector is added element by element to the image patch embedding vector to form a spatiotemporal embedding sequence;

[0071] The spatiotemporal embedding sequence is input into the multilayer encoder of the spatiotemporal Transformer model, and a temporal bias matrix is ​​added to the dot product of the query vector and the key vector in the self-attention mechanism of the multilayer encoder. The temporal bias matrix is ​​constructed based on the relative temporal embedding vector and is used to characterize the temporal correlation strength between different image patches.

[0072] The process of generating image patch embedding vectors includes:

[0073] Before fusing the temporal coding features with the visual features extracted from the image sequence, each frame image is divided into multiple image blocks, and each image block is converted into an embedding vector through linear projection to obtain the image block embedding vector.

[0074] Furthermore, before constructing the time bias matrix, a dynamic gating coefficient is generated at each query position in the spatiotemporal embedding sequence. After generating the dynamic gating coefficient, an initial time bias matrix is ​​constructed based on the relative time embedding vector, and the initial time bias matrix is ​​modulated in the row direction using the dynamic gating coefficient to obtain the modulated time bias matrix.

[0075] Specifically, in this embodiment of the invention, the deep neural network model is limited to a spatiotemporal Transformer model because the core self-attention mechanism of the spatiotemporal Transformer model can naturally and flexibly model long-range dependencies between any elements in a sequence, making it suitable for processing image sequence data with a clear spatiotemporal structure. To deeply integrate physical time information into the model's recognition process, the generation of fusion features in this invention employs a layered injection strategy. Specifically, the time position encoding vector representing the absolute physical moment is added element-wise to the image patch embedding vector extracted from the image content. The purpose of this operation is to bind a specific coordinate on a continuous time axis to each visual unit (image patch) in the initial stage of data input, thereby forming a spatiotemporal embedding sequence that simultaneously carries visual semantics and absolute temporal context, laying the foundation for the model's subsequent temporal understanding.

[0076] When the spatiotemporal embedding sequence is input into the multi-layer encoder of the Transformer, the model needs to explicitly consider the temporal relationship between different positions in the self-attention calculation. To this end, while calculating the dot product of the query vector and the key vector to measure content similarity, an additional temporal bias matrix is ​​added. This matrix is ​​constructed based on the relative temporal embedding vectors, and the value of each element represents the association strength or prior weight between two corresponding positions in the sequence determined by the physical time difference. It is understandable that even if two pieces of visual content are highly similar, their actual correlation may be weak if they are physically far apart; conversely, frames that are physically close may have a stronger contextual association. Therefore, this invention, by adding this bias matrix, guides the attention mechanism to focus on content matching while also following physical temporal logic, thereby making a more reasonable judgment.

[0077] Furthermore, to endow the model with the intelligence to adaptively adjust temporal attention based on image content, this invention first generates a dynamic gating coefficient at each query position based on the spatiotemporal embedding sequence before constructing the temporal bias matrix. This coefficient reflects the degree to which the visual content at the current position depends on precise temporal information. After obtaining the initial temporal bias matrix constructed based on relative temporal embedding, these dynamic gating coefficients are used to modulate the matrix in the row direction. Specifically, the gating coefficient corresponding to each query position scales (modulates) the initial temporal bias row from that query position to all key positions. The purpose is to achieve content-aware temporal adjustment. For query positions where feature display requires strong temporal correlation (such as fast-moving objects), the gating coefficient is larger, enhancing the influence of temporal bias; for positions where temporal correlation is not important (such as static backgrounds), its influence is reduced. The above settings enable the model to dynamically and discriminatively utilize precise temporal information, thereby achieving more accurate spatiotemporal feature fusion and recognition in complex scenes.

[0078] Furthermore, the process of generating the dynamic gating coefficients specifically includes:

[0079] For each query location, perform linear projection and compression operations on the spatiotemporal embedding vector to obtain intermediate values ​​in scalar form;

[0080] The intermediate value is input into the Sigmoid activation function, and the output value is mapped to the (0,1) interval as the dynamic gating coefficient for the query position.

[0081] It should be noted that although the generation process of image patch embedding vectors and the generation process of dynamic gating coefficients both involve linear projection operations, their purposes, input objects and output forms are fundamentally different, serving different stages of the model processing flow.

[0082] In the initial stage of image processing, to extract visual features from an image sequence, each frame needs to be divided into multiple regularly arranged image patches. Each image patch contains a set of raw pixel information. The linear projection operation here aims to perform a basic feature space transformation, taking all pixel values ​​of each image patch—for example, a two-dimensional matrix containing several pixels—as input, and performing linear calculations through a learnable weight matrix to transform it into a high-dimensional, dense embedding vector, i.e., the image patch embedding vector. Essentially, this operation maps the raw, low-level pixel signals to an abstract feature space more suitable for semantic understanding and relational modeling in deep neural networks. Its output is the fundamental visual representation of the entire visual processing flow.

[0083] During the forward propagation of the model, a second linear projection operation is performed when dynamic gating coefficients need to be generated. The input object here is not the original pixels, but rather a pre-formed spatiotemporal embedding vector—a high-level feature rich in semantics that integrates absolute temporal encoding and visual embedding. The purpose of this linear projection is no longer basic feature extraction, but rather information compression and decision focusing. By receiving the high-dimensional spatiotemporal embedding vector and linearly combining and compressing it using learnable weight vectors (or matrices), a scalar intermediate value is finally output. This scalar aims to extract summary information from complex spatiotemporal features that reflects the degree of dependence of the visual content at the current query location on precise temporal correlation.

[0084] After generating the dynamic gating coefficients, this scalar intermediate value is input into the Sigmoid activation function. The Sigmoid function maps any real number to the (0,1) interval, and its output can be viewed as a smooth switch or intensity modulator. Understandably, the spatiotemporal embedding vector already contains the visual content and absolute temporal information of the current location. Through learning, the linear projection layer can identify patterns that suggest strong or weak temporal attention; for example, feature patterns representing high-speed motion correspond to high coefficients, while feature patterns representing static backgrounds correspond to low coefficients. The final coefficients, between 0 and 1, serve as a content-aware modulation signal and are used to scale the temporal bias matrix in the row direction. Through these operations, the model can adaptively and discriminatively adjust the influence of precise temporal information in the attention mechanism based on the specific visual content of the input image sequence, thereby achieving more accurate recognition in complex dynamic scenes.

[0085] Furthermore, the linear projection operation is performed through a linear projection layer; the weight parameters of the linear projection layer are associated with the learnable lookup table in the following way:

[0086] During the training of the spatiotemporal Transformer model, the model parameters are updated based on constraints including a joint regularization loss function, which encourages the changes in the weight parameters to be consistent with the distribution of the changes in the vectors in the learnable lookup table.

[0087] Specifically, in this embodiment of the invention, in order to improve the consistency of the internal components of the spatiotemporal Transformer model in understanding temporal semantics and thus improve the overall efficiency of temporal information utilization, this invention establishes a specific association between the weight parameters of the linear projection layer that generates dynamic gating coefficients and the learnable lookup table that stores relative temporal embedding vectors during model training. The weight parameters of the linear projection layer can be understood as a series of adjustable values ​​that define how the input spatiotemporal embedding vectors are linearly combined and compressed into scalar intermediate values ​​for gating decisions. These weight parameters essentially determine the importance of which information patterns the model extracts and prioritizes from rich spatiotemporal features to determine temporal correlations.

[0088] The aforementioned correlation is achieved by introducing a joint regularization loss function as a constraint during model training. This function not only focuses on the accuracy of the model in performing the main task (such as image recognition) but also imposes an additional constraint to encourage the weight parameters of the linear projection layer to align with the changing trends of the vectors in the learnable lookup table during the update process. Understandably, although these two components perform different functions in the model, they both deal with the core object of time. The vectors in the lookup table encode the semantics of physical time differences (e.g., 1ΔT, 2ΔT), while the linear projection layer needs to learn to determine the extent to which it depends on these temporal relationships based on the content. Through joint regularization, the training process prompts the model to find an optimal balance point, allowing the linear projection layer to adjust its weights to better generate gating coefficients while considering the distribution of the temporal relationship representations currently learned from the lookup table, and vice versa. This correlation essentially promotes the co-evolution of both components within the model optimization objective.

[0089] Through the above operations, contradictory or inconsistent temporal information processing patterns learned by different parts of the model during training are effectively avoided. This ensures that the decision logic for generating gating coefficients and the underlying temporal relationship representation logic are semantically consistent, thereby enhancing the systematicness and consistency of the model's internal processing of temporal information. Ultimately, this enables the model to utilize high-quality temporal data guaranteed by accurate ΔT more stably and efficiently, building content-aware dynamic gating capabilities on a solid and collaborative foundation of temporal semantic understanding, further improving the model's accuracy in recognizing complex dynamic scenes.

[0090] Furthermore, the method also includes multi-target tracking operations, specifically including:

[0091] The multi-target recognition results output by the deep neural network model are input into multiple parallel Kalman filters;

[0092] Based on the precise time interval ΔT, the state transition matrix of each Kalman filter is determined, and the motion state prediction and data association of the corresponding target are performed using each Kalman filter.

[0093] In this embodiment of the invention, the purpose of the multi-target tracking operation is to combine the high-precision perception capability of deep learning with the stable prediction capability of classical state estimation algorithms to form a complete closed-loop mechanism from identification to tracking prediction.

[0094] Specifically, each filter is independently responsible for tracking a specific target. The identification results include, but are not limited to, the target's category, position in the image, size, and other information. This information is used to initialize the state vector of the corresponding Kalman filter (such as position and velocity) or as observations in subsequent frames to update the filter's state.

[0095] Subsequently, based on the precise time interval ΔT, the state transition matrix of each Kalman filter is determined. The key to Kalman filter prediction lies in its state transition matrix, which describes how the target state (e.g., position, velocity) evolves from one moment to the next, and its mathematical form depends on the time interval between two moments. In traditional video analysis, this time interval is typically used as an estimated average frame rate or a coarse difference between the timestamps of consecutive frames. Its inaccuracy and jitter directly contaminate the state transition matrix, leading to inaccurate predicted trajectories. In this invention, thanks to the hardware-guaranteed, uniform, and precise ΔT, the time interval corresponding to each prediction step is a known, constant, and precise physical quantity. Therefore, when constructing or updating the state transition matrix of each Kalman filter, this precise ΔT can be directly used as the time parameter in the matrix, thereby ensuring strict alignment of the filter's prediction model with the target's physical motion in the real world (e.g., uniform velocity or uniform acceleration models) on the time scale.

[0096] After obtaining the recognition results of the current frame, each filter first uses a state transition matrix based on precise ΔT to predict the state (such as predicted position) that the tracked target should have at the exposure time of the next frame. When the next frame arrives, the acquired multiple target detection results are matched and associated with the predicted states of all filters. Since the prediction is calculated based on precise time intervals, its predicted position is more accurate, which greatly improves the success rate of associating the correct observation with the correct filter (i.e., the correct target), effectively solving the tracking continuity challenge in complex scenarios such as target occlusion, intersection, and brief disappearance. Therefore, by injecting the precise temporal basis provided by the front end into the dynamic model of the back-end tracking algorithm, the smoothness and accuracy of multi-target tracking are significantly improved, achieving deep fusion and performance enhancement at the perception and prediction levels.

[0097] Furthermore, in this embodiment of the invention, to enable the spatiotemporal Transformer model to effectively output image recognition results based on the aforementioned fusion features, it is designed to fully learn and utilize the unique temporal information provided by the exposure start timestamp and the precise time interval ΔT during the construction and training process. Specifically, when constructing the input layer, each frame of image is first divided into image patches and converted into embedding vectors. Simultaneously, a temporal position encoding vector generated based on the exposure start timestamp of that frame is added element-wise, thereby binding the absolute temporal context of each visual unit at the beginning of the input, forming an initial spatiotemporal embedding sequence. This sequence is then fed into a backbone network composed of multiple Transformer encoders. In each encoder layer, the self-attention mechanism, based on the calculation of content relevance, introduces a temporal bias matrix constructed from the relative temporal embedding vectors and modulated by dynamic gating coefficients. Through the above operations, the model learns visual semantic associations while explicitly modeling and adjusting its attention distribution according to precise physical temporal relationships. The model maps features that are deeply fused with precise spatiotemporal information to specific image recognition results, such as object bounding boxes, categories, or behavior labels, by adding specific task heads (e.g., detection heads for object detection or classification heads for behavior classification) to the sequence output.

[0098] During training, a large amount of labeled image sequence data is prepared. This data should originate from or simulate the image acquisition system described in this invention, meaning that each frame has an exposure start timestamp and the frame intervals within the sequence are uniform. Each input sample is a structured data packet, the core of which includes an image sequence and a list of exposure start timestamps that strictly correspond to each frame in the sequence. The expected output of the model depends on the downstream recognition task. For example, for an object detection task, the output is the category and precise location (such as bounding box coordinates) of all objects of interest in each frame; for an action recognition task, the output is the specific action category label represented by the entire segment.

[0099] To specifically improve the model's generalization ability to changes in time scale, training sample batches containing different time intervals ΔT are constructed. This means that within a single training batch, samples from high-frame-rate sequences and low-frame-rate sequences can be mixed, or different ΔT can be artificially simulated by sampling the original uniform sequence across frames. Through this exposure, the model is forced to learn and adapt the temporal encoding and fusion logic to various acquisition rhythms.

[0100] Building upon this, the model's overall loss function is designed to include a regularization term related to the time interval ΔT. The core purpose of this regularization term is to encourage the model to learn essential spatiotemporal features that are insensitive to changes in physical time scales. For example, it can penalize the model for outputting excessively inconsistent intermediate features or recognition results for two sequences describing the same physical motion process but sampled at different ΔT. By jointly optimizing this regularization term with the main task loss function (such as detection loss or classification loss), the model learns to accurately complete the recognition task while its internal parameters (especially those handling temporal encoding, dynamic gating, and feature fusion) are guided to a more optimal solution space. In this solution space, the model's understanding of time is based on physical laws rather than a specific sampling rate, thus ensuring stable and high-performance image recognition capabilities based on accurate temporal information even when the sampling frequency changes.

[0101] Example 2

[0102] A second aspect of the present invention discloses an image recognition processing system that integrates time information, the system comprising:

[0103] The data acquisition module is used to acquire image sequences and associated timestamps; the timestamp associated with each frame of the image sequence is the exposure start timestamp, and the inter-frame time intervals of the image sequence are uniform.

[0104] An interval determination module is used to determine the precise time interval ΔT between adjacent frames in the image sequence based on the exposure start timestamp;

[0105] The image recognition module is used to generate time-coded features based on the precise time interval ΔT and the exposure start timestamp;

[0106] The temporal coding features are fused with the visual features extracted from the image sequence to obtain the fused features;

[0107] The deep neural network model outputs image recognition results based on the fused features.

[0108] It should be noted that the specific implementation process of Example 2 is similar to that of Example 1, and will not be repeated in Example 2.

[0109] Finally, it should be noted that the above-described embodiments include multiple parallel implementations of the present invention. Deleting or otherwise adjusting one or more implementations will not affect the implementation of the solution. Furthermore, the image recognition processing method and system for fusing time information disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image recognition processing method incorporating time information, characterized in that, The method includes: Obtain the image sequence and its associated timestamp; the timestamp associated with each frame of the image sequence is the exposure start timestamp, and the time intervals between frames of the image sequence are uniform; Based on the exposure start timestamp, the precise time interval ΔT between adjacent frames in the image sequence is determined; Based on the precise time interval ΔT and the exposure start timestamp, a time-coded feature is generated; The temporal coding features are fused with the visual features extracted from the image sequence to obtain the fused features; The deep neural network model outputs image recognition results based on the fused features.

2. The image recognition processing method with fused time information according to claim 1, characterized in that, The process of generating time-coded features includes: Based on the exposure start timestamp of each frame of the image, a periodic time position encoding vector is generated using sine and cosine functions; The frequency basis parameters of the sine and cosine functions are configured according to the precise time interval ΔT. After configuration, the frequency spectrum of the time position encoding vector changes over the entire time range of the image sequence is inversely proportional to ΔT.

3. The image recognition processing method incorporating time information according to claim 2, characterized in that, The process of generating time-coded features also includes: A relative time embedding vector is generated based on the precise time interval ΔT between multiple consecutive frames and stored using a learnable lookup table. The learnable lookup table is indexed by the number of inter-frame displacement steps, which is obtained by dividing the time difference by the precise time interval ΔT and rounding it down.

4. The image recognition processing method with fused time information according to claim 3, characterized in that, The deep neural network model is a spatiotemporal Transformer model; the generation process of the fused features includes: The temporal location encoding vector is added element by element to the image patch embedding vector to form a spatiotemporal embedding sequence; The spatiotemporal embedding sequence is input into the multilayer encoder of the spatiotemporal Transformer model. A temporal bias matrix is ​​added to the dot product of the query vector and the key vector in the self-attention mechanism of the multilayer encoder. The temporal bias matrix is ​​constructed based on the relative temporal embedding vector and is used to characterize the temporal correlation strength between different image patches. The process of generating the image patch embedding vector includes: Before fusing the temporal coding features with the visual features extracted from the image sequence, each frame image is divided into multiple image blocks, and each image block is converted into an embedding vector through linear projection to obtain the image block embedding vector.

5. The image recognition processing method with fused time information according to claim 4, characterized in that, Before constructing the time bias matrix, the method further includes: Based on the spatiotemporal embedding sequence, a dynamic gating coefficient is generated at each query position in the sequence; After generating the dynamic gating coefficients, the construction process of the time bias matrix specifically includes: Construct the initial time bias matrix based on the relative time embedding vector; The initial time bias matrix is ​​modulated in the row direction using the dynamic gating coefficients to obtain the modulated time bias matrix.

6. The image recognition processing method with fused time information according to claim 5, characterized in that, The process of generating the dynamic gating coefficient specifically includes: For each query location, perform linear projection and compression operations on the spatiotemporal embedding vector to obtain intermediate values ​​in scalar form; The intermediate value is input into the Sigmoid activation function, and the output value is mapped to the (0,1) interval as the dynamic gating coefficient for the query position.

7. The image recognition processing method incorporating time information according to claim 6, characterized in that, The linear projection operation is performed through a linear projection layer; the weight parameters of the linear projection layer are associated with the learnable lookup table in the following way: During the training of the spatiotemporal Transformer model, the model parameters are updated based on constraints including a joint regularization loss function, which encourages the changes in the weight parameters to be consistent with the distribution of the changes in the vectors in the learnable lookup table.

8. The image recognition processing method with fused time information according to claim 1, characterized in that, The method also includes multi-target tracking operations, specifically including: The multi-target recognition results output by the deep neural network model are input into multiple parallel Kalman filters; Based on the precise time interval ΔT, the state transition matrix of each Kalman filter is determined; the motion state prediction and data association of the corresponding target are performed using each Kalman filter.

9. The image recognition processing method incorporating time information according to claim 1, characterized in that, The acquisition of the image sequence and associated timestamps includes: Receives a data stream from an image acquisition system configured to: Image exposure is performed in response to a periodic hardware trigger signal issued by a timer synchronized with a high-precision clock, and the current moment from the high-precision clock is recorded as the exposure start timestamp at the start of the exposure.

10. An image recognition processing system incorporating time information, the system being implemented based on the method described in any one of claims 1-9, characterized in that, The system includes: The data acquisition module is used to acquire image sequences and associated timestamps; the timestamp associated with each frame of the image sequence is the exposure start timestamp, and the inter-frame time intervals of the image sequence are uniform. An interval determination module is used to determine the precise time interval ΔT between adjacent frames in the image sequence based on the exposure start timestamp; The image recognition module is used to generate time-coded features based on the precise time interval ΔT and the exposure start timestamp, fuse the time-coded features with visual features extracted from the image sequence to obtain fused features, and output image recognition results based on the fused features through a deep neural network model.

Images