Audio and video object intelligent tracking optimization method and system combined with deep learning

Abstract

The application relates to the technical field of audio and video processing, and provides an audio and video object intelligent tracking optimization method and system combined with deep learning. In the application, by acquiring a synchronous audio and video data group, cross-modal feature collaborative extraction is performed on an audio stream and a video frame sequence to generate a multi-modal feature set containing audio time domain dynamic features and video space structure features; the multi-modal feature set is input into a pre-trained correlation enhancement network to generate a correlation feature sequence with cross-modal semantic alignment; a tracking stability evaluation model is constructed based on the correlation feature sequence to output a stability index; tracking parameters are dynamically adjusted and initial tracking results are calibrated according to the stability index to output an optimized tracking trajectory. Thus, the application deeply fuses audio and video dual modal features, mines the internal correlation between the modes, combines a dynamic evaluation and calibration mechanism, and improves the precision and stability of object tracking in a complex scene.

Description

Technical Field

[0001] This application relates to the fields of pattern recognition and audio / video processing technology, and in particular to an intelligent tracking optimization method and system for audio / video objects that combines deep learning. Background Technology

[0002] Audio and video object tracking is a technology that analyzes and processes audio streams and video frame sequences to achieve dynamic positioning and trajectory tracking of specific target objects in time and space. It is widely used in fields such as security monitoring, intelligent transportation, human-computer interaction, film and television production, and autonomous driving.

[0003] Existing audio and video object tracking technologies often rely on single-modal data processing. When using only video frame sequences, image features become blurred or even missing when the target object encounters occlusion, drastic changes in lighting conditions, or complex background environments, leading to tracking drift and inaccurate target locking. When relying solely on audio stream data, environmental noise severely interferes with tracking. If the target object emits a weak sound, or multiple sound sources are present simultaneously, it becomes difficult to accurately determine the target's location. Even when some technologies attempt to fuse audio and video dual-modal data, the fusion methods are relatively superficial. They typically involve directly concatenating audio and video features, or simply performing a weighted sum before inputting it into the tracking model. They fail to deeply consider the inherent temporal and spatial relationships between the two modal features, nor do they address the semantic differences between the modalities. This simplistic fusion approach fails to fully leverage the complementary information between the modalities. When the feature quality of one modality deteriorates, the other modality cannot effectively compensate for its deficiencies, resulting in poor tracking stability.

[0004] Furthermore, existing tracking methods lack a dynamic evaluation mechanism for the tracking process, making it impossible to determine the reliability of tracking results in real time. When tracking deviations occur, tracking parameters cannot be adjusted for calibration in a timely manner; instead, they can only passively rely on feature updates in subsequent frames for correction. This leads to the continuous accumulation of errors, ultimately severely impacting tracking accuracy. Simultaneously, in the feature extraction stage, existing technologies do not extract the temporal dynamic features of audio and the spatial structural features of video in sufficient detail. For example, audio features only extract frequency peaks, ignoring the temporal changes in energy fluctuations; video features only focus on the contour information of the target, ignoring texture details. This limits the expressive power of the extracted features, further restricting the improvement of tracking performance. Summary of the Invention

[0005] In view of the above, and aiming to at least partially address the shortcomings of existing technologies and bring new solutions to the field of audio transmission, this application provides, in a first aspect, a method for intelligent tracking and optimization of audio and video objects combining deep learning, the method comprising:

[0006] Acquire multiple audio and video data groups that are continuously collected, wherein the audio and video data groups include synchronously collected audio stream data and video frame sequences;

[0007] Perform cross-modal feature collaborative extraction on the audio stream data and video frame sequence to generate a multimodal feature set containing audio temporal dynamic features and video spatial structure features;

[0008] The multimodal feature set is input into a pre-trained association enhancement network for temporal association modeling, generating an association feature sequence with cross-modal semantic alignment;

[0009] A tracking stability evaluation model is constructed based on the associated feature sequence, and a stability index reflecting the reliability of the tracking process is output.

[0010] Based on the stability index, the tracking parameters are dynamically adjusted and the initial tracking results are calibrated to output the optimized object tracking trajectory for the target object.

[0011] Secondly, embodiments of this application also provide an intelligent audio-visual object tracking optimization system combining deep learning, including a processor and a machine-readable storage medium. The machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the machine-readable storage medium to implement the intelligent audio-visual object tracking optimization method combining deep learning.

[0012] In summary, the intelligent audio-visual object tracking optimization method and system combining deep learning provided in this application can significantly improve the accuracy and stability of audio-visual object tracking. Specifically, this application first acquires synchronized audio-visual data sets, ensuring the temporal consistency of data during subsequent cross-modal processing and avoiding modal information misalignment caused by data asynchrony. Based on this, a cross-modal feature collaborative extraction operation is performed. For audio stream data, temporal dynamic features including frequency distribution patterns and energy fluctuation modes are extracted; for video frame sequences, spatial structural features including contour boundaries and texture distribution are extracted. Compared to existing technologies, this method more comprehensively captures the multi-dimensional features of the target object, enhances the expressive power of the features, and provides richer information for subsequent tracking processing. Furthermore, by using an association enhancement network to perform temporal association modeling on the multimodal feature set, deep alignment of audio temporal dynamic features and video spatial structure features at the semantic level is achieved. This fully explores the inherent association between the two modalities in the temporal evolution process, solving the problems of simple modality fusion and insufficient association mining in existing technologies. It enables the bimodal features to complement each other, improves the robustness of features in describing target objects, and can better identify and track targets even in complex environments.

[0013] Furthermore, a tracking stability assessment model is introduced. By calculating feature consistency parameters, temporal continuity parameters, and cross-modal matching degree parameters, the reliability of the tracking process is quantified in real time, providing a basis for dynamically adjusting tracking parameters. The initial tracking results are calibrated using stability indices, achieving closed-loop optimization of the tracking process, avoiding the accumulation of tracking deviations, and addressing the lack of dynamic evaluation and real-time calibration mechanisms in existing technologies. This allows the tracking results to adaptively adjust with environmental changes, further improving tracking accuracy and stability. Thus, through the organic combination of deep cross-modal feature collaboration, temporal correlation modeling, dynamic evaluation, and parameter adjustment, the object tracking accuracy and stability in complex scenarios (such as target occlusion, noise interference, and illumination changes) are significantly improved, making the tracking results more reliable and robust, and meeting the high-precision tracking requirements of practical applications.

[0014] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the above drawings without creative effort.

[0016] To gain a more complete understanding of this application and its beneficial effects, the following description will be provided in conjunction with the accompanying drawings, wherein the same reference numerals in the following description denote the same parts.

[0017] Figure 1 This is a flowchart illustrating an intelligent tracking and optimization method for audio and video objects that incorporates deep learning, as provided in an embodiment of this application.

[0018] Figure 2 This is a schematic diagram of the architecture of an audio / video object tracking application scenario provided in an embodiment of this application.

[0019] Figure 3 This is a schematic diagram of an intelligent audio and video object tracking optimization system that combines deep learning, provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0021] Please see Figure 1 and Figure 2 As shown, Figure 1 This is a flowchart illustrating the intelligent audio / video object tracking optimization method combining deep learning provided in an embodiment of this application. Figure 2 This is a schematic diagram of an audio / video object tracking application scenario. The application scenario may include an audio / video content processing terminal 100 and an audio / video content acquisition terminal 200 for audio / video content interaction. In this embodiment, the method can be implemented by the audio / video content processing terminal 100. For example, Figure 2 As shown, the audio and video content acquisition terminal may include acquisition devices such as cameras, microphones, camera arrays, and microphone arrays. The audio and video content processing terminal 100 may be a server, server cluster, or computer equipment, for example, a backend server for providing audio and video content monitoring and analysis. This embodiment does not specifically limit this. The following uses tracking the movement trajectory of a specific target object (such as a shopping mall customer, pedestrian, vehicle, stadium athlete, or live stream) in a target environment as an example to describe in detail the intelligent audio and video object tracking optimization method combining deep learning of this application.

[0022] like Figure 1 As shown, the method includes steps S110-S150, which will be described in detail below.

[0023] Step S110: Obtain a group of continuously acquired audio and video data segments, wherein the audio and video data segments include synchronously acquired audio stream data and video frame sequences.

[0024] In this embodiment, as an example, audio and video acquisition devices are deployed in different areas of the target environment. These devices include cameras and microphones, used to acquire audio and video information within the target environment. When it is necessary to track a specific target object, the acquisition devices are activated to collect audio and video data of the target object as it moves within the target environment, forming multiple audio and video data groups. The audio stream data in each audio and video data group records the sound information of the target object and its surrounding environment, while the video frame sequence records the dynamic image of the target object.

[0025] In this embodiment, the audio and video synchronization acquisition device can be activated first, and a hardware triggering mechanism can be used to ensure that the audio sampling and video frame capture are synchronized in time. Then, a fixed acquisition frame rate and sampling rate are set to ensure that the audio stream data and the video frame sequence have the same temporal resolution. For example, the acquisition frame rate of the camera is set to Fv, that is, Fv frames of video images are captured per second; the sampling rate of the microphone is set to Fa, that is, Fa audio samples are acquired per second. In order to ensure consistent temporal resolution and match the time interval corresponding to each video image with the time interval corresponding to each audio sample, the time length corresponding to each video image is set to 1 / Fv, and the time length corresponding to each audio segment containing Fa / Fv audio samples is also 1 / Fv. This ensures that the audio stream data and the video frame sequence are consistent in temporal resolution.

[0026] Next, the acquired raw audio stream data undergoes noise reduction processing, using an adaptive filtering algorithm to suppress environmental noise. Then, the acquired raw video frame sequence is subjected to brightness equalization processing to eliminate image quality fluctuations caused by changes in lighting. Further, the processed audio stream data and video frame sequence are combined in chronological order of acquisition time to form a continuous audio-video data group sequence. In this embodiment, as an example, timestamps can also be added to the processed audio stream data and video frame sequence, with the timestamps accurate to the acquisition time. Then, according to the order of the timestamps, the corresponding audio stream data segments and video frames are combined to form an audio-video data group. Multiple audio-video data groups are arranged in chronological order to form a continuous audio-video data group sequence, with each data group corresponding to the audio-visual information of a specific target object at a certain moment within the target environment.

[0027] Step S120: Perform cross-modal feature collaborative extraction operation on the audio stream data and video frame sequence to generate a multimodal feature set containing audio temporal dynamic features and video spatial structure features.

[0028] In this embodiment, as an example, feature extraction is performed on the audio stream data and video frame sequence obtained in step S110. For the audio stream data, dynamic features reflecting its temporal changes are extracted; for the video frame sequence, features reflecting the spatial structure of the target object are extracted. Then, these two types of features are processed collaboratively to form a multimodal feature set, which integrates information from both audio and video, and is used for subsequent target object tracking processing. In this embodiment, step S120 may include sub-steps S121-S125, which are described in detail below.

[0029] Step S121: Perform time-frequency conversion processing on the audio stream data to extract basic audio features containing frequency distribution patterns and energy fluctuation modes.

[0030] In this embodiment, as an example, the noise-reduced audio stream data is processed by converting the audio signal from the time domain to the frequency domain through time-frequency conversion, thereby analyzing its frequency distribution. Simultaneously, the changes in audio signal energy over time are observed, and features that reflect the frequency distribution pattern and energy fluctuation pattern are extracted as basic audio features. In this embodiment, step S121 may include sub-steps S1211-S1215, which are described in detail below.

[0031] Step S1211: Perform frame segmentation on the audio stream data to generate multiple audio subframes with overlapping windows.

[0032] In this embodiment, as an example, a sliding window method is used to frame the audio stream data. The window length is set to L, and the overlap length is O, where O is less than L. Starting from the beginning of the audio stream data, an audio segment of length L is extracted as the first audio subframe. Then, the window is slid backward by a length of L, and an audio segment of length L is extracted as the second audio subframe. This sliding window continues in the same way until the entire audio stream data is processed, generating multiple audio subframes with overlapping windows. For example, the time corresponding to the window length L is t_len, and the time corresponding to the overlap O is t_overlap. This framing process ensures overlap between adjacent audio subframes, avoiding information loss caused by framing.

[0033] Step S1212: Apply Fast Fourier Transform to each audio subframe to convert the time-domain signal into a frequency-domain signal and extract the spectrogram.

[0034] In this embodiment, as an example, each audio subframe is treated as a time-domain signal, and a Fast Fourier Transform (FFT) algorithm is applied to convert the time-domain signal into a frequency-domain signal. The frequency-domain signal contains the amplitude information of different frequency components in the audio subframe. Based on the amplitude information, a spectrum diagram can be plotted. The horizontal axis of the spectrum diagram represents frequency, and the vertical axis represents amplitude. The distribution of each frequency component in the audio subframe can be intuitively seen through the spectrum diagram.

[0035] Step S1213: Analyze the frequency peak distribution of the spectrum to identify the characteristic frequency range of the sound emitted by the target object.

[0036] In this embodiment, as an example, the spectrogram of each audio subframe is analyzed to identify the frequency points with larger amplitudes, i.e., frequency peaks. The distribution of these frequency peaks on the spectrogram is observed, and combined with the characteristics of the sound emitted by the target object (a specific target object), such as the frequency range of speech, the frequency peaks belonging to the target object's sound are identified, and the frequency interval corresponding to these peaks is determined. This interval is used as the characteristic frequency interval of the sound emitted by the target object, for example, the interval between Freq1 and Freq2.

[0037] Step S1214: Calculate the cumulative energy value of the spectrum within the characteristic frequency range and generate a fluctuation curve reflecting the change of energy over time.

[0038] In this embodiment, as an example, for each audio subframe, the sum of squares of the amplitudes of all frequency components within the characteristic frequency range (Freq1 to Freq2) is calculated on its spectrogram. This sum of squares is the energy value of the audio subframe within the characteristic frequency range. The energy values ​​of each audio subframe are arranged in chronological order, and a curve is plotted with time on the x-axis and energy value on the y-axis. This curve reflects the fluctuation of energy over time, and the change of the target object's sound energy over time can be seen from this curve.

[0039] Step S1215: The characteristic frequency range information and the energy fluctuation curve are fused to form basic audio features that include frequency distribution patterns and energy fluctuation modes.

[0040] In this embodiment, as an example, the characteristic frequency range information can be represented by the starting frequency Freq1 and the ending frequency Freq2 of the range, and the energy fluctuation curve can be represented by the energy values ​​E1, E2, E3, ... corresponding to each time point on the curve. Freq1, Freq2, and E1, E2, E3, ... are arranged in a certain order and combined into a feature vector. This feature vector is the basic audio feature, which simultaneously contains the frequency distribution pattern and energy fluctuation mode of the target object's sound.

[0041] Step S122: Perform motion sensing processing on the video frame sequence, capture the displacement trend of the target object between consecutive frames using optical flow estimation technology, and generate video motion features.

[0042] In this embodiment, as an example, for two consecutive frames in a video frame sequence (referred to as a "consecutive frame pair"), such as frame t and frame t+1, optical flow estimation technology is used to process and obtain video motion features. First, the region where the target object (a specific target object) is located is determined in frame t, and then multiple feature points are selected within this region. The position of the aforementioned feature points in frame t+1 is calculated using an optical flow estimation algorithm. Based on the positional changes of the feature points between the two frames, the displacement vector of each feature point is obtained. The direction and magnitude of the aforementioned displacement vectors are analyzed to determine the displacement trend of the target object between consecutive frames, such as the movement trend in the left, right, forward, and backward directions, as well as the speed of movement. The aforementioned displacement trend information is represented in vector form to generate video motion features.

[0043] As another detailed example, step S122 may be specifically implemented through steps S1221-S1229 described below.

[0044] Step S1221: Extract stable feature points within the region where the target object is located from consecutive frame pairs of the video frame sequence. These stable feature points can be pixels whose grayscale value change rate is lower than a preset threshold in consecutive frames, and are distributed along the edges and in areas of significant texture change of the target object.

[0045] In this embodiment, taking the tracking of a target object in a dynamic scene as an example, a continuous frame pair formed by consecutive frames t and t+1 is selected as the processing object. The region where the target object is located is located by the target detection model. Each pixel in the region is traversed, and its grayscale value change rate between two frames is calculated. Pixels with a change rate lower than a preset threshold are initially screened out. Then, combined with edge detection results and local entropy calculation, pixels located at the contour edge of the target object and in texture-rich areas are retained to form a stable feature point set.

[0046] Step S1222: Calculate the motion vector of each stable feature point in consecutive frames using optical flow estimation technology. The motion vector includes horizontal displacement components and vertical displacement components, forming a set of feature point motion vectors.

[0047] In this embodiment, based on a stable feature point set, optical flow estimation is used to calculate the motion trajectory of each feature point. Let the coordinates of a feature point in frame t be (a, b) and in frame t+1 be (c, d). Then the horizontal displacement component is ca, and the vertical displacement component is db, which together form the motion vector of the feature point. The motion vectors of all stable feature points are arranged in order of feature point number to form a feature point motion vector set.

[0048] Step S1223: Perform vector clustering on the set of feature point motion vectors, and calculate the average vector of all motion vectors within each motion cluster to obtain the local motion trend vector. Motion vectors whose motion direction angle is less than a preset angle and whose displacement magnitude ratio is within a preset range are grouped into the same motion cluster, and each motion cluster corresponds to a local motion unit of the target object. The horizontal component of the local motion trend vector is the arithmetic mean of the horizontal components of all motion vectors within the motion cluster, and the vertical component is the arithmetic mean of the vertical components of all motion vectors within the motion cluster.

[0049] In this embodiment, a clustering algorithm is used to process the set of motion vectors of feature points. The angle between the directions of any two motion vectors (calculated using the vector dot product formula) and the ratio of their displacement magnitudes (larger displacement value / smaller displacement value) are calculated. When the angle between the directions is less than a preset angle and the ratio is within a preset range, the two vectors are grouped into the same cluster. For each motion cluster, the horizontal components of all the motion vectors it contains are summed and divided by the number of vectors to obtain the horizontal component of the local motion trend vector; similarly, the vertical component is calculated to form the local motion trend vector.

[0050] Step S1224: Calculate the proportion of the number of feature points contained in each motion cluster to the total number of feature points in the target object region, and use it as the weight coefficient of the motion cluster. This weight coefficient reflects the contribution of the local motion unit to the overall motion of the target object.

[0051] In this embodiment, the number of stable feature points contained in each motion cluster obtained in step S1223 is counted. For example, if a motion cluster contains m feature points and the total number of feature points in the target object region is n, then the weight coefficient of the motion cluster is m / n. For the target object, motion clusters in certain regions usually contain more feature points and have larger weight coefficients, indicating that the motion in that region contributes more significantly to the overall motion.

[0052] Step S1225: Based on the spatial location distribution of each motion cluster, construct a spatial adjacency matrix. The element values ​​of the spatial adjacency matrix represent the distance between two motion clusters in the image space. The closer the distance, the larger the element value.

[0053] In this embodiment, the average coordinates of all feature points within each motion cluster are calculated and used as the center coordinates of that motion cluster. For a scenario containing k motion clusters, a k×k spatial adjacency matrix is ​​constructed. The element value in the p-th row and q-th column of the matrix is ​​determined as follows: the Euclidean distance between the center coordinates of the p-th and q-th motion clusters is calculated, the distance value is normalized to the range of 0-1, and then 1 is subtracted from the normalized distance value to obtain the element value. For example, some adjacent motion clusters of the target object are close to each other, and the corresponding matrix element values ​​are larger, reflecting a strong spatial correlation between the two.

[0054] Step S1226: Multiply the local motion trend vector, weight coefficients and the element values ​​of the corresponding rows in the spatial adjacency matrix element by element to obtain the weighted local motion features; the weighted local motion features retain the local motion trend while incorporating spatial correlation information.

[0055] In this embodiment, for the p-th motion cluster, its local motion trend vector is (e, f), the weight coefficient is g, and the elements of the p-th row of the spatial adjacency matrix are [h1, h2, ..., hk]. After multiplying e and g, and then multiplying each element by h1, h2, ..., hk, we obtain the horizontal weighted component; similarly, we process f to obtain the vertical weighted component. The combination of the two forms the weighted local motion feature of the motion cluster, which reflects both its own motion trend and its spatial association with other motion clusters.

[0056] Step S1227: Calculate the sum vector of weighted local motion features of all motion clusters to obtain the global motion trend vector of the target object; the horizontal component of the sum vector is the sum of the horizontal components of each weighted local motion feature, and the vertical component is the sum of the vertical components of each weighted local motion feature.

[0057] In this embodiment, the horizontal components of all weighted local motion features are collected and summed to obtain the horizontal component of the global motion trend vector; similarly, all vertical components are summed to obtain the vertical component. For example, if the sum of the weighted horizontal components of each local motion unit of the target object is i and the sum of the vertical components is j, then the global motion trend vector is (i, j), reflecting the overall motion direction and amplitude of the target object.

[0058] Step S1228: Perform residual calculations on all motion vectors in the feature point motion vector set and the global motion trend vector to obtain a residual vector set. The root mean square value of the residual vector set is used as the motion consistency index. The residual vectors in the residual vector set reflect the degree of deviation of local motion from the global trend; the smaller the root mean square value, the more consistent the motion of the target object.

[0059] In this embodiment, for each motion vector (e_p, f_p) in the feature point motion vector set, the horizontal component of the global motion trend vector is subtracted from e_p to obtain the residual horizontal component; the vertical component of the global motion trend vector is subtracted from f_p to obtain the residual vertical component, forming a residual vector. The sum of the squares of the horizontal and vertical components of all residual vectors is calculated, and the square root of the average value is taken to obtain the root mean square value as the motion consistency index. If the target object moves in a coordinated manner and the deviation between the local motion and the global trend is small, the index value is low.

[0060] Step S1229: The weighted local motion features, global motion trend vector, and motion consistency index of each motion cluster are fused to generate video motion features. The dimension of the video motion features is the sum of the total dimension of each weighted local motion feature, the dimension of the global motion trend vector, and the dimension of the motion consistency index.

[0061] In this embodiment, all weighted local motion features can be arranged in motion cluster order to form a vector with a length equal to the sum of the dimensions of each feature. Then, the global motion trend vector and motion consistency index are sequentially concatenated to form the final video motion features. For example, if the total dimension of the weighted local motion features of k motion clusters is l, the total dimension of the concatenated video motion features is l+2+1, which fully covers local motion, global trend, and motion consistency information.

[0062] Step S123: Input the audio basic features and video motion features into the feature complementation module, analyze the semantic overlap area of ​​the audio basic features and video motion features in the time dimension, and extract the audio temporal dynamic features.

[0063] In this embodiment, as an example, the feature complementation module employs a neural network structure comprising an input layer, hidden layers, and an output layer. Audio base features and video motion features are input into the input layer, which converts these two types of features into a format suitable for network processing. The hidden layer consists of multiple neurons that process the input features through connection weights between neurons, analyzing the temporal correlation between the two features. Specifically, it analyzes the semantic correlation between the target object's sound information reflected by the audio base features and the target object's motion information reflected by the video motion features within the same time interval, identifying semantically overlapping regions—areas where both audio and video information indicate the presence of specific activity of the target object within that time interval. Based on these semantically overlapping regions, the hidden layer further extracts features that reflect the dynamic temporal changes of the target object's sound and are associated with motion features. Finally, the output layer outputs these features, which are the audio temporal dynamic features.

[0064] Step S124: Perform target segmentation processing on the video frame sequence, locate the contour boundary of the target object and extract texture distribution features through the region growing algorithm, and generate video spatial structure features.

[0065] In this embodiment, as an example, for each frame in the video frame sequence, the region where the target object (a specific target object) may exist is first determined, and then a region growing algorithm is applied to process this region. Starting from a seed point (a pixel) within the target object region, adjacent pixels with similar characteristics (such as grayscale value, color, etc.) to the seed point are merged into the region where the seed point is located. This process continues until no new pixels can be merged, thereby segmenting the region of the target object. Based on the segmented region, the contour boundary of the target object is determined, and the texture distribution of pixels within the region is analyzed to extract texture features. The contour boundary information and texture distribution features are combined to generate video spatial structure features. In this embodiment, step S124 may include sub-steps S1241-S1248, which are described in detail below.

[0066] Step S1241: Perform Gaussian blur denoising processing on the current video frame of the video frame sequence, and use a Gaussian kernel of a preset size to perform a weighted average operation on the image pixels to generate a denoised frame.

[0067] In this embodiment, as an example, the preset size of the Gaussian kernel is M×M (M is an odd number). Each element in the Gaussian kernel has a corresponding weight value, which is determined according to the Gaussian distribution law, with the weight value at the center being the largest and gradually decreasing towards the edges. For each pixel in the current video frame, an M×M pixel region is taken centered on that pixel. Each pixel value within this region is multiplied by the weight value at the corresponding position in the Gaussian kernel. Then, all products are added together and divided by the sum of the weight values ​​to obtain the weighted average pixel value. This process is performed on all pixels in the current video frame, and the resulting image is the denoised frame. Gaussian blurring reduces noise in the image, making it smoother.

[0068] Step S1242: Perform contrast adaptive enhancement processing on the denoised frame. Adjust the image grayscale distribution by limiting the contrast adaptive histogram equalization algorithm to enhance the grayscale difference between the target object and the background area, and obtain the enhanced frame.

[0069] In this embodiment, as an example, the contrast-limited adaptive histogram equalization algorithm divides the denoised frame into multiple non-overlapping sub-blocks. For each sub-block, its gray-level histogram is calculated, and then the cumulative distribution function is calculated based on the histogram. During the calculation process, a contrast limit threshold is set. When the number of pixels at a certain gray level exceeds this threshold, the excess is evenly distributed to other gray levels to avoid over-enhancing noise. Based on the processed cumulative distribution function, the pixel gray values ​​within each sub-block are mapped to adjust the gray-level distribution of the sub-block, thereby enhancing the contrast within the sub-block. Then, the processing results of each sub-block are stitched together, and bilinear interpolation is used to process the sub-block boundaries to avoid obvious block artifacts, ultimately resulting in an enhanced frame. In the enhanced frame, the gray-level difference between the target object (a specific target object) and the background region is more pronounced, facilitating subsequent target segmentation processing.

[0070] Step S1243: Extract gradient magnitude images from the enhanced frame, calculate horizontal and vertical gradients using an edge detection algorithm, filter candidate seed points based on gradient magnitude non-maximum suppression, and generate a set of seed point coordinates.

[0071] In this embodiment, as an example, the Sobel operator can be used to calculate the horizontal and vertical gradients of the enhanced frame. For each pixel in the enhanced frame, convolution operations are performed with both the horizontal and vertical Sobel operators to obtain the horizontal gradient value Gx and the vertical gradient value Gy. The gradient magnitude of the pixel is calculated based on Gx and Gy, where the gradient magnitude is equal to the square root of the sum of the squares of Gx and Gy. The gradient direction is also calculated. The gradient magnitudes of all pixels are then combined to form a gradient magnitude image.

[0072] Non-maximum suppression (NMS) is applied to the gradient magnitude image. For each pixel, the gradient magnitudes of its two adjacent pixels are compared along its gradient direction. If the gradient magnitude of the pixel is greater than that of its two adjacent pixels, the gradient magnitude of the pixel is retained; otherwise, its gradient magnitude is set to zero, thereby refining the edges. In the image after NMS, pixels with larger gradient magnitudes that are located within the region where the target object may exist are selected as candidate seed points. The coordinates of these candidate seed points are recorded to form a seed point coordinate set. For example, the seed point coordinate set can be represented as (x1, y1), (x2, y2), ..., (xn, yn), where x and y represent the x-coordinate and y-coordinate of the pixel, respectively.

[0073] Step S1244: Using the seed point coordinate set as the starting point for growth, set the gray-level similarity threshold and region connectivity constraints, perform region growth operation, and merge adjacent pixels with gray-level differences less than the threshold into the initial segmentation region.

[0074] In this embodiment, as an example, the grayscale similarity threshold is set to T_gray, and the region connectivity constraint is 4-neighbor connectivity, meaning only the adjacent pixels in the four directions (up, down, left, and right) of the current pixel are considered. For each seed point in the seed point coordinate set, it is used as the initial growth point, and its adjacent pixels in the 4-neighborhood are checked. The grayscale difference between the adjacent pixel and the growth point is calculated. If the grayscale difference is less than T_gray, the adjacent pixel is merged into the current growth region and used as a new growth point to continue growing. The growth region is continuously expanded in the same way until there are no new pixels that meet the conditions to be merged. The regions grown from each seed point are merged, and the overlapping parts are removed to form the initial segmentation region, which roughly includes the region of the target object (the specific target object).

[0075] Step S1245: Perform morphological optimization processing on the initial segmented region by sequentially performing dilation and erosion operations to fill the internal holes of the region and eliminate boundary burrs, thereby generating an optimized segmented region.

[0076] In this embodiment, the dilation operation refers to traversing the target region in the image using a preset structuring element (such as a rectangle, circle, etc.). When the center of the structuring element coincides with a pixel in the target region, all pixels within the area covered by the structuring element are marked as target region pixels, thereby expanding the boundary of the target region outward. This achieves the effects of filling small holes inside the target, connecting adjacent target fragments, or increasing the area of ​​the target region. For example, in a binary image containing the outline of a target object, after the dilation operation, the edge of the target outline will expand outward, and the depressions or small breaks on the original outline may be filled, making the target region more complete. The erosion operation is the opposite of the dilation operation. It also uses a preset structuring element to traverse the image. Only when the structuring element is completely contained within the target region is the pixel corresponding to the center of the structuring element retained as a target region pixel; otherwise, it is marked as a background pixel. This causes the boundary of the target region to shrink inward, achieving the effects of removing small burrs on the target edge, separating mutually adhering targets, or reducing the area of ​​the target region. For example, in a binary image containing the outline of a target object, after the erosion operation, the edges of the target outline will shrink inward, and the small burrs that originally protruded on the outline will be eliminated. If there are two adjacent targets, they may be separated into independent regions.

[0077] As an example of this embodiment, morphological operations are performed using structuring elements of a preset shape and size, such as 3×3 rectangular structuring elements. First, a dilation operation is performed, convolving the initial segmented region with the structuring element. For each pixel within the initial segmented region, all pixels within the coverage area of ​​the structuring element are included in the region, causing the region boundary to expand outwards and fill small holes within the region. After the dilation operation, an erosion operation is performed, again using 3×3 rectangular structuring elements. The dilated region is convolved with the structuring element. For each pixel within the region, it is retained only if all pixels within the coverage area of ​​the structuring element belong to that region; otherwise, it is removed, causing the region boundary to shrink inwards and eliminating burrs and small protrusions on the boundary. The region obtained after the dilation and erosion operations is the optimized segmented region, which more accurately reflects the actual area of ​​the target object.

[0078] Step S1246: Extract the boundary pixel set of the optimized segmentation region, perform polygon fitting on the boundary pixel set, simplify the contour curve and retain key feature points to obtain the contour boundary descriptor of the target object.

[0079] In this embodiment, as an example, a boundary tracking algorithm is used to extract the set of boundary pixels of the optimized segmentation region. For example, the Douglas-Peucker algorithm can be used to start from a certain edge pixel of the optimized segmentation region, track the boundary of the region in a clockwise direction, record the coordinates of each boundary pixel, until the starting point is returned, and form a set of boundary pixels. For example, the set of boundary pixels can be represented as (bx1,by1), (bx2,by2), ..., (bxm,bym).

[0080] The boundary pixel set is input into the Douglas-Peucker algorithm for polygon fitting. First, the start and end points of the boundary pixel set are determined, and the distances from all other points to the lines connecting the start and end points are calculated. The point with the largest distance is found. If this distance is greater than a preset threshold T_dist, this point is considered a key feature point, and the boundary is divided into two segments using this point as the boundary. The above operation is repeated for each segment. If the distance is less than or equal to T_dist, all points between the start and end points are discarded, and the line connecting the start and end points is used to replace this segment of the boundary. This process is iterated continuously until a polygon composed of key feature points is obtained, which is a simplified representation of the target object's contour boundary. The coordinates of these key feature points are arranged in order to form the contour boundary descriptor of the target object.

[0081] Step S1247: Calculate the gray-level co-occurrence matrix within the optimized segmentation region, and extract the energy, entropy, contrast, and correlation parameters of the matrix to form a texture distribution feature vector.

[0082] In this embodiment, as an example, the grayscale values ​​of the optimized segmented region are first quantized, dividing the grayscale value range into several levels to reduce computational load. Then, within the optimized segmented region, the grayscale co-occurrence matrix is ​​calculated at a preset distance d in four directions: 0°, 45°, 90°, and 135°. The grayscale co-occurrence matrix is ​​a two-dimensional matrix, where matrix elements (i, j) represent two pixels at a distance d in a certain direction, and their grayscale values ​​are the probabilities of occurrence of the i-th and j-th levels, respectively.

[0083] For each direction of the gray-level co-occurrence matrix, calculate its energy, entropy, contrast, and correlation parameters. Energy is the sum of squares of all elements in the gray-level co-occurrence matrix, reflecting the uniformity and coarseness of the image texture; entropy is the sum of the negatives of the products of all elements in the gray-level co-occurrence matrix and the logarithm of that element, reflecting the complexity of the image texture; contrast is the sum of the products of the squares of elements (i,j) and (ij) in the gray-level co-occurrence matrix, reflecting the sharpness and depth of the image texture; correlation is the ratio of the product of the covariance of element (i,j) and i and j divided by their respective standard deviations, reflecting the linear correlation of gray values ​​in the image texture.

[0084] The energy, entropy, contrast, and correlation parameters extracted from the gray-level co-occurrence matrices in the four directions are arranged in the order of 0°, 45°, 90°, and 135° to form a multi-dimensional texture distribution feature vector. For example, the texture distribution feature vector can be represented as [energy 0°, entropy 0°, contrast 0°, correlation 0°, energy 45°, entropy 45°, contrast 45°, correlation 45°, energy 90°, entropy 90°, contrast 90°, correlation 90°, energy 135°, entropy 135°, contrast 135°, correlation 135°].

[0085] Step S1248: Concatenate the contour boundary descriptor with the texture distribution feature vector to generate a video spatial structure feature containing spatial shape information and texture detail information. The dimension of the video spatial structure feature is the sum of the dimension of the contour boundary descriptor and the dimension of the texture distribution feature vector.

[0086] In this embodiment, as an example, the contour boundary descriptor consists of the coordinates of n key feature points, each coordinate containing two values, x and y. Therefore, the dimension of the contour boundary descriptor is 2n. The dimension of the texture distribution feature vector is 16 (four directions, four parameters in each direction). All elements of the contour boundary descriptor are arranged sequentially, and then all elements of the texture distribution feature vector are sequentially arranged after the contour boundary descriptor elements to form a new feature vector. This feature vector is the video spatial structure feature, with a dimension of 2n+16, containing both the spatial shape information of the target object and the texture detail information.

[0087] Step S125: Align the audio temporal dynamic features with the video spatial structure features to ensure that the feature vector lengths of the two are consistent, thus forming a multimodal feature set.

[0088] In this embodiment, as an example, the lengths of the audio temporal dynamic feature vector L_audio and the video spatial structure feature vector L_video are first determined. If L_audio is not equal to L_video, dimension alignment is required. If L_audio is greater than L_video, principal component analysis is used to reduce the dimensionality of the audio temporal dynamic feature vector, selecting principal components that reflect the main information of the original features, so that the length of the reduced audio temporal dynamic feature vector is L_video. If L_audio is less than L_video, interpolation is used to increase the dimensionality of the audio temporal dynamic feature vector, inserting appropriate values ​​between the elements of the feature vector, so that the length of the increased audio temporal dynamic feature vector is L_video.

[0089] After dimensional alignment, the audio temporal dynamic feature vector and the video spatial structure feature vector have the same length. These two feature vectors are combined in sequence to form a multimodal feature set, which contains feature information from both audio and video, providing a foundation for subsequent temporal correlation modeling.

[0090] Step S130: Input the multimodal feature set into the pre-trained association enhancement network to perform temporal association modeling and generate an association feature sequence with cross-modal semantic alignment.

[0091] In this embodiment, as an example, the pre-trained association enhancement network is a deep learning-based neural network model. This model has been trained with a large amount of labeled data and has the ability to perform temporal association modeling of multimodal features. The multimodal feature set generated in step S120 is input into this network. The network processes through multiple layers to mine the correlation between audio temporal dynamic features and video spatial structural features in the time series, and achieves semantic alignment of the two modal features. Finally, it outputs an association feature sequence, in which each feature contains aligned cross-modal semantic information. In this embodiment, step S130 may include sub-steps S131-S135, which are described in detail below.

[0092] Step S131: Input the multimodal feature set into the local association layer of the association enhancement network, calculate the local similarity between the audio temporal dynamic features and the video spatial structure features at adjacent time steps, and generate local association features.

[0093] In this embodiment, as an example, the local association layer of the association enhancement network includes two parallel feature mapping sublayers and one similarity calculation sublayer. The audio temporal dynamic features and video spatial structure features from the multimodal feature set are respectively input into the two feature mapping sublayers. The feature mapping sublayers map the input features to the same feature space through linear transformation, obtaining the mapped audio feature A_t and the mapped video feature V_t, where t represents a time step. A time step is the smallest processing unit divided according to time sequence when processing temporal data (such as consecutive frames of audio / video or audio segments). Each time step corresponds to a continuous data segment of a specific duration, used to reflect the progressive relationship of data in the time dimension, facilitating the analysis of the dynamic changes of the target object.

[0094] The similarity calculation sublayer computes the local similarity between mapped audio and video features at adjacent time steps (e.g., time steps t and t+1). For time step t, the similarity S_t between A_t and V_t is calculated; for time step t+1, the similarity S_t+1 between A_{t+1} and V_{t+1} is calculated, along with the similarities between A_t and V_{t+1} and between A_{t+1} and V_t. These similarity values ​​are then fused with the corresponding features to generate locally correlated features, which reflect the local association relationships between audio and video features at adjacent time steps.

[0095] Step S132: Perform long-range dependency analysis on the local association features through the global context layer of the association enhancement network to extract the consistent behavioral patterns of the target object within a continuous time window.

[0096] In this embodiment, as an example, the global context layer of the association enhancement network adopts the self-attention mechanism in the Transformer architecture. Local association features are arranged in chronological order to form a feature sequence. A continuous time window of size W is set, meaning that local association features from W consecutive time steps are processed each time.

[0097] Within each time window, a self-attention mechanism is used to calculate the attention weights between the local correlation features of each time step and the local correlation features of all other time steps within the window. These attention weights reflect the dependencies between features at different time steps. The local correlation features within the window are then weighted and summed based on these attention weights to obtain the global context feature for each time step. This feature contains long-range dependency information within the window. In this way, the consistency features of the target object's behavioral patterns within consecutive time windows are extracted, i.e., the coherence and consistency information between behavioral patterns at different time steps.

[0098] Step S133: Using the semantic alignment module of the association enhancement network, the local association features and global context features are semantically aligned based on the prior feature template of the target object to obtain semantically aligned features, thereby realizing the semantic alignment between the local association features and global context features and eliminating the semantic offset between modalities.

[0099] In this embodiment, as an example, the semantic alignment module compares the local related features and global context features with the prior feature template of the target object to make the two semantically consistent.

[0100] In this embodiment, step S133 may include sub-steps S1331-S1335, which will be described in detail below.

[0101] Step S1331: Extract audio feature templates and video feature templates of the target object from the historical audio and video data during the training phase to construct a prior feature template library; the historical audio and video data consists of multiple groups of labeled audio and video stream data containing the target object, each group of audio and video stream data containing a synchronously acquired continuous audio signal sequence and video frame sequence; the audio feature templates are extracted after performing time-frequency domain transformation processing on the historical audio signal sequence; the video feature templates are extracted after performing target perception processing on the historical video frame sequence.

[0102] In this embodiment, as an example, the historical audio and video data during the training phase consists of multiple groups of labeled audio and video stream data containing specific target objects (target objects). Each group of data is labeled with the location and activity information of the target object. For historical audio signal sequences, audio features are extracted using a time-frequency domain transformation processing method similar to step S121. These audio features are then clustered and averaged to obtain an audio feature template that can represent the audio characteristics of the target object.

[0103] For historical video frame sequences, target-aware processing is performed, including target detection and feature extraction. Video features are extracted, and clustering and averaging are also performed to obtain video feature templates. Audio and video feature templates are classified and stored according to category and feature type to construct a priori feature template library. For example, the priori feature template library contains audio and video feature templates of target objects from different angles and under different activity states.

[0104] Step S1332: Calculate the similarity between the local association features and the global context features and the audio feature templates and video feature templates in the prior feature template library, respectively, to obtain the intramodal matching degree; the local association features are the features output by the local association layer of the association enhancement network, containing the local similarity information between audio temporal dynamic features and video spatial structure features at adjacent time steps; the global context features are the features output by the global context layer of the association enhancement network, containing the long-range behavioral pattern consistency information between audio temporal dynamic features and video spatial structure features within a continuous time window.

[0105] In this embodiment, as an example, cosine similarity is used to calculate the similarity between local associated features and audio and video feature templates in the prior feature template library, as well as the similarity between global context features and audio and video feature templates. For the audio portion of the local associated features, cosine similarity is calculated with the audio feature template to obtain the audio intramodal matching degree; for the video portion of the local associated features, cosine similarity is calculated with the video feature template to obtain the video intramodal matching degree. Similarly, for the audio and video portion of the global context features, similarity is calculated with the corresponding templates to obtain the corresponding intramodal matching degrees.

[0106] Step S1333: Analyze the differences in matching degree within each modality and identify the offset direction of the audio features and video features of the target object in semantic expression.

[0107] In this embodiment, as an example, the matching degree within the audio modality and the matching degree within the video modality are compared. If the matching degree within the audio modality is significantly lower than that within the video modality, it indicates that the audio features deviate from the prior template in terms of semantic expression, and the direction of deviation is the audio feature deviation; if the matching degree within the video modality is significantly lower than that within the audio modality, it indicates that the video features deviate from the prior template in terms of semantic expression, and the direction of deviation is the video feature deviation; if both are significantly lower than a preset matching degree threshold, it indicates that both modal features are deviated.

[0108] Step S1334: Through the parameter adjustment mechanism of the semantic alignment module, the features in the offset direction are weighted and corrected to enhance the semantic consistency between modalities. The parameter adjustment mechanism can be implemented through the adjustment function preset in the semantic alignment module, which includes feature weight adjustment and feature value correction operations. If the offset direction is an audio feature deviation, the weight of the audio part features in the local association features is reduced, the weight of the video part features is increased, and the dimension of the audio part features that deviates from the feature frequency range or energy fluctuation curve is corrected. If the offset direction is a video feature deviation, the weight of the video part features in the local association features is reduced, the weight of the audio part features is increased, and the dimension of the video part features that has insufficient contour boundary continuity or abnormal texture distribution is corrected. If the offset direction is a deviation of both modal features, the weight adjustment and feature value correction are performed on the audio and video part features simultaneously, and the modal features with lower matching degree are corrected first.

[0109] In this embodiment, as an example, the feature weight adjustment in the parameter adjustment mechanism is achieved by setting weight coefficients. For example, if the audio features deviate, the weight coefficient of the audio part features is set to w_audio (w_audio<1), and the weight coefficient of the video part features is set to w_video (w_video>1), and w_audio+w_video=2 (ensuring that the total weight remains unchanged). For the dimensions of the audio part features that deviate from the feature frequency range or energy fluctuation curve, their values ​​are adjusted to the values ​​of the corresponding dimensions in the prior audio feature template. The adjustment range is determined according to the degree of deviation; the greater the deviation, the greater the adjustment range.

[0110] If the video features deviate, set the weight coefficient of the video features to w_video' (w_video'<1) and the weight coefficient of the audio features to w_audio' (w_audio'>1), with w_audio'+w_video'=2. For dimensions in the video features with insufficient contour boundary continuity or abnormal texture distribution, correct them to the corresponding dimensions in the prior video feature template. If both modal features deviate, compare their matching degree, and apply a larger weight adjustment and feature value correction to the modality with lower matching degree.

[0111] Step S1335: Input the corrected local association features and global context features into the classifier of the association enhancement network to verify the semantic alignment effect until the intermodal matching degree difference meets the requirements; the classifier is a pre-trained multimodal classifier in the association enhancement network, and the corrected local association features and global context features are input.

[0112] Furthermore, the category prediction results of the target object can be output; the alignment effect is verified by comparing the matching results of the category prediction results with the real labels. If the difference between the audio modality matching degree and the video modality matching degree is still significant, the parameter adjustment mechanism is returned to readjust the feature weights and feature values; if the difference is reduced to an acceptable range and the category prediction matching meets the requirements, the semantic alignment is determined to be complete, and the corrected local association features and global context features are output as the semantically aligned features.

[0113] In this embodiment, as an example, the classifier of the association enhancement network is a pre-trained multimodal classification model whose input layer can receive corrected local association features and global context features. After these two features are input into the classifier, the classifier processes them through its internal neural network structure and finally outputs the category prediction result of the target object, such as the probability that the object corresponding to the feature is a "specific target object". The ground truth label is the pre-labeled object category corresponding to the feature, i.e., "specific target object". The semantic alignment effect is verified by comparing the matching degree between the probability of "specific target object" in the category prediction result and the ground truth label. At the same time, the difference between the audio modality matching degree and the video modality matching degree is calculated. If the difference is greater than the preset difference threshold, it means that the matching degree difference between the two modalities is still significant, and the parameter adjustment mechanism needs to be returned to readjust the feature weights and feature values; if the difference is less than or equal to the difference threshold, and the probability of "specific target object" in the category prediction result reaches the preset matching threshold, then the semantic alignment is determined to be complete, and the corrected local association features and global context features are used as the semantically aligned feature output.

[0114] Step S134: Input the semantically aligned features into the temporal fusion layer of the association enhancement network, and model the temporal evolution of the feature sequence through gated recurrent units to generate a temporally coherent associated feature sequence.

[0115] In this embodiment, as an example, the temporal fusion layer of the association enhancement network includes a gated recurrent unit, which consists of an input gate, a forget gate, and an output gate. Semantically aligned features are sequentially input into the gated recurrent unit in chronological order. The input gate determines which information in the current feature needs to be retained, the forget gate determines which information in the previous hidden states needs to be forgotten, and the output gate determines the output hidden state based on the current input and the hidden state. Through the synergistic effect of these three gates, the gated recurrent unit can capture the temporal dependencies of the feature sequence and model the temporal evolution of the features. For example, for a feature sequence of a specific target object moving within a target environment, the gated recurrent unit can remember its previous motion and sound features and, combined with the current features, generate a hidden state that reflects its continuous motion and sound changes. Arranging these hidden states in chronological order forms a temporally coherent sequence of associated features.

[0116] Step S135: Normalize the associated feature sequence to balance the contribution weights of different modal features and form an associated feature sequence with cross-modal semantic alignment.

[0117] In this embodiment, as an example, the min-max normalization method is used to process the associated feature sequence. First, the maximum and minimum values ​​of each feature dimension in the associated feature sequence are found. Then, for each feature value, it is transformed to a value between 0 and 1 by calculating (feature value - minimum value) / (maximum value - minimum value). This normalization process ensures that the audio modal features and video modal features maintain a consistent numerical range, avoiding the situation where a certain modal feature value is too large and dominates the entire feature sequence, thus balancing the contribution weights of different modal features. The processed associated feature sequence maintains the cross-modal semantic alignment characteristic and allows each modal feature to participate equally in subsequent tracking processing, forming the final associated feature sequence with cross-modal semantic alignment.

[0118] Step S140: Construct a tracking stability evaluation model based on the associated feature sequence, and output a stability index that reflects the reliability of the tracking process.

[0119] In this embodiment, as an example, the tracking stability evaluation model takes the associated feature sequence as input and evaluates the reliability of the tracking process by analyzing the consistency and continuity of features over time, as well as the matching degree of audio and video modal features. The model internally includes multiple processing modules, which are used to calculate feature consistency parameters, temporal continuity parameters, and cross-modal matching degree parameters, respectively. These parameters are then combined to obtain an index reflecting tracking stability. For example, when the features of a specific target object change little in consecutive time steps, it indicates that the tracking is relatively stable and the stability index is high; when the feature changes significantly or the audio and video features do not match, the stability index is low. In this embodiment, step S140 may include sub-steps S141-S145, which are described in detail below.

[0120] Step S141: Extract the feature consistency parameter of the target object from the associated feature sequence. The feature consistency parameter represents the degree of feature preservation of the target object in consecutive time steps.

[0121] In this embodiment, as an example, the cosine similarity between feature vectors Ft and Ft+1 corresponding to two consecutive time steps t and t+1 in the associated feature sequence is calculated. The cosine similarity is calculated by dividing the dot product of the two vectors by the product of their magnitudes. The closer the result is to 1, the more similar the two feature vectors are, and the higher the feature consistency. The cosine similarities between consecutive time steps are arranged in chronological order to form a similarity sequence, and then the average value of this sequence is calculated and used as the feature consistency parameter. For example, for a specific target object in consecutive video frames and audio segments, the higher the average cosine similarity of its feature vectors, the better its feature preservation, and the larger the feature consistency parameter.

[0122] Step S142: Analyze the temporal continuity parameter of the associated feature sequence, where the temporal continuity parameter represents the smoothness of the target object's position change.

[0123] In this embodiment, as an example, the position coordinates of the target object at each time step are extracted from the associated feature sequence to obtain the position sequence Pt (t=1,2,...n). The position change ΔPt = Pt+1-Pt between two adjacent time steps is calculated, and then the variance of the above change is calculated. The smaller the variance, the smoother the position change and the better the temporal continuity. The reciprocal of the variance is used as the temporal continuity parameter, i.e., temporal continuity parameter = 1 / var(ΔPt), where var(ΔPt) represents the variance of the position change sequence. For example, when a specific target object moves smoothly within the target environment, the variance of the position change is small, and the temporal continuity parameter is large; when the target object suddenly changes its direction or speed, the variance of the position change is large, and the temporal continuity parameter is small.

[0124] Step S143: Calculate the cross-modal matching degree parameter of the associated feature sequence, wherein the cross-modal matching degree parameter represents the degree of agreement between audio features and video features when describing the same target.

[0125] In this embodiment, as an example, the cross-modal matching degree parameter is calculated by analyzing the temporal matching of audio and video features in the associated feature sequence. The specific calculation process will be described in detail in subsequent sub-steps. The larger the value of this parameter, the higher the degree of matching between the audio and video features in describing a specific target object, and vice versa. In this embodiment, step S143 may include sub-steps S1431-S1435, which will be described in detail below.

[0126] Step S1431: Extract the audio temporal dynamic features and video spatial structure features at the same time step in the associated feature sequence.

[0127] In this embodiment, as an example, in the associated feature sequence, the features of each time step include audio temporal dynamic features and video spatial structure features. For time step t, the corresponding audio temporal dynamic features A_t and video spatial structure features V_t are extracted, where A_t is a multi-dimensional vector containing information such as the frequency distribution and energy fluctuations of the audio at that time step; V_t is also a multi-dimensional vector containing information such as the contour and texture of the target object in the video at that time step.

[0128] Step S1432: Input the audio temporal dynamic features and video spatial structure features into the cross-modal matching network, and calculate the semantic similarity between the audio temporal dynamic features and video spatial structure features through dot product operation.

[0129] In this embodiment, as an example, the input layer of the cross-modal matching network receives audio temporal dynamic features A_t and video spatial structure features V_t. First, the dimensions of these two features are adjusted to ensure they are identical. Then, the adjusted features are subjected to a dot product operation, that is, the sum of the products of corresponding dimensional elements of A_t and V_t is calculated to obtain a numerical value S_t. The magnitude of this value S_t reflects the semantic similarity between the two features; the larger S_t is, the higher the semantic similarity between the audio temporal dynamic features and the video spatial structure features at that time step.

[0130] Step S1433: Perform sliding window averaging on the semantic similarity corresponding to consecutive time steps to obtain the average similarity and eliminate local noise interference.

[0131] In this embodiment, as an example, the size of the sliding window is set to W, and the window slides sequentially across the semantic similarity sequence S_t (t=1,2,...n). For each window position, the average of all semantic similarities within the window is calculated to obtain the average similarity M_k, where k is the index of the window. For example, when the window covers time steps t1 to t1+W-1, M_k=(S_t1+S_t1+1+...+S_t1+W-1) / W. This sliding window averaging process can smooth out local fluctuations in the semantic similarity sequence, eliminate outliers caused by noise, and make the obtained average similarity more reflective of the overall semantic matching trend.

[0132] Step S1434: Analyze the changing trend of the average similarity to identify the stable and fluctuating regions of the target object in cross-modal description.

[0133] In this embodiment, as an example, the average similarity sequence M_k is plotted as a curve in chronological order, and the trend of the curve is analyzed. When the fluctuation range of the average similarity within a certain interval is less than a preset stability threshold, the time range corresponding to that interval is determined as a stable region, indicating that the audio and video features describe the target object relatively consistently within this region. When the fluctuation range of the average similarity is greater than the stability threshold, the corresponding time range is a fluctuating region, indicating that there are significant differences in the descriptions of the two modal features. For example, when a specific target object is walking quietly, the average similarity curve fluctuates less, which is a stable region; when sudden noise occurs in the surrounding environment, the curve fluctuates more, which is a fluctuating region.

[0134] Step S1435: Use the proportion of the stable region as the cross-modal matching degree parameter.

[0135] In this embodiment, as an example, the ratio of the total time length of the stable region to the total time length corresponding to the entire associated feature sequence is calculated, and this ratio is used as the cross-modal matching degree parameter C. That is, C = (total time length of the stable region) / (total time length). For example, if the time length corresponding to the entire associated feature sequence is T, and the total time length of the stable region is T_s, then C = T_s / T. The larger the value of C, the higher the proportion of the stable region in the entire time series, and the better the match between the audio and video features.

[0136] Step S144: Input the feature consistency parameter, temporal continuity parameter and cross-modal matching degree parameter into the stability evaluation function, and calculate the stability index by weighted summation.

[0137] In this embodiment, as an example, the stability evaluation function is the weighted sum of these three parameters, that is, the stability index SI = w1×FC + w2×TC + w3×CM, where FC is the feature consistency parameter, TC is the temporal continuity parameter, CM is the cross-modal matching degree parameter, w1, w2, and w3 are the weights of these three parameters respectively, and w1 + w2 + w3 = 1. The above weights are preset according to the influence degree of different parameters on the tracking stability. For example, since the feature consistency parameter has a greater impact on the tracking stability, a relatively large w1 can be set. Through this way of weighted summation, the three parameters are combined to obtain a stability index SI that can comprehensively reflect the reliability of the tracking process. The larger the value of SI, the more reliable the tracking process is.

[0138] Step S145: Perform a dynamic threshold judgment on the stability index to identify possible drift risk points during the tracking process and record the risk position information.

[0139] In this embodiment, as an example, a dynamic threshold Th is preset, and this threshold will be adaptively adjusted according to the changes in the tracking environment. For example, the threshold is appropriately reduced in a complex environment and appropriately increased in a simple environment. Compare the stability index SI with the dynamic threshold Th. When SI < Th, it indicates that the current tracking may have drifted, and the corresponding time step is the drift risk point. For each drift risk point, extract the position coordinates of the target object at this time step from the associated feature sequence and record it as the risk position information for subsequent key calibration of the above position.

[0140] Step S150: Dynamically adjust the tracking parameters according to the stability index and calibrate the initial tracking result, and output the optimized object tracking trajectory.

[0141] In this embodiment, as an example, when the stability index is high, it indicates that the initial tracking result is relatively reliable and there is no need to make large adjustments to the tracking parameters; when the stability index is low, according to the previously recorded risk position information, the tracking parameters are adjusted specifically, such as the weight of position prediction, the threshold of feature matching, etc., and then the initial tracking result is calibrated using the adjusted parameters to correct possible position deviations. Finally, the optimized object tracking trajectory is obtained, and this trajectory can more accurately reflect the movement path of a specific target object in the target environment. In this embodiment, step S150 may include sub-steps S151 - S155, which will be described in detail below.

[0142] Step S151: Obtain the initial tracking trajectory generated based on the initialization tracking algorithm, and the initial tracking trajectory includes the position coordinate sequence of the target object in the video frame sequence.

[0143] In this embodiment, as an example, the initialization tracking algorithm can be the Meanshift algorithm, the Camshift algorithm, etc. The video frame sequence is input into the initialization tracking algorithm, and the algorithm matches and tracks the features of the target object in the initial frame to obtain the position coordinates of the target object in each video frame. The above coordinates are arranged in chronological order to form an initial tracking trajectory. For a specific target object, the sequence of position coordinates P_init = [P1, P2,..., Pn] in the initial tracking trajectory, where Pi represents the position coordinates of the target object in the i-th video frame.

[0144] Step S152: When the stability index is lower than the preset level, extract the audio time-domain dynamic features and video spatial structure features corresponding to the risk positions in the associated feature sequence to obtain risk position features.

[0145] In this embodiment, as an example, the preset level is a fixed threshold Th_init. When the stability index SI < Th_init, it indicates that there may be a large deviation in the initial tracking result. At this time, according to the risk position information recorded in step S145, find the corresponding time step, extract the above audio time-domain dynamic features and video spatial structure features of this time step from the associated feature sequence, and combine the above features to form risk position features F_risk. F_risk contains the audio and video feature information of the target object at the risk position and is used for subsequent parameter adjustment.

[0146] Step S153: Use the risk position features to construct a parameter adjustment model, and optimize the position prediction weights of the initialization tracking algorithm through the backpropagation algorithm to generate adjusted tracking parameters.

[0147] In this embodiment, as an example, the parameter adjustment model is a simple neural network model, whose input is the risk position feature F_risk, and the output is the adjustment amount of the position prediction weight of the initialization tracking algorithm. Input the risk position feature into this model, and the model calculates through the internal neuron connections to obtain the weight adjustment amount. Then, use the backpropagation algorithm to continuously adjust the parameters of the model according to the error between the initial tracking trajectory and the true position (which can be obtained by manually annotating a small number of key frames), so that the output weight adjustment amount can effectively reduce the tracking error. Add the original position prediction weight of the initialization tracking algorithm and the optimized weight adjustment amount to obtain the adjusted tracking parameters, and the above parameters include the proportional coefficient of position prediction, the weight of feature matching, etc.

[0148] Step S154: Perform weighted fusion processing on the adjusted tracking parameters and the position coordinate sequence of the initial tracking trajectory to correct the position deviation caused by feature drift.

[0149] In this embodiment, as an example, the weights of the adjusted tracking parameters and the initial tracking trajectory are determined based on the magnitude of the stability index. The lower the stability index, the greater the weight of the adjusted tracking parameters and the smaller the weight of the initial tracking trajectory. Through weighted fusion, the position predicted by the adjusted tracking parameters is combined with the position of the initial tracking trajectory to correct the position deviation caused by feature drift, resulting in a more accurate position coordinate sequence. In this embodiment, step S154 may include sub-steps S1541-S1545, which are described in detail below.

[0150] Step S1541: Assign dynamic weights to the adjusted tracking parameters and the initial tracking trajectory respectively. The dynamic weights are adaptively adjusted according to the stability index.

[0151] In this embodiment, as an example, let the weight corresponding to the adjusted tracking parameters be w_adjust, and the weight corresponding to the initial tracking trajectory be w_init, and w_adjust + w_init = 1. The dynamic weight allocation rule is: when the stability index SI is high, w_init is large and w_adjust is small; when SI is low, w_adjust is large and w_init is small. For example, it can be calculated by w_adjust = (Th_max - SI) / (Th_max - Th_min), w_init = 1 - w_adjust, where Th_max and Th_min are the maximum and minimum possible values ​​of the stability index, respectively. In this way, the weights are adaptively adjusted according to the stability index.

[0152] Step S1542: Calculate the weighted average of the predicted position corresponding to the adjusted tracking parameters and the measured position of the initial tracking trajectory to generate intermediate position coordinates.

[0153] In this embodiment, as an example, the adjusted tracking parameters predict the target object's position coordinates at each time step, P_adjust=[P_adjust1,P_adjust2,...,P_adjustn], based on risk location characteristics. The measured position coordinates of the initial tracking trajectory are P_init=[P1,P2,...,Pn]. For each time step i, the intermediate position coordinate P_midi=w_adjust×P_adjusti+w_init×Pi. By combining the predicted and measured positions through this weighted average calculation, the intermediate position coordinates are obtained, initially correcting the position deviation.

[0154] Step S1543: Extract the motion trend features of the target object in the associated feature sequence and analyze the consistency between the intermediate position coordinates and the motion trend.

[0155] In this embodiment, as an example, information such as the displacement direction and velocity changes of the target object in continuous time steps are extracted from the associated feature sequence to form a motion trend feature T_trend. For example, T_trend can represent the average movement direction and average velocity of the target object over a period of time. For the intermediate position coordinate sequence P_mid=[P_mid1,P_mid2,...,P_midn], the displacement vector ΔP_midi=P_midi+1-P_midi for each time step is calculated, and the consistency between the above displacement vector and the motion trend feature T_trend is analyzed, such as whether the displacement direction matches the average movement direction and whether the velocity magnitude is near the average velocity.

[0156] Step S1544: If the intermediate position coordinates are inconsistent with the movement trend, the intermediate position coordinates are corrected a second time based on the movement trend features to compensate for the position deviation caused by feature drift.

[0157] In this embodiment, as an example, the secondary corrected position coordinates are all accompanied by corresponding timestamps, which correspond to the time of the video frame sequence. All the secondary corrected position coordinates are arranged sequentially according to the timestamps from earliest to latest, forming an ordered sequence, which is the corrected position coordinate sequence. Each coordinate in this sequence accurately corresponds to the position of a specific target object within the target environment at the corresponding time point, laying the foundation for subsequently generating a smooth tracking trajectory.

[0158] Step S1545: Arrange the corrected position coordinates in chronological order to form a corrected position coordinate sequence.

[0159] In this embodiment, as an example, each position coordinate after secondary correction corresponds to a specific time point in the video frame sequence. This time point can be determined by the timestamp of the video frame; for example, t1, t2, t3…tn correspond to different video frame times. After obtaining all the secondary corrected position coordinates, they are arranged in ascending order according to their corresponding timestamps. For example, the position coordinates corresponding to time t1 (x1, y1), time t2 (x2, y2), time t3 (x3, y3),…, are arranged sequentially to form an ordered sequence [(x1, y1), (x2, y2), (x3, y3),…, (xn, yn)]. This sequence is the corrected position coordinate sequence. This sequence accurately reflects the position information of the target object at different time points, laying the foundation for the subsequent generation of optimized tracking trajectories.

[0160] Step S155: Perform smoothing filtering on the corrected position coordinate sequence to eliminate local abnormal fluctuations and generate a temporally coherent optimized object tracking trajectory.

[0161] In this embodiment, as an example, a moving average filtering algorithm is used to process the corrected position coordinate sequence. A sliding window of size N (N is an integer greater than 1) is set, containing N consecutive position coordinates. Starting from the beginning of the position coordinate sequence, the abscissas of the N position coordinates within the window are averaged to obtain the smoothed abscissa corresponding to that window; the ordinates of the N position coordinates within the window are averaged to obtain the smoothed ordinate corresponding to that window. Then, the window is moved forward one position, and the above calculation process is repeated until the entire position coordinate sequence is processed. In this way, positional deviations caused by local abnormal fluctuations in the sequence can be eliminated, making the obtained position coordinate sequence more consistent in time. Connecting the position coordinate sequences after smoothing and filtering in chronological order generates a temporally consistent optimized object tracking trajectory, which can accurately reflect the movement path of a specific target object within the target environment.

[0162] Based on the above, the training process of the correlation enhancement network and the tracking stability evaluation model is briefly described in the embodiments of this application as follows.

[0163] First, regarding the training of the association enhancement network, a large amount of preprocessed audio and video data containing the movement of different target objects within the target environment can be collected as the training dataset. Cross-modal feature collaborative extraction is performed on this dataset to generate a multimodal feature set as input. At the same time, the associated feature sequence is labeled according to the actual movement trajectory of the target object and the feature association situation as the expected output. The initial parameters of the network are set, and the input data is input into the network. The predicted associated feature sequence is obtained by processing through the local association layer, the global context layer, the semantic alignment module, and the temporal fusion layer. The loss value between the prediction result and the expected output is calculated, and the network parameters are adjusted using the backpropagation algorithm. The training is iteratively continued until the loss value reaches the preset threshold.

[0164] Secondly, regarding the training of the tracking stability assessment model, a large number of associated feature sequences of the tracked target object within the target environment and corresponding tracking stability annotation data (determined based on the degree of deviation between the tracking results and the actual situation) can be collected as training data. The associated feature sequences are input into the initial model, and the model calculates the feature consistency parameter, temporal continuity parameter, and cross-modal matching degree parameter. The predicted stability index is obtained through the stability assessment function. The error between the predicted index and the annotation data is calculated, and the model parameters are adjusted using the gradient descent algorithm. The model is repeatedly trained until the error is within an acceptable range.

[0165] The training of the aforementioned correlation enhancement network and tracking stability evaluation model can be achieved using training methods known in the art, and this embodiment does not limit them.

[0166] like Figure 3The diagram shown is a schematic of an intelligent audio-visual object tracking optimization system combining deep learning, provided in an embodiment of this application. The intelligent audio-visual scene switching optimization system includes components such as a processor, a machine-readable storage medium, and input / output devices. The machine-readable storage medium is connected to the processor, and is used to store programs, instructions, or code. The processor is used to execute the programs, instructions, or code in the machine-readable storage medium to implement the aforementioned intelligent audio-visual object tracking optimization method combining deep learning. The intelligent audio-visual scene switching optimization system can be... Figure 2 The audio and video content processing terminal or a part of its components are not specifically limited in this embodiment.

[0167] The machine-readable storage medium may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), etc. The machine-readable storage medium is used to store a program, which the processor executes upon receiving an execution instruction.

[0168] The processor may be an integrated circuit chip with signal processing capabilities. The processor mentioned above can be, but is not limited to, a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc.

[0169] In summary, the intelligent audio-visual object tracking optimization method and system combining deep learning provided in this application can significantly improve the accuracy and stability of audio-visual object tracking. Specifically, this application first acquires synchronized audio-visual data sets, ensuring the temporal consistency of data during subsequent cross-modal processing and avoiding modal information misalignment caused by data asynchrony. Based on this, a cross-modal feature collaborative extraction operation is performed. For audio stream data, temporal dynamic features including frequency distribution patterns and energy fluctuation modes are extracted; for video frame sequences, spatial structural features including contour boundaries and texture distribution are extracted. Compared to existing technologies, this method more comprehensively captures the multi-dimensional features of the target object, enhances the expressive power of the features, and provides richer information for subsequent tracking processing. Furthermore, by using an association enhancement network to perform temporal association modeling on the multimodal feature set, deep semantic alignment of audio temporal dynamic features and video spatial structural features is achieved. This fully explores the inherent correlation between the two modalities during temporal evolution, addressing the problems of simple modality fusion and insufficient association mining in existing technologies. This allows the bimodal features to complement each other, improving the robustness of features in describing target objects, and enabling better target identification and tracking even in complex environments. Further, a tracking stability evaluation model is introduced. By calculating feature consistency parameters, temporal continuity parameters, and cross-modal matching degree parameters, the reliability of the tracking process is quantified in real time, providing a basis for dynamically adjusting tracking parameters. Combining stability indices with the initial tracking results for calibration achieves closed-loop optimization of the tracking process, avoiding the accumulation of tracking deviations. This overcomes the lack of dynamic evaluation and real-time calibration mechanisms in existing technologies, enabling the tracking results to adaptively adjust with environmental changes, further improving tracking accuracy and stability. Thus, through the organic combination of deep collaboration of cross-modal features, temporal correlation modeling, dynamic evaluation and parameter adjustment, the object tracking accuracy and stability in complex scenarios (such as target occlusion, noise interference, and illumination changes) have been significantly improved, making the tracking results more reliable and robust, and able to meet the needs of high-precision tracking in practical applications.

[0170] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. The embodiments, implementation methods, and related technical features of this application can be combined and substituted with each other without conflict. The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. An audio-video object intelligent tracking optimization method combined with deep learning, characterized in that, The method comprises: acquiring a plurality of continuously collected audio-video data groups, the audio-video data groups comprising synchronously collected audio stream data and video frame sequences; performing cross-modal feature collaborative extraction operations on the audio stream data and the video frame sequences to generate a multi-modal feature set comprising audio time-domain dynamic features and video spatial structure features; inputting the multi-modal feature set into a pre-trained correlation enhancement network to perform time-series correlation modeling and generate a correlation feature sequence with cross-modal semantic alignment; constructing a tracking stability evaluation model based on the correlation feature sequence to output a stability indicator reflecting the reliability of the tracking process; dynamically adjusting tracking parameters and calibrating initial tracking results according to the stability indicator to output an optimized object tracking trajectory for the target object; wherein the cross-modal feature collaborative extraction operations on the audio stream data and the video frame sequences to generate a multi-modal feature set comprising audio time-domain dynamic features and video spatial structure features comprise: performing time-frequency conversion processing on the audio stream data to extract audio basic features comprising frequency distribution rules and energy fluctuation patterns; performing motion perception processing on the video frame sequences to capture the displacement trend of the target object between consecutive frames through optical flow estimation technology to generate video motion features; inputting the audio basic features and the video motion features into a feature complementary module to analyze the semantic overlap area of the audio basic features and the video motion features in the time dimension and extract audio time-domain dynamic features; performing target segmentation processing on the video frame sequences to locate the contour boundary of the target object and extract texture distribution features through a region growing algorithm to generate video spatial structure features; performing dimension alignment processing on the audio time-domain dynamic features and the video spatial structure features to form a multi-modal feature set; wherein the inputting of the multi-modal feature set into the pre-trained correlation enhancement network to perform time-series correlation modeling and generate a correlation feature sequence with cross-modal semantic alignment comprises: inputting the multi-modal feature set into a local correlation layer of the correlation enhancement network to calculate the local similarity of the audio time-domain dynamic features and the video spatial structure features at adjacent time steps to generate local correlation features; performing long-range dependency analysis on the local correlation features through a global context layer of the correlation enhancement network to extract behavior pattern consistency features of the target object within a continuous time window; using a semantic alignment module of the correlation enhancement network to perform semantic alignment on the local correlation features and the global context features based on a prior feature template of the target object to obtain semantically aligned features; inputting the semantically aligned features into a time-series fusion layer of the correlation enhancement network to model the time evolution law of the feature sequence through a gated recurrent unit to generate a time-series coherent correlation feature sequence; performing normalization processing on the correlation feature sequence to balance the contribution weights of different modal features to form a correlation feature sequence with cross-modal semantic alignment.

2. The method of claim 1, wherein the method further comprises: The motion perception processing on the video frame sequences to capture the displacement trend of the target object between consecutive frames through optical flow estimation technology to generate video motion features comprises: From a pair of consecutive frames of the video frame sequence, a stable feature point in a region where a target object is located is extracted, the stable feature point being a pixel point with a low gray value change rate in consecutive frames and lower than a preset threshold; A motion vector of each stable feature point between consecutive frames is calculated by an optical flow estimation technique, the motion vector including a horizontal direction displacement component and a vertical direction displacement component, forming a feature point motion vector set; The feature point motion vector set is subjected to vector clustering, and an average vector of all motion vectors in each motion cluster is calculated, obtaining a local motion trend vector; A proportion of a number of feature points included in each motion cluster to a total number of feature points in the region of the target object is calculated as a weight coefficient of the motion cluster; A spatial adjacency matrix is constructed based on spatial position distribution of each motion cluster, an element value of the spatial adjacency matrix representing a distance between two motion clusters in an image space, and the closer the distance, the greater the element value; The local motion trend vector, the weight coefficient and the element value of the corresponding row in the spatial adjacency matrix are multiplied element by element, obtaining a weighted local motion feature; A sum vector of the weighted local motion features of all motion clusters is calculated, obtaining a global motion trend vector of the target object, a horizontal component of the sum vector being a sum of horizontal components of the weighted local motion features, and a vertical component being a sum of vertical components of the weighted local motion features; Residual operation is performed on all motion vectors in the feature point motion vector set and the global motion trend vector, obtaining a residual vector set, and a root mean square value of the residual vector set is taken as a motion consistency index; The weighted local motion features of each motion cluster, the global motion trend vector and the motion consistency index are fused to generate a video motion feature. 3.The method of claim 1, wherein, The tracking stability evaluation model is constructed based on the associated feature sequence, and a stability index reflecting reliability of a tracking process is output, including: A feature consistency parameter of the target object is extracted from the associated feature sequence, the feature consistency parameter representing a feature maintenance degree of the target object at consecutive time steps; A time sequence continuity parameter of the associated feature sequence is analyzed, the time sequence continuity parameter representing a smooth degree of position change of the target object; A cross-modal matching degree parameter of the associated feature sequence is calculated, the cross-modal matching degree parameter representing an agreement degree of audio features and video features when describing the same target; The feature consistency parameter, the time sequence continuity parameter and the cross-modal matching degree parameter are input into a stability evaluation function, and the stability index is calculated; Dynamic threshold judgment is performed on the stability index, a drift risk point possibly appearing in the tracking process is identified, and risk position information is recorded.

4. The method of claim 1, wherein, The tracking parameters are dynamically adjusted according to the stability index, and the initial tracking result is calibrated, and an optimized object tracking trajectory for the target object is output, including: An initial tracking trajectory generated based on an initialization tracking algorithm is obtained, the initial tracking trajectory including a position coordinate sequence of the target object in the video frame sequence; When the stability index is lower than a preset level, audio time domain dynamic features and video space structure features corresponding to the risk position in the associated feature sequence are extracted, obtaining risk position features. The risk position feature is used to construct a parameter adjustment model, and a position prediction weight of the initialized tracking algorithm is optimized by a back propagation algorithm to generate an adjusted tracking parameter; The adjusted tracking parameter is weighted and fused with a position coordinate sequence of an initial tracking trajectory to correct a position deviation caused by feature drift; The corrected position coordinate sequence is subjected to smoothing filtering processing to eliminate local abnormal fluctuations and generate a time-sequentially coherent optimized object tracking trajectory.

5. The method of claim 1, wherein, The semantic alignment module based on the correlation enhancement network aligns semantics of the local correlation feature and the global context feature based on a prior feature template of the target object to obtain a feature after semantic alignment, including: Audio feature templates and video feature templates of the target object are extracted from historical audio-video data in a training phase to construct a prior feature template library; the historical audio-video data are a plurality of groups of labeled audio-video stream data groups containing the target object, each group of audio-video stream data group containing a continuously collected audio signal sequence and a video frame sequence collected synchronously; the audio feature templates are extracted after time-frequency domain conversion processing is performed on the historical audio signal sequence; the video feature templates are extracted after target perception processing is performed on the historical video frame sequence; Similarity calculation is performed on the local correlation feature and the global context feature respectively with the audio feature templates and the video feature templates in the prior feature template library to obtain intra-modal matching degrees; the local correlation feature is a feature output by a local correlation layer of the correlation enhancement network and contains local similarity information of audio time domain dynamic features and video spatial structure features at adjacent time steps; the global context feature is a feature output by a global context layer of the correlation enhancement network and contains long-range behavior pattern consistency information of the audio time domain dynamic features and the video spatial structure features within a continuous time window; Differences of the intra-modal matching degrees are analyzed to identify a shift direction of audio features and video features of the target object in semantic expression; A feature in the shift direction is weighted and corrected by a parameter adjustment mechanism of the semantic alignment module to enhance semantic consistency between the modalities; the parameter adjustment mechanism includes feature weight adjustment and feature value correction operations; The corrected local correlation feature and the global context feature are input into a classifier of the correlation enhancement network to verify semantic alignment effect until a difference in inter-modal matching degree meets a requirement; the classifier is a pre-trained multi-modal classifier in the correlation enhancement network, which inputs the corrected local correlation feature and the global context feature and outputs a class prediction result of the target object.

6. The method of claim 1, wherein the method further comprises: The audio stream data are subjected to time-frequency conversion processing to extract audio basic features containing frequency distribution rules and energy fluctuation patterns, including: The audio stream data are subjected to frame processing to generate a plurality of audio sub-frames with overlapping windows; Fast Fourier transform is applied to each audio sub-frame to convert a time domain signal into a frequency domain signal and extract a frequency spectrum graph; Frequency peak value distribution of the frequency spectrum graph is analyzed to identify a characteristic frequency interval in which the target object emits sound; An energy accumulation value of the frequency spectrum graph in the characteristic frequency interval is calculated to generate a fluctuation curve reflecting energy change over time; The feature frequency interval information is fused with the energy fluctuation curve to form audio basic features including frequency distribution rules and energy fluctuation modes.

7. The method of claim 1, wherein the method further comprises: The target segmentation processing is performed on the video frame sequence, the contour boundary of the target object is located by using a region growing algorithm, and texture distribution features are extracted to generate video spatial structure features, including: The Gaussian blur denoising processing is performed on the current video frame of the video frame sequence, a Gaussian kernel with a preset size is used to perform weighted average operation on image pixels to generate a denoised frame; The contrast adaptive enhancement processing is performed on the denoised frame, the image gray scale distribution is adjusted by using a limited contrast adaptive histogram equalization algorithm to strengthen the gray scale difference between the target object and the background region to obtain an enhanced frame; The gradient amplitude image is extracted from the enhanced frame, the horizontal and vertical direction gradients are calculated by using an edge detection algorithm, the candidate seed points are screened based on the gradient amplitude non-maximum suppression to generate a seed point coordinate set; The seed point coordinate set is taken as a growing starting point, a gray scale similarity threshold and a region connectivity constraint condition are set, the region growing operation is performed, and adjacent pixel points with a gray scale difference less than the threshold are merged into an initial segmentation region; The morphological optimization processing is performed on the initial segmentation region, the dilation operation and the corrosion operation are sequentially performed to fill the internal holes of the region and eliminate the boundary burrs to generate an optimized segmentation region; The boundary pixel point set of the optimized segmentation region is extracted, the polygon fitting is performed on the boundary pixel point set to obtain the contour boundary descriptor of the target object; The gray scale co-occurrence matrix is calculated in the optimized segmentation region, the energy, entropy, contrast and correlation parameters of the matrix are extracted to form a texture distribution feature vector; The contour boundary descriptor and the texture distribution feature vector are spliced to generate video spatial structure features including spatial shape information and texture detail information, and the dimension of the video spatial structure features is the sum of the dimension of the contour boundary descriptor and the dimension of the texture distribution feature vector.

8. An audio-video object intelligent tracking optimization system combined with deep learning, characterized in that, The processor, the machine readable storage medium, the machine readable storage medium and the processor are connected, the machine readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine readable storage medium to realize the audio and video object intelligent tracking optimization method combined with deep learning in any one of claims 1-7.

Images