Optical focal length automatic adjustment control system based on intelligent sensor

Abstract

The application discloses an optical focal length automatic adjustment control system based on an intelligent sensor, relates to the technical field of optical engineering, and comprises the following modules: a multimodal feature fusion module, which is used for generating a multimodal feature map; a predictive tracking modeling module, which is used for introducing a scanning and state updating mechanism of a Mamba network to improve a TransTrack model and output a multi-target tracking result; a dynamic relationship graph construction module, which is used for constructing a dynamic space-time relationship graph; a graph network priority evaluation module, which is used for evaluating target priority; a target trajectory prediction module, which is used for predicting a target trajectory; a camera control instruction generation module, which is used for generating focusing and aperture control instructions; and a lens driving execution module, which is used for executing focusing and depth of field adjustment actions. The application overcomes the limitations of response lag, blind decision and neglect of the correlation between targets in the traditional optical focal length adjustment method, and provides an efficient and accurate solution.

Description

Technical Field

[0001] This invention relates to the field of optical engineering technology, and in particular to an automatic optical focal length adjustment control system based on intelligent sensors. Background Technology

[0002] With the deep integration of intelligent visual perception and automatic control technologies, traditional optical imaging systems are facing increasingly complex dynamic scenarios. In modern intelligent monitoring, autonomous driving, and human-computer interaction applications, clear imaging of fast-moving, multi-target interactive scenes is not only an important means of capturing key information but also a core element in improving the overall perception accuracy and decision-making reliability of the system. However, most mainstream optical focal length adjustment methods currently rely on passive triggering and local optimization algorithms based on contrast or phase detection, resulting in slow response speeds and a lack of foresight. While these traditional methods can meet basic focusing requirements in static or simple scenes, their lack of a global understanding of the dynamic evolution of the scene and the ability to predict motion trends makes them unsuitable for complex environments with multiple targets moving at high speeds, frequent occlusion, and dramatic changes in depth.

[0003] The main limitations of traditional optical focus adjustment methods lie in their response lag and blind decision-making. Existing methods typically determine focus based on the image information of the current frame. When the target moves rapidly or the system itself has significant inertia, the focusing action lags far behind the actual position of the target, resulting in blurred images. When there are multiple potential focus targets in a scene, traditional methods struggle to intelligently assess the importance of each target, often leading to focusing on non-critical targets or frequent switching between different targets, failing to stably focus on the core object of interest. Especially when faced with complex spatiotemporal interactions between targets, traditional single-target or simple multi-target tracking methods struggle to efficiently and accurately predict future focus, causing the focusing system to be in a passive chasing state, severely impacting image quality and the effectiveness of information acquisition.

[0004] Furthermore, traditional methods often separate target recognition, motion prediction, and camera physical control into independent, sequential modules during focusing and depth-of-field control, neglecting the close coupling between them. For example, in scenarios with multiple interacting targets, traditional focusing strategies cannot effectively integrate the future motion trajectories of the targets, their spatial relationships, and the physical motion characteristics of the lens. This results in a lack of basis for decisions regarding the focus plane and depth of field, making it difficult to achieve optimal imaging results. Even when some methods introduce simple target tracking, they fail to fully explore the deep spatiotemporal relationships and future states of the targets, making it impossible to generate intelligent control commands that combine predictability, relationship awareness, and physical constraints. Consequently, it is difficult to achieve efficient, accurate, and adaptive automatic optical focal length adjustment.

[0005] Therefore, how to provide an automatic optical focal length adjustment control system based on intelligent sensors is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] This invention proposes an automatic optical focal length adjustment control system based on intelligent sensors. By improving the TransTrack model through a Mamba network and combining it with a TGN network, it effectively integrates target tracking, priority evaluation, and future trajectory prediction, significantly improving the perception and response accuracy of the optical system for key targets. It also improves upon traditional passive focusing strategies, enabling the system to proactively calculate the focal plane and intelligently determine the depth of field, effectively overcoming the imaging challenges posed by motion blur and target interaction. This invention overcomes the limitations of traditional optical focal length adjustment methods, such as response lag, blind decision-making, and neglect of inter-target relationships, providing an efficient and precise solution for intelligent imaging and automatic control in complex dynamic scenes.

[0007] An automatic optical focal length adjustment control system based on a smart sensor according to an embodiment of the present invention includes the following modules:

[0008] The multimodal feature fusion module is used to simultaneously acquire data from multiple sources of sensors, extract and fuse visual, geometric and motion modal features, and generate a multimodal feature map;

[0009] The predictive tracking modeling module is used to improve the TransTrack model by introducing the scanning and state update mechanism of the Mamba network, predictively modeling historical trajectories, and fusing the dual context of multimodal feature maps and target detection results to output multi-target tracking results and posterior tracking query matrix.

[0010] The dynamic relationship graph construction module is used to instantiate multi-target tracking results into graph nodes, decode the original relationship scores in parallel through two branches and generate weighted graph edges to form a dynamic spatiotemporal relationship graph;

[0011] The graph network priority evaluation module is used to aggregate neighborhood information and update node features through a TGN network based on a dynamic spatiotemporal relationship graph, and generate a priority score for each target using a priority evaluation linear head and output the priority classification result.

[0012] The target trajectory prediction module is used to filter the corresponding query vector from the posterior tracking query matrix based on the priority classification results, and predict the future three-dimensional spatial position and future radial velocity of the target through an LSTM network.

[0013] The camera control command generation module is used to calculate the focal plane based on the target's future three-dimensional spatial position and future radial velocity, and combine the dynamic spatiotemporal relationship diagram to determine the target interaction relationship and the depth of field range, and generate focus and aperture control commands.

[0014] The lens drive execution module is used to send focus control commands and aperture control commands to the lens drive unit, and control the lens motor and aperture blades to perform corresponding focus and depth-of-field adjustment actions.

[0015] An automatic optical focal length adjustment control method based on a smart sensor according to an embodiment of the present invention includes the following steps:

[0016] S1. Simultaneously acquire data from multiple sensor sources, extract and fuse visual, geometric and motion modal features to generate a multimodal feature map;

[0017] S2. The TransTrack model is improved by introducing the scanning and state update mechanism of the Mamba network, which predictively models the historical trajectory and integrates the dual context of multimodal feature maps and target detection results to output multi-target tracking results and posterior tracking query matrix.

[0018] S3. Instantiate the multi-target tracking results into graph nodes, decode the original relation scores in parallel through two branches and generate weighted graph edges to form a dynamic spatiotemporal relation graph;

[0019] S4. Based on the dynamic spatiotemporal relationship graph, the neighborhood information is aggregated and the node features are updated through the TGN network. The priority evaluation linear head is used to generate the priority score of each target and output the priority classification result.

[0020] S5. Based on the priority classification results, select the corresponding query vector from the posterior tracking query matrix, and predict the future three-dimensional spatial position and future radial velocity of the target through the LSTM network.

[0021] S6. Calculate the focal plane based on the target's future three-dimensional spatial position and future radial velocity, and combine it with the dynamic spatiotemporal relationship diagram to determine the target interaction relationship and the depth of field range, and generate focus and aperture control commands.

[0022] S7. Send focus control commands and aperture control commands to the lens drive unit to control the lens motor and aperture blades to perform corresponding focus and depth-of-field adjustment actions.

[0023] Optionally, S1 specifically includes:

[0024] S11. Real-time acquisition of main image, raw distance map and inertial measurement unit data through sensor array;

[0025] S12. Input the main image into the ResNet network. Through initial convolution, residual block processing and multi-level downsampling, the first scale feature map, the second scale feature map, the third scale feature map and the fourth scale feature map are generated step by step, which together constitute a multi-scale visual feature map.

[0026] S13. Perform median filtering and bilinear interpolation upsampling on the original distance map, align it to the main image coordinate system by projecting it onto the extrinsic parameters, and generate a geometric feature map by convolution.

[0027] S14. Based on the inertial measurement unit data, a motion coding vector is generated through Kalman filtering and linear transformation, and a motion feature map aligned with the visual feature map is constructed through spatial expansion and broadcasting operations.

[0028] S15. The second-scale feature map, geometric feature map, and motion feature map are concatenated along the channel dimension to generate an initial fused feature map, and then convolutional processing is performed to output a multimodal feature map.

[0029] Optionally, S2 specifically includes:

[0030] The improved TransTrack model includes a query decoupling layer, a temporal correlation modeling layer, a detection feature projection layer, a detection interactive coding layer, a tracking feature fusion projection layer, a tracking interactive coding layer, and a prediction output generation layer.

[0031] The query decoupling layer is used to receive the posterior tracking query matrix output from the previous frame and the preset learnable detection query matrix of the current frame; the posterior tracking query matrix is ​​input into the temporal correlation modeling layer, and the detection query matrix is ​​input into the detection feature projection layer.

[0032] The temporal correlation modeling layer is used to model the temporal dependencies of the input posterior tracking query matrix through the scanning and state update mechanism of the Mamba network, and to predict and generate the prior tracking query matrix of the current frame.

[0033] The detection feature projection layer is used to receive the detection query stream and the multimodal feature map; the detection query projection layer maps the detection query matrix to the detection query embedding matrix; the feature map key projection layer maps the multimodal feature map to the visual feature key matrix; and the feature map value projection layer maps the multimodal feature map to the visual feature value matrix.

[0034] The detection interactive coding layer is used to perform parallel cross-attention calculations on the detection query embedding matrix, the visual feature key matrix, and the visual feature value matrix, and to assign contextual information from the feature map to each query vector in the detection query embedding matrix to generate an enhanced detection query matrix.

[0035] The tracking feature fusion projection layer maps the prior tracking query, enhanced detection query, and multimodal feature map into corresponding query, key, and value matrices through multiple parallel projection layers, and fuses the key-value information of visual and detection semantics to generate a fusion context key matrix and value matrix for tracking interaction.

[0036] The tracking interactive coding layer is used to perform parallel cross-attention calculations on the tracking query matrix, the fusion context key matrix, and the fusion context value matrix, and to assign dual context information from the feature map and the augmented detection query matrix to each query vector in the tracking query matrix, thereby generating the posterior tracking query matrix of the current frame.

[0037] The prediction output generation layer is used to decode the posterior tracking query matrix of the current frame and generate a target instance for each query vector of the posterior tracking query matrix; the original posterior tracking query matrix is ​​used as the embedding feature of each target instance and associated with the corresponding target instance; all target instances and their corresponding embedding features are aggregated to form the multi-target tracking result of the current frame.

[0038] Output the multi-target tracking results and the posterior tracking query matrix.

[0039] Optionally, S3 specifically includes:

[0040] S31. Treat each target instance in the multi-target tracking result as a graph node, and extract and concatenate the bounding box, category and embedding features of the target instance into a node feature vector;

[0041] S32. Combine the feature vectors of all nodes into a node feature matrix; perform parallelized matrix multiplication operations on the node feature matrix and the corresponding transpose matrix to generate the original relation score matrix.

[0042] S33. Input the original relation score matrix into the parallel execution of the relation strength evaluation linear layer and the relation type classification linear layer. The relation strength evaluation linear layer calculates and outputs the relation strength matrix through a fully connected layer and a sigmoid activation function; the relation type classification linear layer calculates and outputs the relation type matrix through a fully connected layer and a softmax activation function.

[0043] S34. Based on the relation strength matrix and relation type matrix, generate weighted graph edges connecting corresponding nodes;

[0044] S35. Construct a dynamic spatiotemporal relationship graph based on all graph nodes, node feature vectors, and weighted graph edges.

[0045] Optionally, S4 specifically includes:

[0046] S41. Receive the dynamic spatiotemporal relationship graph, aggregate the neighborhood information of each graph node through the message passing mechanism of the TGN network, update the initial node features, and generate the updated node feature matrix.

[0047] S42. Input the updated node feature matrix into the priority evaluation linear head, calculate and output the target priority score for each graph node through linear transformation of the fully connected layer and Sigmoid activation function;

[0048] S43. Sort all targets in descending order according to their priority scores, and classify them into three priority levels: critical, important, and ordinary, based on a preset score threshold. Output the target priority classification results.

[0049] Optionally, S5 specifically includes:

[0050] The TGN network includes a node memory module, a message encoder, a temporal attention module, a neighborhood aggregator, and an embedding function layer;

[0051] The node memory module is used to maintain a state vector that evolves over time for each node in the graph, which serves as the memory state vector of the node at the current moment.

[0052] The message encoder is used to encode the interaction event when an interaction event occurs between any two nodes in the graph, and generate an interaction message vector.

[0053] The temporal attention module is used to calculate the attention weight between the memory state vector of each node and the corresponding historical interaction message vector;

[0054] The neighborhood aggregator is used to weight and aggregate historical message vectors according to attention weights to generate neighborhood aggregated message vectors.

[0055] The embedding function layer is used to combine the current memory state vector of a node and the neighborhood aggregation message vector to update the memory state of the node and generate updated node features; it also aggregates the final node features of all nodes at the current time to form the updated node feature matrix.

[0056] Optionally, S5 specifically includes:

[0057] S51. Receive the posterior tracking query matrix and the target identifiers of all critical and important targets; index and extract the corresponding query vectors from the posterior tracking query matrix according to the target identifiers, and generate a set of critical and important query vectors;

[0058] S52. Extract the corresponding historical location sequence and velocity information for each query vector in the key-level and importance-level query vector sets;

[0059] S53. Organize the historical position sequence and velocity information into an input sequence according to time steps and input it into the LSTM network. Process the input sequence through internal state update and output the prediction results of the three-dimensional spatial position and future radial velocity within a preset time period.

[0060] Optionally, S6 specifically includes:

[0061] S61. Receive the future three-dimensional spatial position and future radial velocity of important and critical targets, as well as the dynamic spatiotemporal relationship diagram;

[0062] S62. Project the future 3D spatial position along the depth axis to obtain the future depth value; input the future depth value into a preset focus distance mapping table; look up the index in the focus distance mapping table that is closest to the future depth value; extract the corresponding initial focus distance based on the index;

[0063] S63. Input the future radial velocity into the velocity compensation coefficient table, query and extract the corresponding velocity compensation coefficient; multiply the future radial velocity by the velocity compensation coefficient to generate the distance compensation amount;

[0064] S64. Add the initial focusing distance to the distance compensation amount to generate the future focusing plane position;

[0065] S65. Determine whether key-level objectives and important-level objectives constitute a strong interactive unit;

[0066] S66. If it is determined that a strong interactive unit is formed, the future depth values ​​of key-level targets and important-level targets are extracted, the maximum and minimum depth values ​​are calculated, and an interactive depth range is generated; if it is determined that a strong interactive unit is not formed, the future depth values ​​of key-level targets are extracted, the maximum and minimum depth values ​​are calculated, and an independent depth range is generated.

[0067] Optionally, S7 specifically includes:

[0068] S71, receives focus control commands and aperture control commands;

[0069] S72. Based on the focus control command, analyze the target driving distance and driving direction;

[0070] S73. Based on the aperture control command, analyze the target aperture value and drive direction;

[0071] S74. Convert the target driving distance and driving direction into a first pulse width modulation signal for the drive motor;

[0072] S75. Convert the target aperture value into a second pulse width modulation signal that drives the aperture blades;

[0073] S76. Send the first pulse width modulation signal to the lens motor to control the lens to focus; send the second pulse width modulation signal to the aperture drive unit to control the aperture blades to adjust to the target depth of field.

[0074] The beneficial effects of this invention are:

[0075] This invention significantly enhances the ability and accuracy of optical systems to perform forward-looking tracking of multiple targets in dynamic scenes by improving the TransTrack model. Traditional target tracking models often experience tracking interruptions or ID switching when dealing with complex situations such as high-speed motion and frequent occlusion due to a lack of deep temporal modeling of historical trajectories. To address these issues, this invention innovatively introduces the scanning and state update mechanism of the Mamba network to perform predictive temporal correlation modeling on the posterior tracking query matrix, enabling the model to effectively capture the long-term motion trends and state evolution patterns of the target. Simultaneously, through a dual context fusion mechanism of the detection and tracking interactive coding layer, visual features are tightly integrated with detection semantic information, providing richer contextual support for tracking queries. This significantly improves the robustness and accuracy of tracking, providing high-precision target state input for subsequent intelligent focusing decisions. Attached Figure Description

[0076] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0077] Figure 1 This is a schematic diagram of the structure of an automatic optical focal length adjustment control system based on a smart sensor proposed in this invention;

[0078] Figure 2 This is an overall flowchart of an automatic optical focal length adjustment and control method based on intelligent sensors proposed in this invention.

[0079] Figure 3 This is a flowchart illustrating the working principle of the improved TransTrack model, which is based on an intelligent sensor for automatic optical focal length adjustment control. Detailed Implementation

[0080] The invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0081] refer to Figure 1 An automatic optical focal length adjustment control system based on intelligent sensors includes the following modules:

[0082] The multimodal feature fusion module is used to simultaneously acquire data from multiple sources of sensors, extract and fuse visual, geometric and motion modal features, and generate a multimodal feature map;

[0083] The predictive tracking modeling module is used to improve the TransTrack model by introducing the scanning and state update mechanism of the Mamba network, predictively modeling historical trajectories, and fusing the dual context of multimodal feature maps and target detection results to output multi-target tracking results and posterior tracking query matrix.

[0084] The dynamic relationship graph construction module is used to instantiate multi-target tracking results into graph nodes, decode the original relationship scores in parallel through two branches and generate weighted graph edges to form a dynamic spatiotemporal relationship graph;

[0085] The graph network priority evaluation module is used to aggregate neighborhood information and update node features through a TGN network based on a dynamic spatiotemporal relationship graph, and generate a priority score for each target using a priority evaluation linear head and output the priority classification result.

[0086] The target trajectory prediction module is used to filter the corresponding query vector from the posterior tracking query matrix based on the priority classification results, and predict the future three-dimensional spatial position and future radial velocity of the target through an LSTM network.

[0087] The camera control command generation module is used to calculate the focal plane based on the target's future three-dimensional spatial position and future radial velocity, and combine the dynamic spatiotemporal relationship diagram to determine the target interaction relationship and the depth of field range, and generate focus and aperture control commands.

[0088] The lens drive execution module is used to send focus control commands and aperture control commands to the lens drive unit, and control the lens motor and aperture blades to perform corresponding focus and depth-of-field adjustment actions.

[0089] refer to Figure 2 and Figure 3 An automatic optical focal length adjustment control method based on intelligent sensors includes the following steps:

[0090] S1. Simultaneously acquire data from multiple sensor sources, extract and fuse visual, geometric and motion modal features to generate a multimodal feature map;

[0091] S2. The TransTrack model is improved by introducing the scanning and state update mechanism of the Mamba network, which predictively models the historical trajectory and integrates the dual context of multimodal feature maps and target detection results to output multi-target tracking results and posterior tracking query matrix.

[0092] S3. Instantiate the multi-target tracking results into graph nodes, decode the original relation scores in parallel through two branches and generate weighted graph edges to form a dynamic spatiotemporal relation graph;

[0093] S4. Based on the dynamic spatiotemporal relationship graph, the neighborhood information is aggregated and the node features are updated through the TGN network. The priority evaluation linear head is used to generate the priority score of each target and output the priority classification result.

[0094] S5. Based on the priority classification results, select the corresponding query vector from the posterior tracking query matrix, and predict the future three-dimensional spatial position and future radial velocity of the target through the LSTM network.

[0095] S6. Calculate the focal plane based on the target's future three-dimensional spatial position and future radial velocity, and combine it with the dynamic spatiotemporal relationship diagram to determine the target interaction relationship and the depth of field range, and generate focus and aperture control commands.

[0096] S7. Send focus control commands and aperture control commands to the lens drive unit to control the lens motor and aperture blades to perform corresponding focus and depth-of-field adjustment actions.

[0097] In this embodiment, S1 includes:

[0098] S11. Real-time acquisition of main image, raw distance map and inertial measurement unit data through sensor array;

[0099] S12. Input the main image into the ResNet network, perform convolution processing through the initial convolutional layer and max pooling operation through the max pooling layer to generate the initial feature map; the initial feature map is then processed by convolution through multiple residual blocks to generate the first-scale feature map; the first-scale feature map is then processed by three consecutive downsampling convolutions with a stride of 2 to generate the corresponding second, third, and fourth-scale feature maps; the first, second, third, and fourth-scale feature maps together constitute the multi-scale visual feature map;

[0100] S13. Perform median filtering and bilinear interpolation upsampling on the original distance map to obtain the upsampled distance map; collect the extrinsic calibration parameters between the main image and the depth camera, and use the extrinsic calibration parameters to project the upsampled distance map onto the pixel coordinate system of the main image to generate an aligned distance map; generate a geometric feature map by convolution processing the aligned distance map.

[0101] S14. Based on the inertial measurement unit data, a motion state vector is generated through Kalman filtering; the motion state vector is subjected to a fully connected linear transformation, and the output dimension is set to the preset number of motion feature channels to generate a motion encoding vector; the motion encoding vector is expanded in spatial dimension to generate an initial feature vector with a height of 1, a width of 1, and a channel number of the preset number of motion feature channels; the initial feature vector is spatially broadcast until the spatial size is the same as the spatial size of the second-scale feature map to generate a motion feature map;

[0102] S15. Concatenate the second-scale feature map, geometric feature map, and motion feature map along the channel dimension to generate an initial fused feature map; perform convolution processing on the initial fused feature map to output a multimodal feature map.

[0103] This step fuses multi-source sensor data to ultimately generate a unified Richer multimodal feature map representing the scene. A sensor array acquires the main image, raw distance map, and inertial measurement unit (IMU) data in real time to obtain comprehensive raw perception information. After filtering, upsampling, and coordinate alignment of the raw distance map, convolution processing is used to generate a geometric feature map that accurately reflects the spatial structure. Based on the IMU data, Kalman filtering and spatial broadcasting are applied to generate a motion feature map aligned with the visual feature map to capture the global motion state.

[0104] In this embodiment, the improved TransTrack model includes a query decoupling layer, a temporal correlation modeling layer, a detection feature projection layer, a detection interactive coding layer, a tracking feature fusion projection layer, a tracking interactive coding layer, and a prediction output generation layer.

[0105] The query decoupling layer is used to receive the posterior tracking query matrix output from the previous frame and the preset learnable detection query matrix of the current frame; the posterior tracking query matrix is ​​input into the temporal correlation modeling layer, and the detection query matrix is ​​input into the detection feature projection layer.

[0106] The temporal correlation modeling layer is used to model the temporal dependencies of the input posterior tracking query matrix through the scanning and state update mechanism of the Mamba network, predict and generate the prior tracking query matrix of the current frame. This includes mapping the posterior tracking query sequence to a temporal feature matrix; inputting the temporal feature matrix into the selection mechanism input projection layer, selection mechanism incremental projection layer and selection mechanism output projection layer for parallel processing, mapping them to the selection mechanism input matrix, selection mechanism incremental matrix and selection mechanism output matrix respectively; performing discretized state space calculation on the temporal feature matrix, selection mechanism input matrix and selection mechanism incremental matrix to generate a scanning state matrix; gating the scanning state matrix with the selection mechanism output matrix to generate a temporal attention output matrix; and performing residual connection between the temporal attention output matrix and the temporal feature matrix, and performing layer normalization processing to generate a prior tracking query matrix containing motion prediction.

[0107] The detection feature projection layer is used to receive the detection query stream and the multimodal feature map; the detection query projection layer maps the detection query matrix to the detection query embedding matrix; the feature map key projection layer maps the multimodal feature map to the visual feature key matrix; and the feature map value projection layer maps the multimodal feature map to the visual feature value matrix.

[0108] The detection interactive coding layer is used to perform parallel cross-attention computation on the detection query embedding matrix, the visual feature key matrix, and the visual feature value matrix. It assigns contextual information from the feature map to each query vector in the detection query embedding matrix, including transposing the visual feature key matrix, performing matrix multiplication between the detection query embedding matrix and the transposed visual feature key matrix to generate the original attention score matrix, scaling the original attention score matrix using a preset scaling factor to generate the scaled attention score matrix, applying the Softmax activation function to the scaled attention score matrix to generate the attention weight matrix, using the attention weights to perform a weighted summation of the visual feature value matrix to generate the detection attention output matrix, and performing a residual concatenation between the detection attention output matrix and the detection query embedding matrix, followed by layer normalization to generate the enhanced detection query matrix.

[0109] The tracking feature fusion projection layer, through multiple parallel projection layers, maps the prior tracking query, enhanced detection query, and multimodal feature map into corresponding query, key, and value matrices, respectively. It then fuses the key-value information from visual and detection semantics to generate a fusion context key matrix and value matrix for tracking interaction. This includes mapping the prior tracking query matrix to a tracking query matrix through a tracking query projection layer; mapping the multimodal feature map to a visual feature key matrix through a feature map key projection layer; mapping the multimodal feature map to a visual feature value matrix through a feature map value projection layer; mapping the enhanced detection query matrix to a detection semantic key matrix through a detection query key projection layer; mapping the enhanced detection query matrix to a detection semantic value matrix through a detection query value projection layer; concatenating the visual feature key matrix and the detection semantic key matrix along the sequence dimension to generate a fusion context key matrix; and concatenating the visual feature value matrix and the detection semantic value matrix along the sequence dimension to generate a fusion context value matrix.

[0110] The tracking interactive coding layer is used to perform parallel cross-attention computation on the tracking query matrix, the fused context key matrix, and the fused context value matrix. It assigns dual context information from the feature map and the augmented detection query matrix to each query vector in the tracking query matrix. This includes transposing the fused context key matrix, performing matrix multiplication between the tracking query matrix and the transposed fused context key matrix to generate the original attention score matrix, scaling the original attention score matrix using a preset scaling factor to generate the scaled attention score matrix, applying the Softmax activation function to the scaled attention score matrix to generate the attention weight matrix, using the attention weights to perform a weighted summation of the fused context value matrix to generate the tracking attention output matrix, and performing a residual concatenation between the tracking attention output matrix and the tracking query matrix, followed by layer normalization to generate the posterior tracking query matrix for the current frame.

[0111] The prediction output generation layer is used to decode the posterior tracking query matrix of the current frame, including inputting the posterior tracking query matrix into the identity classification linear layer, performing matrix multiplication to generate the original classification logarithm matrix; applying the Softmax activation function to the original classification logarithm matrix and normalizing it along the category dimension to generate the category probability distribution matrix; inputting the posterior tracking query matrix into the boundary regression linear layer, performing matrix multiplication to generate the original bounding box coordinate matrix; applying the ReLU activation function to the original bounding box coordinate matrix to generate the final bounding box coordinate matrix; and inputting the posterior tracking query matrix into the positioning layer. The confidence assessment linear layer performs matrix multiplication to generate the original logarithmic confidence matrix. A sigmoid activation function is applied to the original logarithmic confidence matrix to generate the final confidence score matrix. The class probability distribution matrix, the final bounding box coordinate matrix, and the final confidence score matrix are combined according to the query vector dimension to generate a target instance for each query vector in the posterior tracking query matrix. The original posterior tracking query matrix is ​​used as the embedding feature of each target instance and associated with the corresponding target instance. All target instances and their corresponding embedding features are aggregated to form the multi-target tracking result for the current frame.

[0112] Output the multi-target tracking results and the posterior tracking query matrix.

[0113] This step utilizes an improved TransTrack model, fusing temporal prediction and multimodal context to achieve high-precision multi-target tracking. The query decoupling layer separates historical tracking information from the current detection task, laying the foundation for subsequent parallel processing. The temporal correlation modeling layer uses a Mamba network to predictively model historical trajectories, enhancing the model's ability to predict target motion. The detection feature projection layer maps detection queries and multimodal feature maps into query, key, and value matrices, respectively, preparing data for interactive computation. The detection interactive encoding layer, through a cross-attention mechanism, enables detection queries to focus on key information in the multimodal feature maps, generating a richer enhanced detection query matrix. The tracking feature fusion projection layer integrates visual features and detection semantic information, providing a dual information source for tracking. The tracking interactive encoding layer, through a cross-attention mechanism, enables tracking queries to effectively fuse visual context and detection semantics.

[0114] In this embodiment, S3 specifically includes:

[0115] S31. Treat each target instance in the multi-target tracking result as a graph node, and extract and concatenate the bounding box, category and embedding features of the target instance into a node feature vector;

[0116] S32. Combine the feature vectors of all nodes into a node feature matrix; perform parallelized matrix multiplication operations on the node feature matrix and the corresponding transpose matrix to generate the original relation score matrix.

[0117] S33. Input the original relation score matrix into the parallel execution of the relation strength evaluation linear layer and the relation type classification linear layer. The relation strength evaluation linear layer calculates and outputs the relation strength matrix through a fully connected layer and a sigmoid activation function; the relation type classification linear layer calculates and outputs the relation type matrix through a fully connected layer and a softmax activation function.

[0118] S34. Based on the relation strength matrix and relation type matrix, generate weighted graph edges connecting corresponding nodes;

[0119] S35. Construct a dynamic spatiotemporal relationship graph based on all graph nodes, node feature vectors, and weighted graph edges.

[0120] This step models the dynamic spatiotemporal relationships between targets by instantiating the multi-target tracking results into a graph structure. Each target instance in the multi-target tracking results is treated as a graph node, and node feature vectors are extracted and concatenated from the bounding box, category, and embedding features of the target instances to comprehensively represent the state of each target. An initial relationship score matrix is ​​generated to preliminarily quantify the pairwise relationships between all nodes. A relationship strength matrix is ​​calculated and output to assess the tightness of relationships; a relationship type matrix is ​​calculated and output to identify the specific categories of relationships. A dynamic spatiotemporal relationship graph is constructed to form a structured relationship network, providing a data foundation for subsequent priority assessment and interaction analysis.

[0121] In this embodiment, S4:

[0122] S41. Receive the dynamic spatiotemporal relationship graph, aggregate the neighborhood information of each graph node through the message passing mechanism of the TGN network, update the initial node features, and generate the updated node feature matrix.

[0123] S42. Input the updated node feature matrix into the priority evaluation linear head, calculate and output the target priority score for each graph node through linear transformation of the fully connected layer and Sigmoid activation function;

[0124] S43. Sort all targets in descending order according to their priority scores, and classify them into three priority levels: critical, important, and ordinary, based on a preset score threshold. Output the target priority classification results.

[0125] This step achieves intelligent prioritization of multiple targets by aggregating and evaluating information on a dynamic relationship graph. It aggregates the neighborhood information of each graph node, integrating node features with contextual information. The priority score for each graph node is calculated and output to quantify the importance of each target. Based on preset score thresholds, targets are categorized into three priority levels: critical, important, and ordinary. The target priority classification results are output, providing a clear decision-making basis for subsequent tracking resource allocation and camera control.

[0126] In this embodiment, the TGN network includes a node memory module, a message encoder, a temporal attention module, a neighborhood aggregator, and an embedding function layer;

[0127] The node memory module is used to maintain a state vector that evolves over time for each node in the graph, which serves as the memory state vector of the node at the current moment.

[0128] The message encoder is used to encode interaction events when any two nodes in the graph interact. This includes receiving an interaction event containing a source node ID, a target node ID, an interaction timestamp, and an interaction feature vector; obtaining the memory state vector of the source node from the node memory module through a preset source node embedding lookup layer; obtaining the memory state vector of the target node from the node memory module through a preset target node embedding lookup layer; concatenating the source node memory state vector, the target node memory state vector, and the interaction feature vector along the feature dimension to generate the original message feature; and inputting the original message feature into a message encoding multilayer perceptron, performing linear transformation and ReLU activation function processing to generate the interaction message vector.

[0129] The temporal attention module is used to calculate the attention weights of the memory state vector and the corresponding historical interaction message vector of each node. This includes receiving the target node ID, the current update timestamp, and all historical interaction message vectors related to the current target node; multiplying the time difference between the current update timestamp and the timestamp of each historical message vector by a preset decay coefficient, taking the negative value, to obtain the decay exponent; using the decay exponent as a power, performing a natural exponentiation operation to obtain the original time decay factor; inputting each historical interaction message vector into the attention query projection layer to generate a query vector; inputting each historical interaction message vector into the attention key projection layer to generate a key vector; calculating the original relevance score through the dot product operation of the query vector and the key vector; multiplying the original relevance score element-wise with the original time decay factor to generate the final attention score; and applying the Softmax activation function to the final attention score to generate normalized attention weights.

[0130] The neighborhood aggregator is used to weight and aggregate historical message vectors according to attention weights to generate neighborhood aggregated message vectors.

[0131] The embedding function layer is used to combine the current memory state vector of a node and the neighborhood aggregation message vector to update the memory state of the node and generate updated node features. This includes obtaining the memory state vector of the target node before the update from the node memory module; concatenating the memory state vector of the target node before the update with the neighborhood aggregation message vector along the feature dimension to generate an updated candidate feature vector; inputting the updated candidate feature vector into a memory update multilayer perceptron, performing linear transformation and ReLU activation function processing to generate a memory update increment; performing a residual connection between the memory update increment and the memory state vector of the target node before the update, and performing layer normalization processing to generate the updated memory state vector of the target node; using the updated memory state vector as the final node feature of the current node at the current time and writing it back to the node memory module for storage; and aggregating the final node features of all nodes at the current time to jointly form the updated node feature matrix.

[0132] This step uses a TGN network to update the dynamic relationship graph with time-aware node features to capture the evolving interactions between targets. The node memory module maintains a state vector that evolves over time for each node, enabling continuous memorization of node states. The message encoder encodes interaction events between nodes into interaction message vectors containing the source and target node states and interaction features, thus transforming interactions into computable information. The temporal attention module calculates attention weights for historical interaction messages, prioritizing recent and relevant interactions during aggregation.

[0133] In this embodiment, S5 specifically includes:

[0134] S51. Receive the posterior tracking query matrix and the target identifiers of all critical and important targets; index and extract the corresponding query vectors from the posterior tracking query matrix according to the target identifiers, and generate a set of critical and important query vectors;

[0135] S52. Extract the corresponding historical location sequence and velocity information for each query vector in the key-level and importance-level query vector sets;

[0136] S53. Organize the historical position sequence and velocity information into an input sequence according to time steps and input it into the LSTM network. Process the input sequence through internal state update and output the prediction results of the three-dimensional spatial position and future radial velocity within a preset time period.

[0137] This step uses an LSTM network to predict the future motion state of high-priority targets, achieving proactive situational awareness. It generates key and importance-level query vector sets to accurately locate the key targets requiring prediction. Historical position sequences and velocity information are extracted to construct time-series data reflecting the target's motion patterns. Finally, it predicts the target's three-dimensional spatial position and future radial velocity within a preset time period, thereby enabling accurate projection of the key target's future trajectory and state.

[0138] In this embodiment, S6 specifically includes:

[0139] S61. Receive the future three-dimensional spatial position and future radial velocity of important and critical targets, as well as the dynamic spatiotemporal relationship diagram;

[0140] S62. Project the future 3D spatial position along the depth axis to obtain the future depth value; input the future depth value into a preset focus distance mapping table; look up the index in the focus distance mapping table that is closest to the future depth value; extract the corresponding initial focus distance based on the index;

[0141] S63. Input the future radial velocity into the velocity compensation coefficient table, query and extract the corresponding velocity compensation coefficient; multiply the future radial velocity by the velocity compensation coefficient to generate the distance compensation amount;

[0142] S64. Add the initial focusing distance to the distance compensation amount to generate the future focusing plane position;

[0143] S65. Extract the spatiotemporal relationship weights between key-level targets and important-level targets from the dynamic spatiotemporal relationship diagram, and determine whether key-level targets and important-level targets constitute a strong interaction unit, including comparing the spatiotemporal relationship weights with a preset interaction threshold; if the spatiotemporal relationship weights are greater than or equal to the interaction threshold, it is determined that they constitute a strong interaction unit, otherwise it is determined that they do not constitute a strong interaction unit.

[0144] S66. If it is determined that a strong interactive unit is formed, the future depth values ​​of key-level targets and important-level targets are extracted, the maximum and minimum depth values ​​are calculated, and an interactive depth range is generated; if it is determined that a strong interactive unit is not formed, the future depth values ​​of key-level targets are extracted, the maximum and minimum depth values ​​are calculated, and an independent depth range is generated.

[0145] S67. Generate focus control commands and aperture control commands based on the future focus plane position and the interactive depth of field range or independent depth of field range.

[0146] This step generates intelligent camera focusing and aperture control commands by integrating future motion predictions of high-priority targets with spatiotemporal relationship analysis. It receives future positions and velocities, along with a relationship graph, to obtain decision information. The future positions are projected as depth values, and a mapping table is consulted to determine the basic focus position. Distance compensation is generated to address motion blur by consulting a velocity compensation table and performing calculations. The weights in the relationship graph are analyzed to determine whether targets constitute strong interactive units, deciding whether to adopt a group or independent depth-of-field strategy. Based on the determination results, the interactive or independent depth-of-field range is calculated to intelligently adjust depth-of-field coverage. Finally, by combining the focus plane position and depth-of-field range, focus and aperture control commands are generated, achieving automated and optimized adjustment of camera parameters.

[0147] In this embodiment, S7 specifically includes:

[0148] S71, receives focus control commands and aperture control commands;

[0149] S72. Based on the focus control command, analyze the target drive distance and drive direction, including reading the distance code value in the focus control command and determining the target drive distance according to the preset code-distance mapping table; read the direction position in the focus control command to determine whether the lens motor moves towards the near focal end or the far focal end;

[0150] S73. According to the aperture control command, analyze the target aperture value and driving direction, including reading the aperture setting code in the aperture control command and determining the target aperture value according to the preset target aperture value code mapping table; compare the target aperture value with the current aperture value. If the target aperture value is greater than the current aperture value, the driving direction is to open the aperture wide; otherwise, the aperture is to close the aperture.

[0151] S74. Convert the target driving distance and driving direction into a first pulse width modulation signal for the drive motor, including subtracting the current position distance of the lens from the target driving distance and dividing by the single-step movement distance of the lens motor to calculate the number of steps the motor needs to rotate; calculate the corresponding pulse duty cycle for each step according to the preset speed curve, including loading a preset step duty cycle index table, determining the motion stage to which the current step belongs based on the total number of steps and the preset acceleration / deceleration strategy; if it is in the acceleration stage, use the current step as an index to directly look up the corresponding normalized pulse width modulation signal from the step duty cycle index table. The duty cycle value is adjusted. If the speed is constant, the normalized maximum duty cycle value at the end of the step duty cycle index table is used. If the speed is decelerated, the total number of steps is subtracted from the current number of steps to obtain the reverse step index. The reverse step index is used to look up the corresponding normalized duty cycle value in the step duty cycle index table. The normalized duty cycle value is multiplied by the preset system maximum duty cycle to obtain the pulse duty cycle. The pulse duty cycles of all steps are combined to form a pulse duty cycle sequence. Combined with the driving direction information, a first pulse width modulation signal containing the direction, number of steps, and pulse duty cycle sequence is generated.

[0152] S75. Convert the target aperture value into a second pulse width modulation signal to drive the aperture blades, including: querying a preset aperture-angle mapping table based on the target aperture value to determine the target angle that the aperture blades need to rotate; acquiring the current angle of the aperture, calculating the absolute difference between the target angle and the current angle to obtain the angle deviation; calculating the number of steps the motor needs to rotate based on the angle deviation and the single-step rotation angle of the aperture motor; loading a preset aperture motion duty cycle table, using the calculated number of steps as an index to query the corresponding normalized duty cycle value from the aperture motion duty cycle table; multiplying the queried normalized duty cycle value by the preset maximum duty cycle of the aperture drive system to obtain the pulse duty cycle; combining the pulse duty cycles of all steps to form a pulse duty cycle sequence; generating a second pulse width modulation signal containing the drive direction, number of steps, and pulse duty cycle sequence.

[0153] S76. Send the first pulse width modulation signal to the lens motor to control the lens to focus; send the second pulse width modulation signal to the aperture drive unit to control the aperture blades to adjust to the target depth of field.

[0154] This step achieves physical control of the camera's focus and aperture by parsing and converting high-level control commands into precise motor drive signals. Based on the focus command, the target drive distance and direction are parsed to determine the precise lens movement parameters. Based on the aperture command, the target aperture value and drive direction are parsed to clarify the specific depth-of-field adjustment requirements. A first pulse-width modulation (PWM) signal is sent to the lens motor, and a second PWM signal is sent to the aperture drive unit, thereby completing automated, physically precise control of focus and depth of field.

[0155] Example 1:

[0156] To verify the feasibility of this invention in autofocusing of intelligent optical systems, it was applied to an intelligent camera module (hereinafter referred to as "Camera A") of a high-end security monitoring system. Traditional security monitoring cameras typically employ passive focusing algorithms based on contrast detection or phase detection. These methods are not only slow in response but also ineffective in handling complex scenes with multiple targets moving at high speeds, frequent occlusion, and drastic depth changes, easily leading to focusing lag, target defocusing, and image blurring. To address these issues, Camera A adopted the automatic optical focal length adjustment control system based on intelligent sensors proposed in this invention.

[0157] During implementation, camera A first uses its integrated multimodal sensor array to acquire the main image, raw distance map, and inertial measurement unit data in real time. After data synchronization, coordinate alignment, and feature extraction processing, a unified multimodal feature map is generated. Simultaneously, camera A uses its built-in target detection and tracking module to initially identify multiple moving targets in the scene.

[0158] Camera A, employing an improved TransTrack model that integrates the predictive modeling capabilities of the Mamba network, performs temporal correlation analysis on historical trajectories, outputting high-precision multi-target tracking results. Subsequently, a dynamic spatiotemporal graph construction module instantiates targets as graph nodes and utilizes bi-branch decoding to generate weighted graph edges, modeling the complex interactions between targets. Building upon this, a temporal graph network (TGN) aggregates neighborhood information to intelligently evaluate the priority of each target, identifying critical and important targets. Finally, an LSTM network predicts the future 3D spatial positions and radial velocities of these high-priority targets, providing a forward-looking decision-making basis for camera control.

[0159] During implementation, the technical team behind Camera A discovered that, compared to traditional passive focusing methods, the method of this invention significantly improves image clarity and response speed in dynamic scenes. Traditional methods cannot predict target movement and ignore the relationships between targets, resulting in persistent focusing lag. In contrast, the method of this invention achieves "predictive" precise focusing on key targets through predictive tracking, relationship awareness, and intelligent decision-making.

[0160] To further verify the actual performance of the method of the present invention, camera A conducted a detailed comparative test between the method of the present invention and the traditional method. The specific performance data is shown in Table 1:

[0161] Table 1. Performance Comparison of Optical Focal Length Adjustment Methods for Camera A

[0162] Key target focus success rate (%) 78.5 96.2 +22.5% Average focus time (milliseconds) 350 85 -75.7% Moving target image sharpness (MTFp@0.5) 0.42 0.68 +61.9% Defocusing rate (%) in multi-target interaction scenarios 25.3 4.1 -83.8% Error between predicted focal plane and actual focal point (mm) 0.85 0.18 -78.8% Time (milliseconds) for the first clear image after a scene change 620 150 -75.8% System power consumption (W) 2.8 3.1 +10.7% User subjective satisfaction rating (out of 5) 3.2 4.7 +46.9%

[0163] As shown in Table 1, the focusing and imaging performance of camera A was comprehensively improved after applying the method of this invention. The success rate of focusing on key targets increased from 78.5% with the traditional method to 96.2%, and the image clarity of moving targets improved by 61.9%, significantly enhancing the usability of the monitoring footage. The average focusing time decreased from 350 milliseconds to 85 milliseconds, and the time for the first clear image after scene switching was reduced by 75.8%, greatly enhancing the system's response speed and real-time performance. The defocusing rate in multi-target interactive scenes decreased significantly from 25.3% to 4.1%, and the error between the predicted focus plane and the actual focus was reduced by 78.8%, demonstrating the superior performance of the method of this invention in complex dynamic scenes. Although the system power consumption increased slightly, the user's subjective satisfaction score jumped from 3.2 to 4.7, indicating a fundamental improvement in user experience.

[0164] Through the method of this invention, camera A successfully achieves forward-looking and high-precision autofocus in complex dynamic scenes, effectively improving the quality and information value of surveillance video, ensuring continuous and clear tracking of key targets, and greatly enhancing the environmental adaptability and intelligence level of the intelligent camera, providing strong technical support for the next generation of intelligent visual perception systems.

[0165] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An automatic optical focal length adjustment control system based on intelligent sensors, characterized in that, Includes the following modules: The multimodal feature fusion module is used to simultaneously acquire data from multiple sources of sensors, extract and fuse visual, geometric and motion modal features, and generate a multimodal feature map; The predictive tracking modeling module is used to improve the TransTrack model by introducing the scanning and state update mechanism of the Mamba network, predictively modeling historical trajectories, and fusing the dual context of multimodal feature maps and target detection results to output multi-target tracking results and posterior tracking query matrix. The dynamic relationship graph construction module is used to instantiate multi-target tracking results into graph nodes, decode the original relationship scores in parallel through two branches and generate weighted graph edges to form a dynamic spatiotemporal relationship graph; The graph network priority evaluation module is used to aggregate neighborhood information and update node features through a TGN network based on a dynamic spatiotemporal relationship graph, and generate a priority score for each target using a priority evaluation linear head and output the priority classification result. The target trajectory prediction module is used to filter the corresponding query vector from the posterior tracking query matrix based on the priority classification results, and predict the future three-dimensional spatial position and future radial velocity of the target through an LSTM network. The camera control command generation module is used to calculate the focal plane based on the target's future three-dimensional spatial position and future radial velocity, and combine the dynamic spatiotemporal relationship diagram to determine the target interaction relationship and the depth of field range, and generate focus and aperture control commands. The lens drive execution module is used to send focus control commands and aperture control commands to the lens drive unit, and control the lens motor and aperture blades to perform corresponding focus and depth-of-field adjustment actions.

2. The automatic optical focal length adjustment control system based on intelligent sensors according to claim 1, characterized in that, The modules are connected through the following steps: S1. Simultaneously acquire data from multiple sensor sources, extract and fuse visual, geometric and motion modal features to generate a multimodal feature map; S2. The TransTrack model is improved by introducing the scanning and state update mechanism of the Mamba network, which predictively models the historical trajectory and integrates the dual context of multimodal feature maps and target detection results to output multi-target tracking results and posterior tracking query matrix. S3. Instantiate the multi-target tracking results into graph nodes, decode the original relation scores in parallel through two branches and generate weighted graph edges to form a dynamic spatiotemporal relation graph; S4. Based on the dynamic spatiotemporal relationship graph, the neighborhood information is aggregated and the node features are updated through the TGN network. The priority evaluation linear head is used to generate the priority score of each target and output the priority classification result. S5. Based on the priority classification results, select the corresponding query vector from the posterior tracking query matrix, and predict the future three-dimensional spatial position and future radial velocity of the target through the LSTM network. S6. Calculate the focal plane based on the target's future three-dimensional spatial position and future radial velocity, and combine it with the dynamic spatiotemporal relationship diagram to determine the target interaction relationship and the depth of field range, and generate focus and aperture control commands. S7. Send focus control commands and aperture control commands to the lens drive unit to control the lens motor and aperture blades to perform corresponding focus and depth-of-field adjustment actions.

3. The automatic optical focal length adjustment control system based on intelligent sensors according to claim 2, characterized in that, S1 includes: S11. Real-time acquisition of main image, raw distance map and inertial measurement unit data through sensor array; S12. Input the main image into the ResNet network. Through initial convolution, residual block processing and multi-level downsampling, the first scale feature map, the second scale feature map, the third scale feature map and the fourth scale feature map are generated step by step, which together constitute a multi-scale visual feature map. S13. Perform median filtering and bilinear interpolation upsampling on the original distance map, align it to the main image coordinate system by projecting it onto the extrinsic parameters, and generate a geometric feature map by convolution. S14. Based on the inertial measurement unit data, a motion coding vector is generated through Kalman filtering and linear transformation, and a motion feature map aligned with the visual feature map is constructed through spatial expansion and broadcasting operations. S15. The second-scale feature map, geometric feature map, and motion feature map are concatenated along the channel dimension to generate an initial fused feature map, and then convolutional processing is performed to output a multimodal feature map.

4. The automatic optical focal length adjustment control system based on intelligent sensors according to claim 2, characterized in that, The improved TransTrack model includes a query decoupling layer, a temporal correlation modeling layer, a detection feature projection layer, a detection interactive coding layer, a tracking feature fusion projection layer, a tracking interactive coding layer, and a prediction output generation layer. The query decoupling layer is used to receive the posterior tracking query matrix output from the previous frame and the preset learnable detection query matrix of the current frame; the posterior tracking query matrix is ​​input into the temporal correlation modeling layer, and the detection query matrix is ​​input into the detection feature projection layer. The temporal correlation modeling layer is used to model the temporal dependencies of the input posterior tracking query matrix through the scanning and state update mechanism of the Mamba network, and to predict and generate the prior tracking query matrix of the current frame. The detection feature projection layer is used to receive the detection query stream and the multimodal feature map; the detection query projection layer maps the detection query matrix into a detection query embedding matrix; and the feature map key projection layer maps the multimodal feature map into a visual feature key matrix. The multimodal feature map is mapped into a visual feature value matrix through a feature map value projection layer; The detection interactive coding layer is used to perform parallel cross-attention calculations on the detection query embedding matrix, the visual feature key matrix, and the visual feature value matrix, and to assign contextual information from the feature map to each query vector in the detection query embedding matrix to generate an enhanced detection query matrix. The tracking feature fusion projection layer maps the prior tracking query, enhanced detection query, and multimodal feature map into corresponding query, key, and value matrices through multiple parallel projection layers, and fuses the key-value information of visual and detection semantics to generate a fusion context key matrix and value matrix for tracking interaction. The tracking interactive coding layer is used to perform parallel cross-attention calculations on the tracking query matrix, the fusion context key matrix, and the fusion context value matrix, and to assign dual context information from the feature map and the augmented detection query matrix to each query vector in the tracking query matrix, thereby generating the posterior tracking query matrix of the current frame. The prediction output generation layer is used to decode the posterior tracking query matrix of the current frame and generate a target instance for each query vector of the posterior tracking query matrix; the original posterior tracking query matrix is ​​used as the embedding feature of each target instance and associated with the corresponding target instance. Aggregate all target instances and their corresponding embedded features to form the multi-target tracking result of the current frame; Output the multi-target tracking results and the posterior tracking query matrix.

5. The automatic optical focal length adjustment control system based on intelligent sensors according to claim 2, characterized in that, S3 specifically includes: S31. Treat each target instance in the multi-target tracking result as a graph node, and extract and concatenate the bounding box, category and embedding features of the target instance into a node feature vector; S32. Combine the feature vectors of all nodes into a node feature matrix; perform parallelized matrix multiplication operations on the node feature matrix and the corresponding transpose matrix to generate the original relation score matrix. S33. Input the original relation score matrix into the parallel execution of the relation strength evaluation linear layer and the relation type classification linear layer. The relation strength evaluation linear layer calculates and outputs the relation strength matrix through a fully connected layer and a sigmoid activation function; the relation type classification linear layer calculates and outputs the relation type matrix through a fully connected layer and a softmax activation function. S34. Based on the relation strength matrix and relation type matrix, generate weighted graph edges connecting corresponding nodes; S35. Construct a dynamic spatiotemporal relationship graph based on all graph nodes, node feature vectors, and weighted graph edges.

6. The automatic optical focal length adjustment control system based on intelligent sensors according to claim 2, characterized in that, S4 specifically includes: S41. Receive the dynamic spatiotemporal relationship graph, aggregate the neighborhood information of each graph node through the message passing mechanism of the TGN network, update the initial node features, and generate the updated node feature matrix. S42. Input the updated node feature matrix into the priority evaluation linear head, calculate and output the target priority score for each graph node through linear transformation of the fully connected layer and the Sigmoid activation function; S43. Sort all targets in descending order according to their priority scores, and classify them into three priority levels: critical, important, and ordinary, based on a preset score threshold. Output the target priority classification results.

7. The automatic optical focal length adjustment control system based on intelligent sensors according to claim 2, characterized in that, The TGN network includes a node memory module, a message encoder, a temporal attention module, a neighborhood aggregator, and an embedding function layer; The node memory module is used to maintain a state vector that evolves over time for each node in the graph, which serves as the memory state vector of the node at the current moment. The message encoder is used to encode the interaction event when an interaction event occurs between any two nodes in the graph, and generate an interaction message vector. The temporal attention module is used to calculate the attention weight between the memory state vector of each node and the corresponding historical interaction message vector. The neighborhood aggregator is used to weight and aggregate historical message vectors according to attention weights to generate neighborhood aggregated message vectors. The embedding function layer is used to combine the node's current memory state vector and the neighborhood aggregation message vector to update the node's memory state and generate updated node features. Aggregate the final node features of all nodes at the current time to form the updated node feature matrix.

8. The automatic optical focal length adjustment control system based on intelligent sensors according to claim 2, characterized in that, S5 specifically includes: S51. Receive the posterior tracking query matrix and the target identifiers of all critical and important targets; index and extract the corresponding query vectors from the posterior tracking query matrix according to the target identifiers, and generate a set of critical and important query vectors; S52. Extract the corresponding historical location sequence and velocity information for each query vector in the key-level and importance-level query vector sets; S53. Organize the historical position sequence and velocity information into an input sequence according to time steps and input it into the LSTM network. Process the input sequence through internal state update and output the prediction results of the three-dimensional spatial position and future radial velocity within a preset time period.

9. The automatic optical focal length adjustment control system based on intelligent sensors according to claim 2, characterized in that, S6 specifically includes: S61. Receive the future three-dimensional spatial position and future radial velocity of important and critical targets, as well as the dynamic spatiotemporal relationship diagram; S62. Project the future 3D spatial position along the depth axis to obtain the future depth value; input the future depth value into a preset focus distance mapping table; look up the index in the focus distance mapping table that is closest to the future depth value; extract the corresponding initial focus distance based on the index; S63. Input the future radial velocity into the velocity compensation coefficient table, query and extract the corresponding velocity compensation coefficient; multiply the future radial velocity by the velocity compensation coefficient to generate the distance compensation amount; S64. Add the initial focusing distance to the distance compensation amount to generate the future focusing plane position; S65. Determine whether key-level objectives and important-level objectives constitute a strong interactive unit; S66. If it is determined that a strong interactive unit is formed, the future depth values ​​of key-level targets and important-level targets are extracted, the maximum and minimum depth values ​​are calculated, and an interactive depth range is generated; if it is determined that a strong interactive unit is not formed, the future depth values ​​of key-level targets are extracted, the maximum and minimum depth values ​​are calculated, and an independent depth range is generated. S67. Generate focus control commands and aperture control commands based on the future focus plane position and the interactive depth of field range or independent depth of field range.

10. The automatic optical focal length adjustment control system based on intelligent sensors according to claim 2, characterized in that, Specifically, S7 includes: S71, receives focus control commands and aperture control commands; S72. Based on the focus control command, analyze the target driving distance and driving direction; S73. Based on the aperture control command, analyze the target aperture value and drive direction; S74. Convert the target driving distance and driving direction into a first pulse width modulation signal for the drive motor; S75. Convert the target aperture value into a second pulse width modulation signal that drives the aperture blades; S76. Send the first pulse width modulation signal to the lens motor to control the lens to focus; send the second pulse width modulation signal to the aperture drive unit to control the aperture blades to adjust to the target depth of field.

Images