Neuromorphic visual object tracking method and system based on image processing

Abstract

The present application provides a neuromorphic visual target tracking method and system based on image processing, and relates to the technical field of neuromorphic computing. The present application fuses event camera and conventional image sensor input, constructs a joint input tensor, introduces multi-scale convolution and synapse event driving mechanism, realizes fast response to high-speed moving targets and stable feature extraction, effectively avoids the deletion of coding blocks and the loss of tracking caused by the rapid movement of the target, thereby reducing the system delay. At the same time, combined with the saliency entropy difference evaluation and dynamic brightness enhancement mechanism, the target judgment accuracy in low light and complex background is improved; through the saliency weight function and confidence calculation, the redundant noise block is screened out, the redundant calculation burden is reduced, and the system lightweight characteristic is guaranteed. Further, the memory trajectory tensor based on the recurrent neural network and the dynamic template adjustment mechanism are introduced, the time sequence consistency verification and adaptive update are realized, and the stability and robustness of the tracking process are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of neuromorphic computing technology, and in particular to a neuromorphic visual target tracking method and system based on image processing. Background Art

[0002] In recent years, the integration of deep learning and neuromorphic chips has further improved their real-time tracking performance in complex environments, becoming a research hotspot in fields such as autonomous driving and robotics.

[0003] The prior art, with publication number CN118628530A, is titled "A Lightweight Transformer-Based Neuromorphic Visual Target Tracking Method." The specific steps include: converting neuromorphic visual data into a frame image; segmenting the converted frame image into multiple sub-image blocks; eliminating code blocks irrelevant to the tracked target using an entropy-guided code block elimination method; eliminating redundant code blocks from the remaining code blocks using a content-guided code block elimination method; and inputting the remaining code blocks into a prediction head to achieve target positioning. The present invention is capable of tracking targets in complex scenes such as fast motion and low light in real time. The entropy-guided code block elimination technique can evaluate the code block elimination rate based on the amount of information in the scene, thereby filtering out more useless information. The content-guided code block elimination technique retains code blocks that are more relevant to the tracked target, effectively removing information irrelevant to target tracking, lightweighting the model while removing interference from background noise.

[0004] However, when applied to drones, these technical solutions often suffer from insufficient dynamic adaptability in high-speed motion scenarios, leading to transient events in which coding blocks mistakenly delete fast-moving targets, resulting in tracking loss and triggering reinitialization, increasing latency.

[0005] Secondly, in low-light or complex backgrounds, the robustness is limited. Entropy evaluation is easily affected by noise and misjudgment, which not only reduces tracking accuracy but also offsets the lightweight advantage due to redundant calculations.

[0006] The above technical shortcomings thus form a vicious cycle - target loss and noise misjudgment superimpose on each other, jointly deteriorating the real-time performance and reliability of the system.

[0007] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0008] The purpose of the present invention is to provide a neuromorphic visual target tracking method and system based on image processing to solve the problems raised in the above background technology.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] The neuromorphic visual target tracking method based on image processing includes the following steps:

[0011] Step S1: Obtain multimodal visual input of the target area, including event stream information and frame image data, and perform spatiotemporal alignment to construct a joint input tensor Ic;

[0012] Step S2: Perform neuromorphic feature extraction on the joint input tensor Ic, use a multi-scale convolutional network to extract spatial features, and construct a synaptic event-driven mechanism to extract time series change features;

[0013] Calculate the current inter-frame motion vector Vm and adaptively adjust the receptive field size based on the joint input tensor Ic;

[0014] Step S3: Based on the spatial domain features, extract the local information entropy Hl and the global background entropy Hg in the target area, and calculate the target saliency entropy difference ΔH; and introduce dynamic brightness enhancement for local adjustment;

[0015] Step S4: Based on the current inter-frame motion vector Vm and the target saliency entropy difference ΔH, calculate and perform a screening operation on the candidate coding blocks according to the saliency weight function w, filter out redundant blocks triggered by background noise, and calculate the tracking confidence Ct to update the target state;

[0016] Step S5: Construct a memory trace tensor Tm containing the historical tracking state, and perform temporal consistency verification by the recurrent neural network, combining the current tracking confidence Ct with the previous moment tracking confidence Ct -1 Perform dynamic template adjustment and neural weight fine-tuning.

[0017] Furthermore, event stream information is collected by an event camera, and the event stream information includes pixel brightness change data triggered asynchronously in time, and is recorded as a sequence of event points including a timestamp, a pixel position, and a polarity identifier;

[0018] Frame image data is collected by image sensors and contains spatial brightness distribution information of the target area at a specific time point. It is used to record the structural details, texture features and background relationship of the target in a static or slowly changing state.

[0019] Furthermore, according to the timestamp of each event in the event stream, corresponding interpolation and the selection of the closest frame image time point are performed to align the event stream information with the frame image data in time;

[0020] In the spatial dimension, the coordinates of event pixels and image pixels are corrected and mapped according to the sensor field of view, resolution and distortion model;

[0021] After completing the spatiotemporal alignment, the event stream information is encoded into a sparse tensor E(x, y, t), and the frame image data is encoded into a dense tensor F(x, y, t), and then fused according to the channel dimension to construct a joint input tensor Ic.

[0022] Furthermore, the joint input tensor Ic is input into a multi-scale convolutional network to extract spatial feature maps at multiple scales, which are labeled F1, F2, ..., Fs in sequence. Based on the activation value set K(x, y) corresponding to the target area in the feature map at each scale, the average response value R is calculated.

[0023] Finally, the spatial feature maps at all scales are labeled with response values ​​to form R1, R2, ..., Rs. The average response values ​​R at all scales are sorted in descending order, and the scale feature layer with the largest average response value R is selected as the dominant spatial feature map of the current frame input.

[0024] Construct a synaptic event-driven mechanism, encode the time distribution of the sparse tensor E(x, y, t) into a pulse sequence, and calculate the neuron pulse discharge frequency Fsp based on the pulse sequence;

[0025] Finally, the spatial feature map is fused with the time series impulse response feature to generate a neuromorphic joint feature map.

[0026] Furthermore, the optical flow estimation algorithm is used to calculate the current inter-frame motion vector Vm by the target position change in two consecutive frames of image data in the neuromorphic joint feature map. Subsequently, the current inter-frame motion vector Vm and the joint input tensor Ic are dimensionlessly processed and correlated and fitted to calculate the receptive field adjustment factor Gy. The specific calculation formula is as follows:

[0027] ;

[0028] Where, Represents the square root of the sum of the elements of the tensor, which is used to measure the overall numerical strength; e is a small constant to prevent the denominator from being zero; is the matrix inner product, which represents the sum of the element-wise products of Vm and Ic;

[0029] The preset receptive field adjustment thresholds include G1 and G2, and G1>G2. Compare and evaluate with the receptive field adjustment factor Gy to adaptively adjust the receptive field size. The specific contents are as follows:

[0030] If the receptive field adjustment factor Gy ≥ G1, it is determined to be a strong motion association, and the receptive field is expanded to capture a wider range of features;

[0031] If the receptive field adjustment factor Gy≤G2, it is determined to be a weak motion association, and the receptive field is reduced to reduce redundant calculations;

[0032] G2<receptive field adjustment factor Gy<G1: maintain the current receptive field and avoid frequent adjustments.

[0033] Furthermore, the target saliency entropy difference ΔH is calculated based on the local information entropy Hl and the global background entropy Hg. The specific calculation formula is as follows:

[0034] ;

[0035] The preset entropy threshold H is compared with the target significance entropy difference ΔH for evaluation. The specific evaluation content is as follows:

[0036] When the entropy value threshold H ≥ the target saliency entropy difference ΔH, it is determined to be a target salient area, the current area is retained, and participates in tracking prediction;

[0037] When the entropy value threshold H is less than the target significance entropy difference ΔH, it is judged as a noise area, and dynamic brightness enhancement is triggered at this time; it includes calculating the adaptive coefficient d based on the local information entropy Hl of the current area, and performing nonlinear enhancement processing on the pixel value I of the current area based on the adaptive coefficient d, while setting the upper limit of the pixel value.

[0038] Furthermore, the significance weight function w is calculated by the following formula:

[0039] ;

[0040] The preset significance weight threshold WA is compared with the significance weight function w to screen out redundant blocks triggered by background noise. The specific evaluation content is as follows:

[0041] When the saliency weight function w ≥ the saliency weight threshold WA, it is determined to be a valid target candidate block, which is then retained and participates in subsequent tracking prediction;

[0042] When the saliency weight function w is less than the saliency weight threshold WA, it is determined to be a background noise redundant block and is directly removed without participating in the tracking calculation.

[0043] Furthermore, based on the filtered coding block, the similarity S is obtained by calculating the current coding block features and the historical target template features. The tracking confidence Ct is calculated using the similarity S. The specific calculation formula is:

[0044] ;

[0045] Where M represents the total number of candidate coding blocks that are screened and retained; Indicates the matching value of the i-th coding block, ranging from 0 to 1;

[0046] The preset tracking confidence threshold C is compared with the tracking confidence Ct for evaluation. The specific contents are as follows;

[0047] When the tracking confidence Ct ≥ the tracking confidence threshold C, it is determined to be a valid tracking state. At this time, the target state is updated according to the current coding block, including position and scale;

[0048] When the tracking confidence Ct is less than the tracking confidence threshold C, it is determined to be a potential tracking failure and local re-detection is performed.

[0049] Furthermore, a memory trajectory tensor Tm is constructed to record the spatial position information, appearance features and confidence parameters of the target at multiple time steps;

[0050] The recurrent neural network is used to check the temporal consistency of the memory trajectory tensor Tm; by analyzing the current tracking confidence Ct and the previous moment tracking confidence Ct -1 The difference between , obtain the confidence relative change rate ΔC;

[0051] A confidence relative change threshold C is preset and compared with the confidence relative change rate ΔC for evaluation; when the confidence relative change threshold C exceeds the confidence relative change rate ΔC, the template adjustment mechanism is triggered;

[0052] The template adjustment mechanism includes: progressively updating the current target template and fusing historical templates with current frame feature information;

[0053] The connection weights of the recurrent neural network are adaptively fine-tuned based on the relative change rate of confidence ΔC, including dynamically adjusting the network connection weights using a gradient update method.

[0054] The neuromorphic visual target tracking system based on image processing includes:

[0055] The multimodal fusion perception module obtains multimodal visual input of the target area, including event stream information and frame image data, and performs spatiotemporal alignment to construct a joint input tensor Ic;

[0056] The neural feature extraction module extracts neuromorphic features from the joint input tensor Ic, uses a multi-scale convolutional network to extract spatial features, and constructs a synaptic event-driven mechanism to extract time series change features;

[0057] Calculate the current inter-frame motion vector Vm and adaptively adjust the receptive field size based on the joint input tensor Ic;

[0058] The saliency judgment module extracts the local information entropy Hl and the global background entropy Hg in the target area based on spatial domain features, and calculates the target saliency entropy difference ΔH; and introduces dynamic brightness enhancement for local adjustment;

[0059] The weight screening update module calculates and performs a screening operation on the candidate coding block according to the saliency weight function w based on the current inter-frame motion vector Vm and the target saliency entropy difference ΔH, screens out redundant blocks triggered by background noise, and calculates the tracking confidence Ct to update the target state for judgment.

[0060] The memory trajectory optimization module constructs a memory trajectory tensor Tm containing historical tracking states, and performs time sequence consistency verification by a recurrent neural network, combines the current tracking confidence Ct with the tracking confidence Ct of the previous moment -1 Dynamic template adjustment and neural weight fine-tuning are performed.

[0061] Compared with the prior art, the present application has the following advantages:

[0062] According to the multi-modal fusion characteristics of the event camera and the conventional image sensor, the present application enhances the transient response capability of the system in the high-speed motion scene by constructing a joint input tensor and introducing a neuromorphic feature extraction mechanism, effectively solves the problem of coding block deletion caused by the rapid movement of the target in the application of the unmanned aerial vehicle, reduces the number of tracking loss and reinitialization, and thus significantly reduces the system delay.

[0063] The present application also improves the target determination accuracy under low light and complex background conditions through saliency entropy difference evaluation and dynamic brightness enhancement mechanism; combined with saliency weight screening and tracking confidence evaluation strategy, the redundant blocks triggered by background noise can be effectively filtered, the tracking precision is improved, and the erosion of redundant calculation on the lightweight performance is avoided; at the same time, the memory trajectory tensor constructed by the recurrent neural network and the dynamic template adjustment mechanism realize the continuous and stable tracking of the target state, and guarantee the overall robustness of the system in time sequence consistency and adaptive ability. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 The present application is a whole method flowchart;

[0065] Figure 2 The present application is a whole system framework schematic diagram. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with specific embodiments.

[0067] It should be noted that, unless otherwise defined, the technical or scientific terms used in the present invention should have the usual meanings understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0068] Example 1:

[0069] See also Figure 1 The present invention provides a technical solution: 1. A neuromorphic visual target tracking method based on image processing, characterized in that the specific steps include:

[0070] Step S1: Obtain multimodal visual input of the target area, including event stream information and frame image data, and perform spatiotemporal alignment to construct a joint input tensor Ic;

[0071] Step S2: Perform neuromorphic feature extraction on the joint input tensor Ic, use a multi-scale convolutional network to extract spatial features, and construct a synaptic event-driven mechanism to extract time series change features;

[0072] Calculate the current inter-frame motion vector Vm and adaptively adjust the receptive field size based on the joint input tensor Ic;

[0073] Step S3: Based on the spatial domain features, extract the local information entropy Hl and the global background entropy Hg in the target area, and calculate the target saliency entropy difference ΔH; and introduce dynamic brightness enhancement for local adjustment;

[0074] Step S4: Based on the current inter-frame motion vector Vm and the target saliency entropy difference ΔH, calculate and perform a screening operation on the candidate coding blocks according to the saliency weight function w, filter out redundant blocks triggered by background noise, and calculate the tracking confidence Ct to update the target state;

[0075] Step S5: Construct a memory trace tensor Tm containing the historical tracking state, and perform temporal consistency verification by the recurrent neural network, combining the current tracking confidence Ct with the previous moment tracking confidence Ct -1 Perform dynamic template adjustment and neural weight fine-tuning.

[0076] In this embodiment, S1 to S2 are the basis of multi-modal data fusion, S1 constructs a joint input tensor through the spatio-temporal alignment of event stream and frame image, and provides a spatio-temporal synchronous multi-modal data basis for the neuro-morphological feature extraction of S2, and the two together solve the perception problem of fast motion and complex scene. The dynamic receptive field adjustment of S2 directly depends on the inter-frame motion vector input by S1, forming a motion adaptive mechanism.

[0077] S2 is enhanced by S3 to S4 in terms of feature and saliency; the spatio-temporal feature extracted by S2 provides a structured representation for the local / global entropy calculation of S3, while the target saliency entropy difference ΔH of S3 quantifies the target saliency, which is jointly input into the weight function of S4 with the motion vector Vm of S2, forming a "motion-brightness" double-driven screening mechanism. S4 feeds back to the feature encoding of S2 through confidence update, forming a dynamic optimization closed loop.

[0078] S4 to S5 are time series memory and system stability, the tracking confidence Ct output by S4 is integrated with the historical state in S5 by the memory trajectory tensor, and the time series verification is realized through the recurrent neural network. The dynamic template adjustment of S5 reversely guides the candidate block screening standard of S4 to avoid false deletion, and the weight fine-tuning further optimizes the feature extraction network of S2, forming a long-time robustness guarantee.

[0079] S1 is used to enhance the transient response capability in fast motion scene;

[0080] S2 is used to realize stable feature encoding in dynamic scene;

[0081] The target saliency entropy difference ΔH in S3 is used to determine the saliency of the current tracking block, effectively reducing the tracking misjudgment caused by light interference;

[0082] S4 is used to prevent target loss and re-initialization caused by false deletion;

[0083] S5 is used for dynamic update and network weight adaptive optimization, so as to improve the tracking stability and robustness;

[0084] Each step forms a feedforward-feedback coupling through motion vector, target saliency entropy difference, confidence, etc., realizing the whole-link cooperation from transient response to continuous tracking.

[0085] Embodiment 2:

[0086] The event stream information is collected by an event camera, and the event stream information includes pixel brightness change data triggered asynchronously by time, recorded as an event point sequence containing a timestamp, a pixel position and a polarity identifier;

[0087] Frame image data is collected by image sensors and contains spatial brightness distribution information of the target area at a specific time point. It is used to record the structural details, texture features and background relationship of the target in a static or slowly changing state.

[0088] Furthermore, frame image data is used to provide spatial resolution supplement and contextual semantic support for event stream information.

[0089] According to the timestamp of each event in the event stream, the corresponding interpolation and the selection of the closest frame image time point are performed to align the event stream information with the frame image data in time;

[0090] In the spatial dimension, the coordinates of event pixels and image pixels are corrected and mapped according to the sensor field of view, resolution and distortion model;

[0091] Furthermore, this process is used to ensure that visual information from different modalities falls under the same spatial reference frame;

[0092] After completing the spatiotemporal alignment, the event stream information is encoded into a sparse tensor E(x, y, t), and the frame image data is encoded into a dense tensor F(x, y, t), and then fused according to the channel dimension to construct a joint input tensor Ic.

[0093] Furthermore, the combined input tensor Ic is Ic=[F(x,y,t),E(x,y,t)]. This tensor construction method has been applied in existing multimodal perception technologies and is often used to fuse sparse and dense visual information to balance the accuracy of temporal response and spatial semantics.

[0094] The joint input tensor Ic is fed into a multi-scale convolutional network to extract spatial feature maps at multiple scales, labeled F1, F2, ..., Fs, and the average response R is calculated based on the activation value set K(x, y) corresponding to the target region in the feature map at each scale.

[0095] Furthermore, the spatial feature maps at multiple scales are spatial feature maps formed by different convolution kernels;

[0096] Finally, the spatial feature maps at all scales are labeled with response values ​​to form R1, R2, ..., Rs. The average response values ​​R at all scales are sorted in descending order, and the scale feature layer with the largest average response value R is selected as the dominant spatial feature map of the current frame input.

[0097] Construct a synaptic event-driven mechanism, encode the time distribution of the sparse tensor E(x, y, t) into a pulse sequence, and calculate the neuron pulse discharge frequency Fsp based on the pulse sequence;

[0098] Finally, the spatial feature map is fused with the time series impulse response feature to generate a neuromorphic joint feature map.

[0099] Furthermore, the calculation formula for the average response value R under all scales is as follows:

[0100] ;

[0101] in, Represents the average response value R within the target area σ in the feature map at scale s; Indicates the corresponding target area in the feature map at scale s The vector set of all activation values ​​in , Indicates the target area The total number of pixels or feature points within is used for normalization; Represents a normalization factor used to maintain consistency with the original mean unit while preventing high response areas from interfering with the results due to their size.

[0102] The average response value R is used to measure the structural representation ability of the target at the current scale. It is usually derived from the statistical measurement of the local region feature response and is used to measure the activation degree of the feature map in the target area. It is calculated as the mean of all activation values ​​in the target area. This parameter is widely used in target detection and attention mechanisms for scale selection and dominant feature determination.

[0103] The synaptic event-driven mechanism is constructed by simulating the impulse response behavior of biological neurons. Specifically, the time series event stream output by the event camera is encoded into a discrete pulse signal sequence and input into the pulse neural network structure. In this process, each event point is converted into a synaptic input with a timestamp and polarity. When the accumulated potential exceeds the neuron threshold, a pulse discharge is triggered, and its discharge frequency fsp is recorded as a quantitative indicator of the time change characteristics. This mechanism combines the asynchrony and sparsity of the event stream to achieve low-power, high-time resolution dynamic feature extraction, effectively improving the tracking system's response capability to target timing changes.

[0104] The neuron pulse discharge frequency Fsp represents the dynamic activity of the target in the temporal dimension and is used to extract the temporal variation characteristics of the target. The specific calculation formula of the neuron pulse discharge frequency Fsp is:

[0105] ;

[0106] Where T represents the length of the observation time window; It represents the cumulative number of pulses triggered by the neuron in the time window T, that is, the number of discharges triggered by the event point;

[0107] The optical flow estimation algorithm is used to calculate the current inter-frame motion vector Vm based on the target position change in two consecutive frames of image data in the neuromorphic joint feature map;

[0108] Furthermore, the current inter-frame motion vector Vm represents the target's moving speed and direction in space. Subsequently, the current inter-frame motion vector Vm and the joint input tensor Ic are dimensionlessly processed and correlated to fit, and the receptive field adjustment factor Gy is calculated. The specific calculation formula is as follows:

[0109] ;

[0110] Where, Represents the square root of the sum of the elements of the tensor, which is used to measure the overall numerical strength; e is a small constant to prevent the denominator from being zero; is the matrix inner product, which represents the sum of the element-wise products of Vm and Ic;

[0111] The preset receptive field adjustment thresholds include G1 and G2, and G1>G2. Compare and evaluate with the receptive field adjustment factor Gy to adaptively adjust the receptive field size. The specific contents are as follows:

[0112] If the receptive field adjustment factor Gy ≥ G1, it is determined to be a strong motion association, and the receptive field is expanded to capture a wider range of features;

[0113] If the receptive field adjustment factor Gy≤G2, it is determined to be a weak motion association, and the receptive field is reduced to reduce redundant calculations;

[0114] G2<receptive field adjustment factor Gy<G1: maintain the current receptive field and avoid frequent adjustments.

[0115] In this embodiment, by introducing a multimodal fusion mechanism of event cameras and image sensors, the event stream information E(x, y, t) and the frame image data F(x, y, t) are combined to construct a joint input tensor Ic, achieving complementary advantages in temporal response and spatial semantic expression.

[0116] Through timestamp interpolation and spatial coordinate mapping, the temporal and spatial alignment of the two modal information is achieved, ensuring the spatial consistency and accuracy of feature fusion. The multi-scale convolutional network extracts the spatial feature map and calculates the average response value R of the target area at each scale. This is used to measure the ability of the current scale to express the target structural characteristics. This is used as the basis for the selection of dominant spatial domain features to improve the adaptive recognition ability of targets of different sizes.

[0117] A synaptic event-driven mechanism is constructed to convert event stream information into neuronal pulse discharge frequency Fsp, which is used to measure the dynamic activity of the target in the temporal dimension and improve the system's recognition sensitivity to rapid temporal changes. The inter-frame motion vector Vm is calculated using the target position change in the neuromorphic joint feature map. This is further correlated with the joint input tensor Ic and fitted to calculate the receptive field adjustment factor Gy, which is used to assess the target's motion intensity and degree of change in directional characteristics. The receptive field adjustment factor Gy is dynamically adjusted by comparing it with preset thresholds G1 and G2, ensuring that the system maintains an efficient perception range while suppressing redundant calculations.

[0118] The above steps together achieve low-latency, high-robustness, and high-dynamic-response target tracking capabilities, significantly enhancing the system's adaptability under complex motion and lighting conditions.

[0119] Example 3:

[0120] The target saliency entropy difference ΔH is calculated based on the local information entropy Hl and the global background entropy Hg. The specific calculation formula is as follows:

[0121] ;

[0122] Furthermore, Hl is derived from the spatiotemporal event distribution characteristics of the target candidate region, and Hg is derived from the background event statistical characteristics of the entire frame image;

[0123] The preset entropy threshold H is compared with the target significance entropy difference ΔH for evaluation. The specific evaluation content is as follows:

[0124] When the entropy value threshold H ≥ the target saliency entropy difference ΔH, it is determined to be a target salient area, the current area is retained, and participates in tracking prediction;

[0125] When the entropy value threshold H is less than the target significance entropy difference ΔH, it is judged as a noise area, and dynamic brightness enhancement is triggered at this time; it includes calculating the adaptive coefficient d based on the local information entropy Hl of the current area, and performing nonlinear enhancement processing on the pixel value I of the current area based on the adaptive coefficient d, while setting the upper limit of the pixel value.

[0126] Furthermore, the specific calculation formula of the adaptive coefficient d is as follows:

[0127] ;

[0128] in, is an adjustable gain parameter;

[0129] The significance weight function w is calculated by the following formula:

[0130] ;

[0131] The preset saliency weight threshold WA is compared with the saliency weight function w for evaluation to filter out redundant blocks triggered by background noise; the specific evaluation content is as follows:

[0132] When the saliency weight function w is greater than the saliency weight threshold WA, it is determined that the candidate block is an effective target, and at this time, the candidate block is retained and participates in subsequent tracking prediction;

[0133] When the saliency weight function w is less than the saliency weight threshold WA, it is determined that the candidate block is a background noise redundant block, and at this time, the candidate block is directly excluded and does not participate in tracking calculation.

[0134] In this embodiment, the target saliency entropy difference AH is obtained by calculating the local information entropy Hl of the target candidate region and the global background entropy Hg of the entire frame image, and the target region is screened by comparison and evaluation of the preset entropy value threshold H, wherein Hl represents the texture complexity of the target, and Hg reflects the background interference intensity;

[0135] The adaptive coefficient d based on Hl is used to perform nonlinear enhancement processing on the low-illumination region pixel value I and limit the upper limit of the pixel value to suppress overexposure. At the same time, the saliency weight function w combined with the target saliency entropy difference AH and the inter-frame motion vector Vm is compared with the preset saliency weight threshold WA for evaluation, which effectively distinguishes between real target candidate blocks and background noise redundant blocks. This scheme works through a multi-level parameterized evaluation mechanism, which not only retains the characteristics of high saliency moving targets but also significantly suppresses complex background and low-illumination noise interference, achieving higher precision in target tracking robustness and better computational efficiency compared to existing technologies.

[0136] Embodiment 4:

[0137] Based on the screened encoding block, the similarity S is obtained by calculating the current encoding block feature and the historical target template feature, and the tracking confidence Ct is calculated using the similarity S, and the specific calculation formula is:

[0138]

[0139] In the formula, M represents the total number of candidate encoding blocks selected and retained; Mi represents the matching value of the i th encoding block, which is usually in the range of 0 to 1;

[0140] The preset tracking confidence threshold C is compared with the tracking confidence Ct for evaluation, and the specific content is as follows:

[0141] When the tracking confidence Ct is greater than the tracking confidence threshold C, it is determined that the tracking state is effective, and at this time, the target state is updated according to the current encoding block, including the position and the scale;

[0142] When the tracking confidence Ct is less than the tracking confidence threshold C, it is determined that the tracking is potentially invalid, and at this time, local re-detection is performed.​

[0143] By constructing a memory trajectory tensor Tm, it is used to record the spatial location information, appearance features and confidence parameters of the target at multiple time steps;

[0144] The recurrent neural network is used to check the temporal consistency of the memory trajectory tensor Tm; by analyzing the current tracking confidence Ct and the previous moment tracking confidence Ct -1 The difference between , obtain the confidence relative change rate ΔC;

[0145] A confidence relative change threshold C is preset and compared with the confidence relative change rate ΔC for evaluation; when the confidence relative change threshold C exceeds the confidence relative change rate ΔC, the template adjustment mechanism is triggered;

[0146] The template adjustment mechanism includes: progressively updating the current target template and fusing historical templates with current frame feature information;

[0147] The connection weights of the recurrent neural network are adaptively fine-tuned based on the relative change rate of confidence ΔC, including dynamically adjusting the network connection weights using a gradient update method.

[0148] In this embodiment, the similarity S between the filtered coding block features and the historical template features is calculated and the tracking confidence Ct is obtained based on the formula, where M represents the number of valid coding blocks and Si reflects the local matching accuracy;

[0149] The tracking state is determined by comparing the preset tracking confidence threshold C with the tracking confidence Ct, and the target position and scale parameters are updated based on the evaluation content. Otherwise, local re-detection is triggered to avoid target loss. The spatiotemporal target features, including position, appearance, and confidence, are integrated by constructing a memory trajectory tensor Tm, and a recurrent neural network is used for temporal consistency verification.

[0150] Based on the current confidence Ct and the historical value Ct −1 Calculate the relative rate of change ΔC. When the relative rate of change ΔC exceeds the preset threshold C, initiate a progressive template update mechanism to dynamically integrate historical and current features. Simultaneously, use the gradient method to adaptively fine-tune the recurrent neural network weights based on the magnitude of the relative rate of change ΔC, focusing on the output gate and forget gate.

[0151] The above steps construct a closed-loop tracking system through multi-parameter collaborative evaluation, which significantly improves the tracking continuity and adaptability in complex scenarios compared to existing technologies. At the same time, the confidence-driven dynamic adjustment mechanism effectively balances the algorithm robustness and computational efficiency.

[0152] Example 5:

[0153] See also Figure 2, a neuromorphic visual target tracking system based on image processing, including:

[0154] The multimodal fusion perception module obtains multimodal visual input of the target area, including event stream information and frame image data, and performs spatiotemporal alignment to construct a joint input tensor Ic;

[0155] The neural feature extraction module extracts neuromorphic features from the joint input tensor Ic, uses a multi-scale convolutional network to extract spatial features, and constructs a synaptic event-driven mechanism to extract time series change features;

[0156] Calculate the current inter-frame motion vector Vm and adaptively adjust the receptive field size based on the joint input tensor Ic;

[0157] The saliency judgment module extracts the local information entropy Hl and the global background entropy Hg in the target area based on spatial domain features, and calculates the target saliency entropy difference ΔH; and introduces dynamic brightness enhancement for local adjustment;

[0158] The weighted screening and updating module calculates and filters candidate coding blocks based on the saliency weight function w based on the current inter-frame motion vector Vm and the target saliency entropy difference ΔH, removes redundant blocks triggered by background noise, and calculates the tracking confidence Ct to update the target state.

[0159] The memory track optimization module constructs a memory track tensor Tm containing the historical tracking state, and performs time sequence consistency verification by the recursive neural network, combining the current tracking confidence Ct with the previous moment tracking confidence Ct -1 Perform dynamic template adjustment and neural weight fine-tuning.

[0160] It should be noted that: All calculation formulas in this application document use regression analysis including but not limited to machine learning algorithms to deeply analyze the relevant parameters collected and identify their natural trends and relationships. Use professional software, such as Python's Scikit-learn library or R language, to automatically generate mathematical models that match the data. Then, objectively evaluate the performance of the model through methods such as cross-validation, and combine continuous feedback and optimization to ensure that the created formula truly reflects the inherent laws of the data, thereby ensuring its effectiveness and accuracy. In all calculation formulas in this application, the parameters in each formula are dimensionlessly processed in a consistent range to ensure that different physical quantities are compared on the same scale; dimensionless technical means include but are not limited to Min-Max Normalization and Z-Score standardization;

[0161] The technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0162] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0163] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A neuromorphic visual target tracking method based on image processing, characterized in that: The specific steps include: Step S1: Obtain multimodal visual input of the target area, including event stream information and frame image data, and perform spatiotemporal alignment to construct a joint input tensor Ic; Step S2: Perform neuromorphic feature extraction on the joint input tensor Ic, use a multi-scale convolutional network to extract spatial features, and build a synaptic event-driven mechanism to extract time series change features, specifically: The joint input tensor Ic is fed into a multi-scale convolutional network to extract spatial feature maps at multiple scales, labeled F1, F2, ..., Fs, and the average response R is calculated based on the activation value set K(x, y) corresponding to the target region in the feature map at each scale. Finally, the spatial feature maps at all scales are labeled with response values ​​to form R1, R2, ..., Rs. The average response values ​​R at all scales are sorted in descending order, and the scale feature layer with the largest average response value R is selected as the dominant spatial feature map of the current frame input. Construct a synaptic event-driven mechanism, encode the time distribution of the sparse tensor E(x, y, t) into a pulse sequence, and calculate the neuron pulse discharge frequency Fsp based on the pulse sequence; finally, fuse the spatial feature map with the time series pulse response feature to generate a neuromorphic joint feature map; Calculate the current inter-frame motion vector Vm and adaptively adjust the receptive field size based on the joint input tensor Ic; Step S3: Based on the spatial domain features, extract the local information entropy Hl and the global background entropy Hg in the target area, and calculate the target saliency entropy difference ΔH; and introduce dynamic brightness enhancement for local adjustment; Step S4: Based on the current inter-frame motion vector Vm and the target saliency entropy difference ΔH, calculate and perform a screening operation on the candidate coding blocks according to the saliency weight function w, filter out redundant blocks triggered by background noise, and calculate the tracking confidence Ct to update the target state; Step S5: Construct a memory trace tensor Tm containing the historical tracking state, and perform temporal consistency verification by the recurrent neural network, combining the current tracking confidence Ct with the previous moment tracking confidence Ct -1 Perform dynamic template adjustment and neural weight fine-tuning.

2. The neuromorphic visual target tracking method based on image processing according to claim 1, characterized in that: The event stream information is collected by an event camera, and the event stream information includes pixel brightness change data triggered asynchronously in time, and is recorded as a sequence of event points including timestamps, pixel positions, and polarity identifiers; Frame image data is collected by image sensors and contains spatial brightness distribution information of the target area at a specific time point. It is used to record the structural details, texture features and background relationship of the target in a static or slowly changing state.

3. The neuromorphic visual target tracking method based on image processing according to claim 2, characterized in that: According to the timestamp of each event in the event stream, the corresponding interpolation and the selection of the closest frame image time point are performed to align the event stream information with the frame image data in time; In the spatial dimension, the coordinates of event pixels and image pixels are corrected and mapped according to the sensor field of view, resolution and distortion model; After completing the spatiotemporal alignment, the event stream information is encoded into a sparse tensor E(x, y, t), and the frame image data is encoded into a dense tensor F(x, y, t), and then fused according to the channel dimension to construct a joint input tensor Ic.

4. The neuromorphic visual target tracking method based on image processing according to claim 3, characterized in that: The optical flow estimation algorithm is used to calculate the current inter-frame motion vector Vm by the target position change in two consecutive frames of image data in the neuromorphic joint feature map. Subsequently, the current inter-frame motion vector Vm and the joint input tensor Ic are dimensionlessly processed and correlated to fit, and the receptive field adjustment factor Gy is calculated. The specific calculation formula is as follows: ； Where, Represents the square root of the sum of the elements of the tensor, which is used to measure the overall numerical strength; e is a small constant to prevent the denominator from being zero; is the matrix inner product, which represents the sum of the element-wise products of Vm and Ic; The preset receptive field adjustment thresholds include G1 and G2, and G1>G2. Compare and evaluate with the receptive field adjustment factor Gy to adaptively adjust the receptive field size. The specific contents are as follows: If the receptive field adjustment factor Gy ≥ G1, it is determined to be a strong motion association, and the receptive field is expanded to capture a wider range of features; If the receptive field adjustment factor Gy≤G2, it is determined to be a weak motion association, and the receptive field is reduced to reduce redundant calculations; G2<receptive field adjustment factor Gy<G1: maintain the current receptive field and avoid frequent adjustments.

5. The neuromorphic visual target tracking method based on image processing according to claim 4, characterized in that: The target saliency entropy difference ΔH is calculated based on the local information entropy Hl and the global background entropy Hg. The specific calculation formula is as follows: ； The preset entropy threshold H is compared with the target significance entropy difference ΔH for evaluation. The specific evaluation content is as follows: When the entropy value threshold H ≥ the target saliency entropy difference ΔH, it is determined to be a target salient area, the current area is retained, and participates in tracking prediction; When the entropy value threshold H is less than the target significance entropy difference ΔH, it is judged as a noise area, and dynamic brightness enhancement is triggered at this time; it includes calculating the adaptive coefficient d based on the local information entropy Hl of the current area, and performing nonlinear enhancement processing on the pixel value I of the current area based on the adaptive coefficient d, while setting the upper limit of the pixel value.

6. The neuromorphic visual target tracking method based on image processing according to claim 5, characterized in that: The significance weight function w is calculated by the following formula: ； The preset significance weight threshold WA is compared with the significance weight function w to screen out redundant blocks triggered by background noise. The specific evaluation content is as follows: When the saliency weight function w ≥ the saliency weight threshold WA, it is determined to be a valid target candidate block, which is then retained and participates in subsequent tracking prediction; When the significance weight function w is less than the significance weight threshold WA, it is determined to be a background noise redundant block and is directly removed without participating in the tracking calculation.

7. The neuromorphic visual target tracking method based on image processing according to claim 6, characterized in that: Based on the filtered coding block, the similarity S is obtained by calculating the current coding block features and the historical target template features. The tracking confidence Ct is calculated using the similarity S. The specific calculation formula is: ； Where M represents the total number of candidate coding blocks that are screened and retained; Indicates the matching value of the i-th coding block, ranging from 0 to 1; The preset tracking confidence threshold C is compared with the tracking confidence Ct for evaluation. The specific contents are as follows; When the tracking confidence Ct ≥ the tracking confidence threshold C, it is determined to be a valid tracking state. At this time, the target state is updated according to the current coding block, including position and scale; When the tracking confidence Ct is less than the tracking confidence threshold C, it is determined to be a potential tracking failure and local re-detection is performed.

8. The neuromorphic visual target tracking method based on image processing according to claim 7, characterized in that: By constructing a memory trajectory tensor Tm, it is used to record the spatial location information, appearance features and confidence parameters of the target at multiple time steps; Use recurrent neural network to perform temporal consistency check on the memory trace tensor Tm; By analyzing the current tracking confidence Ct and the previous moment tracking confidence Ct -1 The difference between , obtain the confidence relative change rate ΔC; A confidence relative change threshold C is preset and compared with the confidence relative change rate ΔC for evaluation; when the confidence relative change threshold C exceeds the confidence relative change rate ΔC, the template adjustment mechanism is triggered; The template adjustment mechanism includes: progressively updating the current target template and fusing historical templates with current frame feature information; The connection weights of the recurrent neural network are adaptively fine-tuned based on the relative change rate of confidence ΔC, including dynamically adjusting the network connection weights using a gradient update method.

9. A neuromorphic visual target tracking system based on image processing, characterized by: The system is used to execute the neuromorphic visual target tracking method based on image processing according to any one of claims 1 to 8, comprising: The multimodal fusion perception module obtains multimodal visual input of the target area, including event stream information and frame image data, and performs spatiotemporal alignment to construct a joint input tensor Ic; The neural feature extraction module extracts neuromorphic features from the joint input tensor Ic, uses a multi-scale convolutional network to extract spatial features, and constructs a synaptic event-driven mechanism to extract time series change features; Calculate the current inter-frame motion vector Vm and adaptively adjust the receptive field size based on the joint input tensor Ic; The saliency judgment module extracts the local information entropy Hl and the global background entropy Hg in the target area based on spatial domain features, and calculates the target saliency entropy difference ΔH; and introduces dynamic brightness enhancement for local adjustment; The weighted screening and updating module calculates and filters candidate coding blocks based on the saliency weight function w based on the current inter-frame motion vector Vm and the target saliency entropy difference ΔH, removes redundant blocks triggered by background noise, and calculates the tracking confidence Ct to update the target state. The memory track optimization module constructs a memory track tensor Tm containing the historical tracking state, and performs time sequence consistency verification by the recursive neural network, combining the current tracking confidence Ct with the previous moment tracking confidence Ct -1 Perform dynamic template adjustment and neural weight fine-tuning.

Images