Visual processing method and system for multi-time-step attention migration

Abstract

The invention belongs to the technical field of artificial intelligence and video monitoring, and provides a visual processing method and system for multi-time-step attention migration, and the technical scheme is that a multi-channel fusion technology and an attention pruning technology are adopted, foreground and background information can be efficiently separated, a region of interest can be selected through an attention mechanism, and the visual processing efficiency of the multi-time-step attention migration is improved. And the detection precision and the response speed of the emergencies are further improved. The method can be applied to a dynamic vision sensor camera, realizes accurate identification of a rapidly changing dynamic scene by simulating a multi-time-step perception mechanism of human vision, and adapts to a complex vision environment. And in combination with edge computing equipment, real-time image processing can be completed under low delay, and the method is suitable for low-power-consumption and high-speed processing scenes.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and video monitoring technology, and in particular relates to a visual processing method and system for multi-time-step attention shift. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of artificial intelligence, computer vision, and dynamic visual sensor technologies, real-time video surveillance and detection systems are increasingly being used in a variety of scenarios. These systems can replace traditional manual monitoring to complete tasks such as security patrols, anomaly detection, and environmental monitoring. This not only improves work efficiency but also reduces labor costs. They also significantly enhance the safety and real-time performance of operations, especially in dangerous or high-frequency monitoring environments.

[0004] However, existing technologies still have limitations in real-time monitoring, accurate detection, and multi-task processing in dynamic environments. In particular, the response speed and detection accuracy of unexpected events (such as abnormal behavior) need to be improved. For example, when detecting abnormal behavior in dense crowds, traditional detection algorithms (such as YOLO and Faster R-CNN) rely on static frame analysis, making it difficult to track fast-moving or partially visible targets. In the event of a pickpocket or a sudden fall, the system may miss detection due to occlusion of the target or mistakenly identify overlapping areas of multiple people as a single entity. Furthermore, event cameras process redundant data in static scenes. Event cameras only respond to changes in pixel brightness. When monitoring static scenes (such as warehouses), slight light fluctuations (such as sunlight shifts) generate massive event streams, which consume computing resources and reduce the efficiency of detecting true anomalies (such as intruders). The same camera must simultaneously perform tasks such as face recognition, vehicle tracking, and behavioral analysis. Existing systems use a fixed resource allocation strategy, resulting in excessive computing power allocated to high-priority tasks (such as license plate recognition) and reduced performance for low-priority tasks (such as pedestrian fall detection). Summary of the Invention

[0005] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a visual processing method and system for multi-time-step attention shift, which can cope with real-time monitoring, accurate detection and multi-task processing in dynamic environments, and improve the response speed and detection accuracy when detecting emergencies.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A first aspect of the present invention provides a visual processing method for multi-time-step attention shift, comprising the following steps:

[0008] The acquired visual image is subjected to multi-layer convolution operations to generate a multi-channel feature map, and the multi-channel feature map is fused to obtain a fused feature map;

[0009] Binarize the multi-channel fusion feature map to obtain a binary feature map;

[0010] Combined with the attention scoring mechanism, each feature module of the binary feature map is scored to obtain the attention score of each module. Regions with attention scores less than the set score are pruned to generate an attention heat map.

[0011] Combined with the multi-time step analysis mechanism, in each time step, the key feature area of the current time step is identified based on the attention heat map; the confidence level of the prediction result of the current time step is judged to be higher than the set threshold. If so, the prediction result is output; otherwise, the image resolution is increased, the features corresponding to all resolution images are integrated, and the prediction result of the current time step is output;

[0012] The features of the current time step are fused with the features of the previous time step to obtain the cross-time step fusion features.

[0013] Furthermore, the feature map is binarized using LIF neurons to simulate the pulse emission behavior of biological neurons and generate a binary feature map.

[0014] Furthermore, the feature map is binarized using LIF neurons to simulate the pulse emission behavior of biological neurons, and the generated binary feature map is:

[0015] V[t]=H[t](1-S[t])+V reset S[t],

[0016]

[0017] Where V[t] and V[t-1] represent the membrane potential at time steps t and t-1, S[t] represents the pulse emission state at time step t, and V threshold Represents the threshold voltage, V reset Represents the reset voltage, H[t] represents the intermediate variable at time step t, τ is the time constant, and the membrane potential V[t] is gradually updated according to the input X[t]. When V[t] exceeds the threshold voltage V threshold When , a pulse is emitted, that is, S[t]=1, and a binary feature map S[t] is generated; X[t] is a multi-channel fusion feature map.

[0018] Furthermore, after outputting the results at each time step, the confidence evaluation is performed on the output results at each time step. The confidence evaluation formula is:

[0019] C t =σ(Wc AvgPool(F t ⊙A t )+b t 0,

[0020] A t =Softmax(W a ·F t +b a 0,

[0021] Among them, C t is the confidence score, the range is [0,1], σ is the Sigmoid function, which normalizes the score, W c is the convolution operation, AvgPool is the average pooling of the output, ⊙ is the principal element multiplication, F t is the feature map, A t represents the attention heat map at the current time step t; W a and b a Represents the learnable weight matrix and bias term.

[0022] Furthermore, the method of increasing the image resolution, fusing the features corresponding to images of all resolutions, and outputting the prediction result of the current time step includes:

[0023] Capturing global features of the image based on the first resolution image, outputting a first classification result based on the global features, and determining whether a confidence level of the initial classification result exceeds a preset threshold; if so, outputting a predicted first classification result; otherwise, increasing the resolution of the image to obtain a second resolution image;

[0024] Extracting first granularity detail features based on the second resolution image and the global features, outputting a second classification result based on the first granularity detail features, and determining whether a confidence level of the second classification result exceeds a preset threshold; if so, outputting a predicted second classification result; otherwise, increasing the resolution of the image to obtain a third resolution image;

[0025] Based on the third resolution image and the first granularity detail feature extraction, the second granularity detail feature is obtained, and based on the second granularity detail feature, a third classification result is output. It is determined whether the confidence of the third classification result exceeds a preset threshold. If so, the predicted third classification result is output. Otherwise, the resolution of the image is continued to be increased until the confidence meets the set conditions, and the final prediction result is output.

[0026] Furthermore, after outputting the results at each time step, the confidence evaluation is performed on the output results at each time step. The confidence evaluation formula is:

[0027] C t =σ(W c AvgPool(F t ⊙At )+b t ),

[0028] A t =Softmax(W a ·F t +b a ),

[0029] Among them, C t is the confidence score, the range is [0,1], σ is the Sigmoid function, which normalizes the score, W c is the convolution operation, AvgPool is the average pooling of the output, ⊙ is the principal element multiplication, F t is the feature map, A t represents the attention heat map at the current time step t; W a and b a Represents the learnable weight matrix and bias term.

[0030] Furthermore, it is determined whether the confidence level of the prediction result of the current time step is higher than the set threshold, specifically including:

[0031] If C t ≥θ c , that is, the confidence level is higher than the set threshold, and the prediction result is output;

[0032] If C t <θ c , that is, the confidence level is lower than the set threshold, increase the resolution of the image, and combine the output features F of the historical time step t-1 The dependency between the features of the current time step, correcting the allocation of the current attention area, and predicting the focus feature F of the current time step t , where C t is the confidence score, θ c is the threshold.

[0033] Furthermore, the loss function for the fusion of all time-step features is as follows:

[0034]

[0035] Among them, CE represents the final cross entropy loss, y t+1 and y t denote the output probability distributions of the t+1th and tth time steps respectively, and represents the probability value of the output probability distribution of the t+1th and tth time steps in the i-th category, γ t is a weight parameter used to control the influence of guidance loss between different time steps, y true is the true label.

[0036] Furthermore, the method also includes preprocessing the visual image, performing bilinear interpolation scaling on the visual image, and reducing its size to a target size.

[0037] A second aspect of the present invention provides a visual processing device for multi-time-step attention shift, comprising:

[0038] The feature extraction module is used to generate a multi-channel feature map by performing multi-layer convolution operations on the acquired visual image, and fuse the multi-channel feature maps to obtain a fused feature map;

[0039] The attention heatmap generation module is used to binarize the multi-channel fusion feature map to obtain a binary feature map. In combination with the attention scoring mechanism, each feature module of the binary feature map is scored to obtain the attention score of each module. Regions with attention scores less than the set score are pruned to generate an attention heatmap.

[0040] The multi-time-step attention offset module is used to combine the multi-time-step analysis mechanism. In each time step, the key feature area in the current time step is obtained based on the attention heat map identification. It determines whether the confidence of the prediction result of the current time step is higher than the set threshold. If so, the prediction result is output. Otherwise, the resolution of the image is increased, the features corresponding to all resolution images are fused, and the prediction result of the current time step is output. The features of the current time step are fused with the features of the previous time step to obtain the cross-time step fusion features.

[0041] A third aspect of the present invention provides a computer-readable storage medium.

[0042] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the visual processing method of multi-time-step attention shift as described above.

[0043] A fourth aspect of the present invention provides a computer device.

[0044] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the visual processing method for multi-time-step attention shift as described above are implemented.

[0045] A fifth aspect of the present invention provides a program product.

[0046] A program product, which is a computer program product, includes a computer program, and when the computer program is executed by a processor, it implements the steps in the visual processing method of multi-time-step attention shift as described above.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. The present invention proposes a visual processing method of multi-time-step attention shift. Through multi-channel fusion, adaptive attention mechanism and multi-time-step analysis mechanism, by simulating the multi-time-step perception mechanism of human vision, it can achieve accurate recognition of rapidly changing dynamic scenes and high real-time detection of key targets in dynamic scenes, effectively improving the system's response speed and detection accuracy in complex environments, and solving the limitations of visual processing in dynamic environments.

[0049] 2. This invention introduces an intelligent attention scoring and pruning mechanism. The system can flexibly adjust the pruning strategy according to the complexity of the image content, further reducing the computational burden while optimizing the detection accuracy of the attention area.

[0050] 3. This invention proposes a cross-time-step feature fusion and continuous tracking method, achieving dynamic feature fusion within multiple time steps, making target detection results more continuous and stable. This method is suitable for applications such as real-time video surveillance that require cross-time-step analysis. This method is suitable for dynamic visual sensors, real-time video surveillance systems, and other scenarios requiring high-precision detection and real-time processing. It is particularly well-suited for factories, warehouses, hospitals, and other places requiring intelligent security inspections, and has broad market value and application prospects.

[0051] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0053] Figure 1 This is a flow chart of a visual processing method for multi-time-step attention shifting provided by an embodiment of the present invention;

[0054] Figure 2 The embodiment of the present invention provides a multi-time-step based perception module architecture;

[0055] Figure 3 2 is a schematic diagram of the structure of a visual processing device for multi-time-step attention shifting provided by an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of the structure of a visual processing device for multi-time-step attention shift provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0058] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0059] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0060] Existing technologies still have certain limitations in real-time monitoring, accurate detection and multi-tasking processing in dynamic environments, especially in the response speed and detection accuracy when detecting emergencies (such as abnormal behavior), which need to be improved.

[0061] To address the above-mentioned issues, the present invention proposes a method and system for monitoring emergencies using multi-time-step perception in dynamic environments. This method utilizes multi-channel fusion and attention pruning techniques, which not only efficiently separate foreground and background information but also selects regions of interest through an attention mechanism, further improving the accuracy and speed of emergency detection. This method can be applied to dynamic visual sensor cameras. By simulating the multi-time-step perception mechanism of human vision, it enables accurate recognition of rapidly changing dynamic scenes and adapts to complex visual environments. Combined with edge computing devices, it can perform real-time image processing with low latency, making it suitable for low-power, high-speed processing scenarios.

[0062] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0063] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods. Figure 1 A flowchart of a visual processing method for multi-time-step attention shift provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0064] S101, acquiring a visual image and preprocessing the visual image;

[0065] In this embodiment, the visual image can be acquired in real time by an event-driven visual sensor camera or other device;

[0066] In this embodiment, the visual image I orig Perform bilinear interpolation scaling to reduce its size to the target size I small , in order to reduce the amount of subsequent calculations; this process not only helps to reduce the interference of irrelevant information, but also maintains the overall structure of the image, so that the features of the key areas are retained. The formula is expressed as:

[0067] I small (x,y)=I orig (r*x,r*y),

[0068] Where r is the reduction ratio (e.g., r=0.5 means reduction to half the size), and x and y are the pixels of the image.

[0069] In order to ensure that the visual features of the reduced image are clear, the reduced image is further preprocessed, including grayscale enhancement and edge preservation filtering, to maintain the clarity of the image.

[0070] S102, generating a multi-channel feature map by convolution of the preprocessed image, and fusing the multi-channel feature map to obtain a fused feature map;

[0071] In this embodiment, multi-layer convolution operation is performed on the preprocessed image, and the channel feature map generated by each convolution layer is represented as F c , the multi-channel feature is recorded as F;

[0072] The multi-channel feature map F c Fusion is performed to obtain the fused feature map F fusion , expressed as:

[0073]

[0074] Where C is the number of channels, w c The fusion weight assigned to each channel.

[0075] Assign appropriate fusion weights w to each channel c , adjust the relative importance between channels and calculate the multi-channel fusion map F fusion To achieve foreground and background separation, thereby enhancing the focus on important areas, and passing the fused feature map to the next step for refined feature extraction.

[0076] This invention utilizes multi-channel fusion technology to replace maximum pooling, dynamically separating foreground and background to enhance focus on key areas. Compared to traditional maximum pooling and strided convolution, multi-channel fusion can more precisely preserve the characteristic details of foreground objects while effectively suppressing background noise. This step also enhances the system's perception of small objects and boundaries, making foreground and background separation more accurate and suitable for feature extraction in complex environments.

[0077] S103, performing binarization processing on the multi-channel fusion feature map to obtain a binary feature map;

[0078] In this embodiment, LIF neurons are used to perform binarization processing on the feature map to simulate the pulse emission behavior of biological neurons and generate a binary feature map S.

[0079] The specific formula is:

[0080] V[t]=H[t](1-S[t])+V reset S[t],

[0081]

[0082] Where V[t] and V[t-1] represent the membrane potential at time steps t and t-1, S[t] represents the pulse emission state at time step t, and V threshold Represents the threshold voltage, V reset Represents the reset voltage, H[t] represents the intermediate variable at time step t, τ is the time constant, and the membrane potential V[t] is gradually updated according to the input X[t]. When V[t] exceeds the threshold voltage V threshold When , a pulse is emitted (ie, S[t] = 1), generating a binary feature map S[t]; X[t] is the fused feature map, i.e., F fusion .

[0083] The binary feature map S is then passed to the next step as the simplified important features for attention scoring.

[0084] S104: Combine the binary feature map with the attention scoring mechanism to score each feature module, obtain the attention score A of each module, prune the area with an attention score less than the set score, and generate the key feature map F. t ;

[0085] In this embodiment, a weight score is assigned to each feature module, and the weight score reflects the importance of each region to the model task.

[0086] Specifically, the attention mechanism, through methods such as self-attention or convolutional attention layers, enables the model to dynamically focus on areas that contribute most to the task outcome. This not only enhances the model's discriminative capabilities but also allows for flexible adjustments based on the diversity and complexity of image content, effectively processing diverse input data. This mechanism reduces redundant computation while enhancing the model's ability to perceive key features, ensuring full utilization of high-attention areas.

[0087] It's important to note that pruning regions with attention scores below a set score reduces computational effort and accelerates the model's inference process. This pruning operation is based on the attention score of each feature module, selectively removing regions that contribute less to the model's task. Specifically, the pruning algorithm not only considers the attention level of a region but also makes adaptive adjustments based on the complexity of the input image content. For simple images, the system may remove more redundant feature modules, while for complex images, it may retain more features to ensure accuracy.

[0088] S105: In combination with the multi-time-step analysis mechanism, in each time step, the key feature region in the current time step is identified based on the key feature map, and whether the confidence level of the prediction result of the current time step is higher than a set threshold is determined. If so, the prediction result is output; otherwise, the resolution of the image is increased, the features corresponding to the images of all resolutions are integrated, and the prediction result of the current time step is output. The specific steps include the following:

[0089] S501, in each time step, based on the key feature graph F t Generate attention heat map A t , according to the attention heat map A t Dynamically identify the key areas in the current frame or time step to obtain the focus features;

[0090] A t =Softmax(W a ·F t +b a ),

[0091] Among them, A t Represents the attention heat map at the current time step t, dimension and input features Figure 1 The value range is [0,1], indicating the importance of each region, W a and b a Represents a learnable weight matrix and bias term. This process combines the attention mechanism with the temporal feature extraction network, enabling the model to allocate different resources to the input image or data at each moment. Specifically, by calculating the attention score for each region, the system can evaluate in real time which features are most important to the target task at the current time step and allocate more computing resources to these high-attention regions.

[0092] Generated attention heatmap A t It can effectively visualize the degree of attention the model pays to different areas of the image. The heat map shows the attention weight of each area, expressing the model's focus in a color-coded manner, helping to analyze which areas have the greatest impact on task results. It is also used to guide the processing of subsequent time steps, retaining the feature information of high-scoring areas and passing it to the next multi-time-step focusing module.

[0093] S502: After outputting the result at each time step, a confidence evaluation is performed on the output result at each time step to ensure the reliability of the detection result. The confidence evaluation formula is:

[0094] C t =σ(W c AvgPool(F t ⊙A t )+b t ),

[0095] Among them, C t is the confidence score, the range is [0,1], σ is the Sigmoid function, which normalizes the score, W c It is a convolution operation, AvgPool performs average pooling on the output, and ⊙ is the principal element multiplication, which is used to focus on key area features.

[0096] S503, determining whether the confidence level of the prediction result of the current time step is higher than a set threshold;

[0097] If C t ≥θ c (such as θ c =0.8), that is, the confidence level is higher than the set threshold, and the prediction result is output;

[0098] If C t <θ c , that is, the confidence level is lower than the set threshold, increase the resolution of the image, and combine the output features F of the historical time step t-1 The dependency between the features of the current time step, correcting the allocation of the current attention area, and predicting the focus feature F of the current time step t ;

[0099] Specifically include:

[0100] For the confidence threshold determination of the above image, the embodiment takes the input image as an example, which is first adjusted to three different resolutions: 56x56, 112x112, and 224x224. This illustrates that by gradually increasing the resolution, the model can model the target from coarse-grained to fine-grained, gradually deepening the feature representation.

[0101] S5031. Capture global features of the image based on the first resolution image, output a first classification result based on the global features, and determine whether the confidence level of the initial classification result exceeds a preset threshold. If so, output the predicted first classification result; otherwise, go to S5032.

[0102] In the initial stage of image processing, the model starts with the lowest resolution (56x56). At this stage, the model extracts low-dimensional features of the image through convolutional layers and performs a preliminary classification of these features. The main goal of low-resolution processing is to quickly capture the global features in the image, aiming to quickly generate a rough classification result without consuming a lot of computing resources. At this stage, the model focuses on the macroscopic structure of the image, such as the overall outline and the positioning of major objects, to ensure that a preliminary judgment can be made as early as possible given limited computing resources. If the confidence of the preliminary classification exceeds the preset threshold, the model will directly output the classification result, avoiding further high-resolution calculations, thereby saving unnecessary computing resources. This strategy enables the model to quickly obtain results when faced with simple or clear images, while reducing computing time and overhead.

[0103] S5032: Increase the resolution of the image to obtain a second-resolution image, extract first-granularity detail features based on the second-resolution image and the global features, output a second classification result based on the first-granularity detail features, and determine whether the confidence level of the second classification result exceeds a preset threshold. If so, output a predicted second classification result; otherwise, proceed to S5033.

[0104] For samples that the low-resolution stage doesn't provide sufficient support for, the model sequentially inputs higher-resolution images (112x112 and 224x224) for finer-grained feature extraction. During this stage, the model processes richer image details, such as object texture, edge details, and color transitions, to obtain more precise classification information. As the resolution gradually increases, the model is able to identify and understand image details at a more microscopic level, ultimately providing more accurate classification results. In this process, the model gradually refines the feature representation, capturing more complex details and further improving classification accuracy. Furthermore, this gradual increase in resolution effectively optimizes computing resources. When the model only needs a rough understanding of the image, it can rely on low-resolution features for fast inference. As image complexity increases, it can compensate for the missing information by increasing the resolution, ensuring that the model operates effectively across diverse scenarios and resource constraints.

[0105] S5033: Increase the resolution of the image to obtain a third-resolution image; extract a second-granularity detail feature based on the third-resolution image and the first-granularity detail feature; output a third classification result based on the second-granularity detail feature; determine whether the confidence level of the third classification result exceeds a preset threshold; if so, output the predicted third classification result; otherwise, continue to increase the image resolution until the confidence level meets the preset condition, and output a final prediction result;

[0106] S504: Fuse the focus feature of the current time step with the output result of the previous time step to obtain the final cross-time step fusion feature:

[0107] F fusion =αF t +(1-α)F t-1 +βConv(I′ t ),

[0108] Among them, α and β are learnable fusion weights, and their initial values are dynamically adjusted by the attention score. t-1 is the feature map of the historical time step, F t is the focus feature of the current time step, Conv represents the convolution calculation of the input, I′ t is the input image.

[0109] This iterative fusion approach enhances the global understanding and tracking capabilities of the target. This process effectively addresses the problem of error accumulation caused by changes in target features over a short period of time, allowing the system to maintain accurate target detection in complex scenarios.

[0110] The loss function for fusion of features at all time steps is as follows:

[0111]

[0112] For the output {y1,y2,y3,...,y T}, CE represents the final cross entropy loss, γ t is a weight parameter used to control the influence of guidance loss between different time steps, y t+1 and y t denote the output probability distributions of the t+1th and tth time steps respectively, Represents the probability value of the distribution in the i-th category. By minimizing the KL divergence, y can be guided t Closer to y t+1 distribution, thus gradually approaching the final prediction result in the early time step, y true is the true label.

[0113] The time-step guidance mechanism based on knowledge distillation aims to accelerate the model inference process by gradually guiding the early time steps to approach the output distribution of the subsequent time steps, thereby quickly obtaining the features required for the final result. This mechanism introduces KL divergence into the loss function and uses the output of the subsequent time steps as a guidance signal, allowing the early time steps to gradually learn the key information of the subsequent time steps and optimize the inference process. Specifically, the model first quickly extracts global features at a low resolution and dynamically determines whether the features need to be further refined based on the confidence level during the inference process. When the preliminary results meet a certain confidence level, the inference can end early, avoiding unnecessary calculations and improving efficiency.

[0114] To enhance the learning capabilities of early time steps, the model designs a gradually decaying loss weighting strategy. As training progresses, the contribution of later time steps to the loss gradually increases, helping earlier time steps approach the final output more quickly. This gradually increasing guidance strategy not only accelerates the learning of early time steps, but also reduces dependence on later time steps, thereby improving the prediction quality of early time steps. During the inference phase, the model uses early stopping conditions to determine the similarity between the current time step and the final time step, thereby dynamically deciding whether to end inference early. This strategy reduces unnecessary consumption of computing resources and further improves overall inference efficiency. Therefore, the time-step guidance mechanism based on knowledge distillation not only accelerates model inference and improves accuracy, but also optimizes the use of computing resources, performing particularly well in real-time inference tasks.

[0115] The present invention can provide efficient and stable support for real-time monitoring and detection in complex environments, significantly improving the intelligence level and application value of the system. The present invention realizes the integration of visual processing system through a pulse neural network based on multi-time-step attention offset, and simulates the multi-time-step response mechanism of human vision in the computational model. The system integration includes a data acquisition module, a feature extraction module, an attention heat map generation module, a multi-time-step attention offset module and a final classification evaluation module. During the integration process, visual data is first collected in real time through event-driven visual sensor cameras and other devices, and the input image is sampled at multiple scales to form the initial input of the model. The neural network computing unit consists of a standard convolutional layer, a LIF neuron and an attention module, which gradually forms a multi-time-step perception of the input through preliminary feature extraction, neuron membrane potential update and dynamic attention calculation.

[0116] It should be added that this system can be embedded in intelligent visual processing terminals, such as smart cameras, autonomous vehicles, and industrial inspection systems, to achieve more accurate classification and target recognition by performing dynamic analysis and multi-time-step enhanced perception of visual information collected in real time.

[0117] The system can be embedded in intelligent visual processing terminals, such as smart cameras, autonomous vehicles, and industrial detection systems, and can achieve more accurate classification and target recognition by dynamically analyzing and multi-time-step enhanced perception of visual information collected in real time. For example, in an autonomous driving scenario, the visual processing system based on the present invention can identify pedestrians, vehicles, traffic signs and other targets in a complex environment in real time, and improve its adaptability and response speed to scene changes through multi-time-step response and attention mechanisms. In addition, the system is also suitable for fields such as autonomous navigation of drones, intelligent robots, and security monitoring. Through the neuron pruning mechanism and dynamic attention module, it can effectively reduce computational complexity and power consumption, and improve the real-time response performance of the equipment. In complex dynamic environments, such as drone navigation and robot path planning in narrow spaces, the system can effectively avoid obstacles and capture subtle changes in the target, and has extremely high robustness and flexibility.

[0118] In the above embodiment, the visual processing method of multi-time-step attention shift is described in detail. The present invention also provides a corresponding embodiment of the visual processing device of multi-time-step attention shift.

[0119] Figure 3 Schematic diagram of the structure of the image and text data matching detection device provided by the embodiment of the present invention. Figure 3 As shown, the device includes:

[0120] A data acquisition module 10 is used to acquire visual images;

[0121] The feature extraction module 11 is used to generate a multi-channel feature map by performing multi-layer convolution operations on the visual image, and fuse the multi-channel feature maps to obtain a fused feature map;

[0122] The attention heat map generation module 12 is used to binarize the multi-channel fusion feature map to obtain a binary feature map; combine the attention scoring mechanism to score each feature module of the binary feature map to obtain the attention score of each module, prune the area with an attention score less than the set score, and generate an attention heat map;

[0123] The multi-time-step attention shift module 13 is used to combine the multi-time-step analysis mechanism, and in each time step, obtain the key feature area in the current time step based on the attention heat map identification; determine whether the confidence of the prediction result of the current time step is higher than the set threshold, and if so, output the prediction result; otherwise, increase the resolution of the image, combine the output features of the historical time step and the dependency between the features of the current time step, correct the allocation of the current attention area, and fuse the features of the current time step with the features of the previous time step to obtain the cross-time step fusion feature.

[0124] Since the embodiments of the apparatus part correspond to the embodiments of the method part, the embodiments of the apparatus part refer to the description of the embodiments of the method part and are not repeated here. The same beneficial effects as the visual processing method of multi-time-step attention shift mentioned above are achieved.

[0125] Figure 4 This is a schematic diagram of the structure of a visual processing device for multi-time-step attention shifting provided by an embodiment of the present invention. This embodiment is based on the hardware perspective, such as Figure 4 As shown, the visual processing device for multi-time-step attention shift includes:

[0126] Memory 20, for storing computer programs;

[0127] The processor 21 is configured to implement the steps of the visual processing method for multi-time-step attention shift mentioned in the above embodiment when executing a computer program.

[0128] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), and a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU; the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0129] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the visual processing method of multi-time-step attention shift disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include but is not limited to the data involved in the above-mentioned visual processing method of multi-time-step attention shift, etc.

[0130] In some embodiments, the image-text data matching detection device may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0131] Those skilled in the art will understand that Figure 4 The structure shown in does not constitute a limitation of the visual processing device for multi-time-step attention shifting and may include more or fewer components than shown in the figure.

[0132] The visual processing device for multi-time-step attention shift provided by an embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the following method: a visual processing method for multi-time-step attention shift, with the same effect as above.

[0133] Finally, the present invention also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiment.

[0134] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and executes all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0135] The computer-readable storage medium provided by the present invention includes the above-mentioned visual processing method of multi-time-step attention shift, and the effect is the same as above.

[0136] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0137] The above is a detailed introduction to the visual processing method, device, equipment, medium and program product for multi-time-step attention shift provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A visual processing method for multi-time-step attention shift, characterized by: The steps include: The acquired visual image is subjected to multi-layer convolution operations to generate a multi-channel feature map, and the multi-channel feature map is fused to obtain a fused feature map; Binarize the multi-channel fusion feature map to obtain a binary feature map; Combined with the attention scoring mechanism, each feature module of the binary feature map is scored to obtain the attention score of each module. Regions with attention scores less than the set score are pruned to generate an attention heat map. Combined with the multi-time step analysis mechanism, in each time step, the key feature area of the current time step is identified based on the attention heat map; the confidence level of the prediction result of the current time step is judged to be higher than the set threshold. If so, the prediction result is output; otherwise, the image resolution is increased, the features corresponding to all resolution images are integrated, and the prediction result of the current time step is output; The features of the current time step are fused with the features of the previous time step to obtain the cross-time step fusion features.

2. The visual processing method for multi-time-step attention shift according to claim 1, wherein: LIF neurons are used to binarize the feature map, simulating the pulse emission behavior of biological neurons to generate a binary feature map.

3. The visual processing method for multi-time-step attention shift according to claim 2, wherein: Use LIF neurons to binarize the feature map, simulate the pulse emission behavior of biological neurons, and generate a binary feature map as follows: V[t]=H[t](1-S[t])+V reset S[t], Where V[t] and V[t-1] represent the membrane potential at time steps t and t-1, S[t] represents the pulse emission state at time step t, and V threshold Represents the threshold voltage, V reset Represents the reset voltage, H[t] represents the intermediate variable at time step t, τ is the time constant, and the membrane potential V[t] is gradually updated according to the input X[t]. When V[t] exceeds the threshold voltage V threshold When , a pulse is emitted, that is, S[t]=1, and a binary feature map S[t] is generated; X[t] is a multi-channel fusion feature map.

4. The visual processing method for multi-time-step attention shift according to claim 1, characterized in that: After outputting the results at each time step, the confidence evaluation is performed on the output results at each time step. The confidence evaluation formula is: C t =σ(W c ·AvgPool(F t ⊙A t )+b t ), A t =Softmax(W a ·F t +b a ), Among them, C t is the confidence score, the range is [0,1], σ is the Sigmoid function, which normalizes the score, W c is the convolution operation, AvgPool is the average pooling of the output, ⊙ is the principal element multiplication, F t is the feature map, A t represents the attention heat map at the current time step t; W a and b a Represents the learnable weight matrix and bias term.

5. The visual processing method for multi-time-step attention shift according to claim 1, characterized in that: The process of increasing the image resolution, fusing the features corresponding to images of all resolutions, and outputting the prediction result of the current time step includes: Capturing global features of the image based on the first resolution image, outputting a first classification result based on the global features, and determining whether a confidence level of the initial classification result exceeds a preset threshold; if so, outputting a predicted first classification result; otherwise, increasing the resolution of the image to obtain a second resolution image; Extracting first granularity detail features based on the second resolution image and the global features, outputting a second classification result based on the first granularity detail features, and determining whether a confidence level of the second classification result exceeds a preset threshold; if so, outputting a predicted second classification result; otherwise, increasing the resolution of the image to obtain a third resolution image; Based on the third resolution image and the first granularity detail feature extraction, the second granularity detail feature is obtained, and based on the second granularity detail feature, a third classification result is output. It is determined whether the confidence of the third classification result exceeds a preset threshold. If so, the predicted third classification result is output. Otherwise, the resolution of the image is continued to be increased until the confidence meets the set conditions, and the final prediction result is output.

6. The visual processing method for multi-time-step attention shift according to claim 1, characterized in that: Determine whether the confidence level of the prediction result at the current time step is higher than the set threshold, including: If C t ≥θ c , that is, the confidence level is higher than the set threshold, and the prediction result is output; If C t <θ c , that is, the confidence level is lower than the set threshold, increase the resolution of the image, and combine the output features F of the historical time step t-1 The dependency between the features of the current time step, correcting the allocation of the current attention area, and predicting the focus feature F of the current time step t , where C t is the confidence score, θ c is the threshold.

7. The visual processing method for multi-time-step attention shift according to claim 1, characterized in that: The loss function for fusion of features at all time steps is as follows: Among them, CE represents the final cross entropy loss, y t+1 and y t denote the output probability distributions of the t+1th and tth time steps respectively, and represents the probability value of the output probability distribution of the t+1th and tth time steps in the i-th category, γ t is a weight parameter used to control the influence of guidance loss between different time steps, y true is the true label.

8. The visual processing method for multi-time-step attention shift according to claim 1, characterized in that: The method further includes preprocessing the visual image and performing bilinear interpolation scaling on the visual image to reduce the size of the visual image to a target size.

9. A visual processing device with multi-time-step attention shift, characterized in that: include: The feature extraction module is used to generate a multi-channel feature map by performing multi-layer convolution operations on the acquired visual image, and fuse the multi-channel feature maps to obtain a fused feature map; The attention heat map generation module is used to binarize the multi-channel fusion feature map to obtain a binary feature map; Combined with the attention scoring mechanism, each feature module of the binary feature map is scored to obtain the attention score of each module. Regions with attention scores less than the set score are pruned to generate an attention heat map. The multi-time-step attention offset module is used to combine the multi-time-step analysis mechanism. In each time step, the key feature area in the current time step is obtained based on the attention heat map identification. It determines whether the confidence of the prediction result of the current time step is higher than the set threshold. If so, the prediction result is output. Otherwise, the resolution of the image is increased, the features corresponding to all resolution images are fused, and the prediction result of the current time step is output. The features of the current time step are fused with the features of the previous time step to obtain the cross-time step fusion features.

10. A visual processing device with multi-time-step attention shift, characterized in that The device comprises: memory for storing computer programs; A processor, configured to implement the steps of the multi-time-step attention shift visual processing method as described in any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the multi-time-step attention shift visual processing method according to any one of claims 1 to 8.

12. A program product, wherein the program product is a computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the visual processing method of multi-time-step attention shift are implemented as described in any one of claims 1 to 8.

Images