Automatic data annotation method of ISP image signal processing visual sensor

Abstract

The invention relates to an automatic data labeling method of an ISP image signal processing visual sensor. The method comprises the following steps: carrying out noise reduction and signal enhancement on an original image data stream acquired by an ISP image signal processing visual sensor to obtain a standardized image sequence; performing time sequence feature extraction to obtain a time sequence feature tensor; performing spatial feature extraction to obtain spatial-temporal feature representation; inputting a double-fold teacher-student network for semi-supervised training to obtain an initial image annotation model; performing quality evaluation based on the prediction probability distribution and the time sequence consistency to obtain a labeling result with a quality score; and storing a high-quality sample in the labeling result with the quality score into a sample buffer pool, carrying out sample screening based on entropy and distance measurement, and carrying out parameter updating on the initial image labeling model through an incremental learning strategy to obtain a target image labeling model. According to the invention, the reliability evaluation of the labeling result is realized, and the accuracy of automatic labeling is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of image signal processing, and in particular to an automatic data labeling method for an ISP image signal processing visual sensor. Background Art

[0002] The image data generated by ISP image signal processing visual sensors has shown explosive growth. These image data have the characteristics of high resolution, complex scene content and multiple signal representation forms, making it difficult for traditional manual data annotation methods to meet the needs of large-scale data processing, which seriously restricts the development and application of visual intelligence systems.

[0003] Most current image processing solutions rely on preset fixed processing parameters, which are difficult to adapt to complex and changing environments in actual application scenarios. When image acquisition conditions change, such as lighting conditions, scene complexity and other factors, fixed parameter processing methods often lead to a decline in annotation quality, affecting the accuracy of subsequent visual analysis. In addition, existing automated annotation technologies mainly rely on fully supervised learning methods, which require a large amount of manually annotated data to train the model. This method is not only time-consuming and labor-intensive, but also difficult to handle continuously changing actual scenes. Especially in the case of unstable image quality and changing scenes, the performance of the annotation model will be significantly reduced, and there is a lack of effective adaptive update mechanism. Summary of the invention

[0004] The main purpose of the present invention is to provide an automatic data labeling method for an ISP image signal processing visual sensor. The present invention realizes the reliability evaluation of the labeling results and ensures the accuracy of automatic labeling.

[0005] To achieve the above object, the present invention provides an automatic data labeling method for an ISP image signal processing visual sensor, comprising the following steps: The original image data stream collected by the ISP image signal processing visual sensor is subjected to noise reduction and signal enhancement to obtain a standardized image sequence; Inputting the standardized image sequence into a temporal attention encoder to extract temporal features, thereby obtaining a temporal feature tensor; Inputting the temporal feature tensor into an adaptive convolutional layer and a dynamic feature converter to extract spatial features to obtain spatiotemporal feature representation; Inputting the spatiotemporal feature representation into a bi-fold teacher-student network for semi-supervised training, and optimizing parameters through cross entropy loss and consistency loss to obtain an initial image annotation model; Inputting the target image data stream collected in real time into the initial image annotation model for feature extraction and annotation prediction, and performing quality assessment based on the predicted probability distribution and temporal consistency to obtain an annotation result with a quality score; The high-quality samples in the annotation results with quality scores are stored in a sample buffer pool, the samples are screened based on entropy and distance metrics, and the parameters of the initial image annotation model are updated through an incremental learning strategy to obtain a target image annotation model.

[0006] The present invention also provides an automatic data labeling device for an ISP image signal processing visual sensor, comprising: A noise reduction module is used to reduce noise and enhance signals on the original image data stream collected by the ISP image signal processing visual sensor to obtain a standardized image sequence; A first extraction module, configured to input the standardized image sequence into a temporal attention encoder to extract temporal features and obtain a temporal feature tensor; A second extraction module is used to input the temporal feature tensor into an adaptive convolution layer and a dynamic feature converter to extract spatial features and obtain a spatiotemporal feature representation; A training module, used for inputting the spatiotemporal feature representation into a bi-fold teacher-student network for semi-supervised training, and performing parameter optimization through a cross entropy loss and a consistency loss to obtain an initial image annotation model; An evaluation module is used to input the target image data stream collected in real time into the initial image annotation model for feature extraction and annotation prediction, and to perform quality evaluation based on the predicted probability distribution and temporal consistency to obtain an annotation result with a quality score; The updating module is used to store the high-quality samples in the annotation results with quality scores into a sample buffer pool, screen the samples based on entropy and distance metrics, and update the parameters of the initial image annotation model through an incremental learning strategy to obtain a target image annotation model.

[0007] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0008] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0009] In summary, the technical solution provided by the present invention effectively improves the quality of images collected by the ISP sensor by designing a multi-level image preprocessing process, including time division, normalization mapping, adaptive threshold filtering and other steps. The feature extraction method combining the temporal attention encoder and the adaptive convolution layer is adopted to make full use of the temporal information and spatial features of the image, and enhance the model's feature expression ability for complex scenes; the introduction of a double-fold teacher-student network for semi-supervised training significantly reduces the dependence on manually labeled data, and at the same time improves the generalization ability of the model through the joint optimization of cross entropy loss and consistency loss; the quality evaluation mechanism based on the predicted probability distribution and temporal consistency realizes the reliability evaluation of the labeling results and ensures the accuracy of automatic labeling; by establishing a sample buffer pool and an incremental learning strategy, the continuous optimization and updating of the model is realized, so that the system can adapt to the ever-changing image acquisition environment; the sample screening mechanism based on entropy and distance metrics ensures the sample quality used for model updating and avoids the negative impact of low-quality samples on model performance; the present invention significantly improves data processing efficiency and reduces labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a schematic diagram of the steps of an automatic data labeling method of an ISP image signal processing visual sensor in one embodiment of the present invention; Figure 2 It is a structural block diagram of an automatic data labeling device of an ISP image signal processing visual sensor in one embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0011] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0013] Reference Figure 1 This embodiment provides an automatic data labeling method for an ISP image signal processing visual sensor, comprising the following steps: S1, performs noise reduction and signal enhancement on the original image data stream collected by the ISP image signal processing visual sensor to obtain a standardized image sequence; Among them, the original image data stream is collected by the ISP image signal processing visual sensor, and the data stream is divided into time series, and the continuous image data is grouped according to a certain time interval to form an image sequence block with a fixed number of frames or time windows. The purpose of time series division is to capture the dynamic change characteristics in the image data stream and provide an operable time and space segmentation unit for subsequent processing. The normalization mapping of pixel values ​​is performed on each frame of the image in the image sequence block, and each pixel value in the original image is adjusted to a standardized range (such as [0,1] or [0,255]), thereby reducing the deviation caused by the difference in the dynamic range of the sensor or the change in lighting conditions. The normalized image sequence forms a normalized image sequence, on which each frame of the image is processed by regional block. Each frame of the image is divided into multiple small sub-regions, which are grid units of fixed size. The role of regional block is to enhance the local feature analysis capability, provide higher spatial resolution support for subsequent signal processing, and refine the description of image content. The local signal intensity of each sub-region in the image sub-region sequence is calculated. By analyzing the statistical distribution of pixel values ​​in each sub-region, the signal intensity characteristics of the sub-region are determined, and the adaptive threshold sequence is calculated based on these characteristics. The adaptive threshold generation method takes into account the spatial heterogeneity of the image content and adjusts the processing strategy according to the noise characteristics in different sub-regions. Based on the adaptive threshold sequence, the image sub-region sequence is filtered, and the noise component is suppressed by the filter to obtain a denoised image sequence. The denoised image sequence is subjected to histogram statistics, and the histogram feature sequence is calculated by analyzing the grayscale distribution of each frame of the image. The histogram reflects the distribution law of image pixel values ​​and is an important indicator for measuring image contrast and brightness distribution. Based on the histogram feature sequence, the gamma correction coefficient is calculated to adjust the contrast and brightness of the image. Gamma correction enhances the dark details and bright levels in the image through nonlinear transformation, making the image look more natural and clear. After applying gamma correction to the denoised image sequence, a contrast-enhanced image is obtained. Edge detection and detail enhancement processing are performed on the contrast-enhanced image. Edge detection extracts object contours and edge information by detecting areas in the image where the grayscale value changes dramatically, while detail enhancement improves the clarity and texture expression of the image by emphasizing high-frequency information. The combination of the two can significantly enhance the structural features of the image, making the important details in the image more prominent. After the above multi-level processing, a standardized image sequence is finally obtained.

[0014] S2, input the standardized image sequence into the temporal attention encoder to extract temporal features and obtain the temporal feature tensor; Specifically, the standardized image sequence is input into the first layer of the temporal attention network of the temporal attention encoder for preliminary processing. The first layer of the temporal attention network consists of a first self-attention module and a first feedforward neural network, which work together to efficiently capture the time-dependent characteristics of the input data. At this stage, the standardized image sequence is calculated by the first self-attention module. The attention mechanism dynamically pays attention to the frames with higher correlation in the input sequence by weighted summation, thereby strengthening the key time series signals during feature extraction and suppressing redundant or irrelevant information. After this calculation process, the first layer of attention weighted features are obtained, which comprehensively consider the global correlation between each time frame in the input sequence. The first layer of attention weighted features are input into the first feedforward neural network for nonlinear feature transformation. The first feedforward neural network maps the input features through a series of fully connected layers and activation functions, thereby enhancing its expressive power and highlighting the implicit temporal pattern. After the feature transformation is completed, in order to avoid the degradation problem in the feature learning process, the residual connection mechanism is introduced to directly add the transformed features to the original input features to obtain the first layer of temporal features. The first layer of temporal features are input into the second layer of the temporal attention network of the temporal attention encoder for processing. The second-layer temporal attention network has a similar structure to the first-layer network, and also includes a second self-attention module and a second feedforward neural network. The second self-attention module performs a deeper correlation analysis on the first-layer temporal features by repeating the attention calculation process to extract richer temporal dependency patterns. The second feedforward neural network strengthens the feature expression ability at this stage, maps the high-order information in the multi-dimensional feature space into a more meaningful expression, and obtains the second-layer temporal features. The second-layer temporal features are input into the position encoding module for position encoding processing. The position information corresponding to each time frame is calculated by sine and cosine functions to generate a position embedding vector. The position embedding vector is an additional feature dimension that can help the network retain the relative position information of the time frame when processing disordered data and enhance the network's temporal modeling ability. After the position embedding vector is combined with the second-layer temporal features, temporal features with position information are obtained. The temporal features with position information are input into the third-layer temporal attention network of the temporal attention encoder for multi-head attention calculation. The multi-head attention mechanism captures the complex interaction patterns between different time frames at a higher level by decomposing the input features into multiple subspaces and performing attention calculations on them separately, and then splicing and integrating the results. The multi-head mechanism can process multiple correlation patterns in the feature space at the same time, making the final time series features more expressive. After multi-head attention calculation, multi-head attention features are generated. The multi-head attention features are input to the layer normalization module for normalization. Layer normalization adjusts the mean and standard deviation of the features to eliminate the distribution differences of the input features, thereby accelerating the convergence of the model and improving the robustness of the model in the feature processing process.The normalized features are input into the fully connected layer, and the dimension of the features is compressed through dimensionality reduction. The compressed features after dimensionality reduction are fused with the normalized multi-head attention features. Multiple feature expressions are combined through feature concatenation, weighted summation or other nonlinear combination methods to generate a more complete and descriptive time series feature representation and obtain a time series feature tensor.

[0015] S3, input the temporal feature tensor into the adaptive convolution layer and the dynamic feature converter to extract spatial features and obtain spatiotemporal feature representation; It should be noted that the temporal feature tensor is input into the first adaptive convolution layer for convolution operation. The first adaptive convolution layer adopts a 3×3 convolution kernel configuration, and its convolution step is set to 1. The padding method is SAME, so as to ensure that the spatial size of the input and output are consistent. The convolution kernel design can extract fine-grained features of local areas, and nonlinearly transform the output through the ReLU activation function to generate a first-scale feature map to capture the local detail information in the temporal feature tensor. At the same time, the same temporal feature tensor is input into the second adaptive convolution layer, which adopts a 5×5 convolution kernel, and its step is also set to 1, and the padding method is also SAME. Compared with the first convolution layer, the larger convolution kernel can cover a larger receptive field, thereby extracting more mid-scale spatial structure information in the input features. After the nonlinear transformation of the ReLU activation function, the output is the second-scale feature map, which contains the mid-level semantic features in the temporal feature tensor. In order to enrich the feature expression capability, the temporal feature tensor is input into the third adaptive convolution layer, which uses a 7×7 convolution kernel with a step size of 1 and a padding method of SAME. Since a larger convolution kernel can capture a wider range of contextual information, the features extracted in this process tend to reflect global spatial information. After the nonlinear mapping of the ReLU activation function, the third-scale feature map is generated, which can provide key global characteristics for subsequent feature fusion. The dynamic feature converter calculates the feature similarity of the first-scale feature map, the second-scale feature map, and the third-scale feature map to obtain the feature weight coefficient. The dynamic feature converter calculates the similarity between feature maps, identifies the most important information area in each feature map, and assigns appropriate weights to each feature to ensure that the subsequent fusion process can fully utilize the advantages of multi-scale features. According to these weight coefficients, the first-scale feature map, the second-scale feature map, and the third-scale feature map are fused according to the pyramid structure to generate a pyramid feature map. Channel attention is calculated on the pyramid feature map to optimize the feature expression. In this process, the importance of each channel is calculated to generate a channel weight vector, thereby capturing the difference in the contribution of each channel to the overall feature in the pyramid feature map. The channel weight vector is multiplied element by element with each channel of the pyramid feature map. The selectivity of feature expression is enhanced by emphasizing high-weight channels and suppressing low-weight channels to obtain channel-weighted features. In order to reduce feature dimensions and improve computational efficiency, channel weighted features are compressed. Channel compression removes redundant information through dimensionality reduction technology while retaining the core information that is important for feature expression to generate compressed spatial features. On this basis, the compressed spatial features are matrix multiplied with the feature weight coefficients to complete the generation of the final spatiotemporal feature representation.

[0016] S4, input the spatiotemporal feature representation into the bi-fold teacher-student network for semi-supervised training, and optimize the parameters through cross entropy loss and consistency loss to obtain the initial image annotation model; Specifically, the spatiotemporal feature representation is divided into labeled training data and unlabeled training data to construct a training data set, where the labeled training data is samples with manual labels, and the unlabeled training data is samples without manual labels. The labeled data is used to guide the learning of unlabeled data, giving full play to the advantages of semi-supervised learning. The labeled training data in the training data set is input into the teacher network in the bi-fold teacher-student network. The teacher network consists of a feature extraction layer, an attention layer, and a classification layer. Among them, the feature extraction layer is the core part of the teacher network, which contains four convolution blocks. Each convolution block consists of two 3×3 convolution layers and a maximum pooling layer. This structure can gradually extract deep features in the input data, while reducing the feature dimension through pooling operations, retaining key information and reducing computational complexity. After being processed by the feature extraction layer of the teacher network, the generated features will be passed to the attention layer for processing. The attention layer focuses on the important parts of the input by dynamically adjusting the feature weights, and passes the optimized features to the classification layer. The classification layer is responsible for the final classification of the features and generates teacher features as output. After completing the processing of the labeled training data, the unlabeled training data is input into the teacher network. For unlabeled data, the teacher network generates a category probability distribution through feature extraction, attention allocation and classification processing. In order to generate pseudo labels for unlabeled data, according to the output category probability distribution of the teacher network, the category with a predicted probability value higher than the set threshold is used as a pseudo label to obtain pseudo labeled data. The pseudo labeled data is combined with the labeled training data to form a new training data set, and the data set is input into the student network in the two-fold teacher-student network. The structure of the student network is exactly the same as that of the teacher network, including a feature extraction layer, an attention layer and a classification layer, but its parameters are initialized independently, and the label information provided by the teacher network is relearned through the student network. In this process, the student network learns the feature distribution of unlabeled data through pseudo labeled data, and learns the distribution of true labels through labeled training data, thereby improving the overall generalization ability. In order to ensure the feature consistency between the teacher network and the student network, the teacher features and the student features are aligned, and the Euclidean distance between the normalized feature vectors is calculated as the consistency loss. The purpose of the consistency loss is to narrow the feature distribution difference between the teacher network and the student network, so that the student network can better inherit the feature expression ability of the teacher network. At the same time, in order to improve the classification performance, the cross entropy loss is calculated on the pseudo-labeled data and the labeled training data respectively to evaluate the deviation between the prediction result and the real label or pseudo-label. On this basis, the consistency loss and the classification cross entropy loss are combined to construct a joint loss function. The joint loss function balances the contribution of consistency loss and cross entropy loss through a weighted coefficient α, where the weight of consistency loss is α and the weight of cross entropy loss is 1-α. By minimizing the joint loss function, the parameters of the teacher network and the student network are optimized.During the optimization process, the teacher network gradually improves the reliability of pseudo labels, while the student network continuously learns better feature expression and classification capabilities from the teacher network. After the optimization is completed, the teacher network in the optimized two-fold teacher-student network is used as the final initial image annotation model. This model has strong annotation capabilities and also has the potential to process unlabeled data.

[0017] S5, inputting the real-time collected target image data stream into the initial image annotation model for feature extraction and annotation prediction, and performing quality assessment based on the predicted probability distribution and temporal consistency to obtain an annotation result with a quality score; The target image data stream collected in real time is preliminarily segmented, and the continuously input image data stream is divided according to a fixed time window to decompose it into image sequence blocks with a fixed number of frames. For each image sequence, a preprocessing operation is performed to remove noise and redundant information, and a standardization process is performed to generate input data suitable for feature extraction. Then, the preprocessed image sequence is subjected to deep feature extraction through the feature extraction layer to obtain the target image feature sequence. The target image feature sequence is input into the feature extraction layer of the initial image annotation model. The feature extraction layer consists of four convolution blocks, each of which contains two 3×3 convolution layers and a maximum pooling layer. Each convolution block extracts local spatial information in the feature sequence through stepwise convolution and pooling operations, and gradually improves the semantic level of the feature while compressing the spatial dimension of the feature. After the mapping process of the four convolution blocks, the output deep features of the target image contain high-order feature representations of the input data. The deep features of the target image are input into the attention layer of the initial image annotation model for processing. In the attention layer, through spatial attention calculation, the model can dynamically assign weights to features, so that the focus on specific areas is enhanced while suppressing the influence of background or irrelevant areas. The attention mechanism effectively improves the representation ability of the target object and finally generates attention-weighted features. The attention-weighted features are input into the classification layer of the initial image annotation model for category prediction. In the classification layer, the features are linearly transformed through a series of fully connected operations, and the probability distribution of each category is calculated using the Softmax function to generate a prediction probability vector. The prediction probability vector reflects the confidence that the target image belongs to each category, where the maximum probability value corresponds to the main prediction result of the model. In order to screen out reliable annotation results, the maximum probability value in the prediction probability vector is compared with the preset confidence threshold. Only the prediction results above the threshold are considered as high-confidence annotations to obtain preliminary annotation results. The preliminary annotation results are evaluated for temporal consistency. The temporal correlation of the annotation results is measured by calculating the cosine similarity of the annotation results between adjacent frames. Cosine similarity is an effective method that can quantify the directional consistency between the feature vectors of adjacent frames and reflect the temporal coherence of the annotation results. Through this calculation, a temporal consistency score is assigned to each annotation result. In order to comprehensively consider the static confidence and dynamic consistency of the annotation, the maximum value of the predicted probability vector is taken as the confidence score, and is weighted and combined with the temporal consistency score to form a comprehensive quality score. During the combination process, weights β and 1-β are assigned to the confidence score and the temporal consistency score respectively to achieve a balance between the two. The preliminary annotation results are associated with the comprehensive quality score to generate annotation results with quality scores.

[0018] S6, stores high-quality samples in the annotation results with quality scores into the sample buffer pool, screens samples based on entropy and distance metrics, and updates the parameters of the initial image annotation model through an incremental learning strategy to obtain the target image annotation model.

[0019] Specifically, the annotation results with quality scores are sorted in descending order according to the comprehensive quality score, and samples with higher scores are given priority. After the sorting is completed, the dynamic threshold selection method is used to determine the screening threshold, and the threshold is dynamically adjusted according to the sample distribution to ensure that the selected high-quality samples have high representativeness and stability. Samples above the screening threshold are selected as high-quality samples and stored in the sample buffer pool. After entering the sample screening stage, the predicted probability distribution of each sample in the sample buffer pool is normalized to eliminate the influence of category distribution differences. Based on the normalized probability distribution, the entropy value of each category is calculated and summed using the Shannon entropy formula to obtain a standardized sample entropy value sequence. The lower the sample entropy value, the more concentrated the predicted distribution and the higher the confidence. Conversely, it means that the category uncertainty of the sample is higher. This information is used to evaluate the importance of the sample. At the same time, the initial image annotation model is used to extract the deep features corresponding to each sample in the sample buffer pool, and the cosine distance metric is used to calculate the similarity between sample pairs to generate a sample distance matrix. The sample distance is transformed by the Gaussian kernel function, and the distance between samples is expressed as a similarity distribution, which effectively reflects the clustering of samples in the feature space. The sample importance score is constructed based on the sequence of standardized sample entropy values ​​and sample similarity distribution. The entropy value measures the prediction certainty of the sample, while the similarity distribution evaluates the diversity of the sample. By combining the two, the weight of each sample is calculated to form a sample weight vector. The samples in the sample buffer pool are stratified and sampled according to the sample weight vector to ensure that the incremental training set is balanced in terms of sample category distribution and feature diversity. In order to improve the effectiveness of the samples, data enhancement is performed on the incremental training set, such as applying operations such as rotation, cropping, flipping or color transformation to generate enhanced training samples. When the enhanced training samples are input into the initial image annotation model for incremental training, the model parameters are fine-tuned by knowledge distillation. Knowledge distillation generates soft labels through the teacher network. These soft labels can provide rich category distribution information, thereby guiding the student network to learn sample features more finely. This training method can not only improve the adaptability of the student network to difficult-to-classify samples, but also ensure the stability of the model in incremental training. After fine-tuning, the generated updated model parameters need to be evaluated on the validation set to comprehensively consider their performance in terms of classification accuracy, consistency and generalization to obtain a comprehensive performance score. The optimal model parameters are selected according to the comprehensive performance score and updated to the initial image annotation model to complete the model optimization and obtain the target image annotation model.

[0020] In one example, noise reduction and signal enhancement are performed on the raw image data stream collected by the ISP image signal processing visual sensor to obtain a standardized image sequence, including: The ISP image signal processing visual sensor collects the original image data stream and divides the original image data stream into time series to obtain image sequence blocks; Normalizing and mapping the pixel values ​​of each frame of the image in the image sequence block to obtain a normalized image sequence, and performing regional block division on the normalized image sequence to divide each frame of the image into multiple sub-regions to obtain an image sub-region sequence; Calculating the local signal strength of each sub-region in the image sub-region sequence to obtain an adaptive threshold sequence, and filtering the image sub-region sequence based on the adaptive threshold sequence to obtain a denoised image sequence; Perform histogram statistics on the denoised image sequence, calculate the grayscale distribution of each frame of the image, obtain the histogram feature sequence, calculate the gamma correction coefficient based on the histogram feature sequence, perform gamma correction processing on the denoised image sequence, and obtain a contrast enhanced image; Edge detection and detail enhancement are performed on the contrast-enhanced image to obtain a standardized image sequence.

[0021] In this example, the ISP image signal processing visual sensor collects the original image data stream in real time, and the data stream needs to be divided into time series. Assume that the original data stream is a continuous frame image sequence ,in Indicates time The image frame at the moment, is the total number of frames. Divide into segments, that is, each segment contains Frames, forming image sequence blocks ,in It is For each frame of image in each image sequence block Normalize the pixel values ​​and adjust them to a standardized range, thereby reducing intensity fluctuations caused by changes in acquisition equipment or ambient light. Assume that the pixel value range is , the normalized formula is: ; in, is the normalized image frame, and Represent the minimum and maximum pixel values ​​respectively. After normalization, a normalized image sequence is obtained. Based on the normalized image sequence, each frame of image The image is further divided into multiple sub-regions. The image is divided into fixed-size grids, assuming that the image resolution is , the size of each sub-region is , then each frame image is divided into sub-regions. The image sub-region sequence after division is expressed as ,in Indicates Frame No. For each sub-region in the image sub-region sequence Calculate the local signal strength. The local signal strength is measured by calculating the standard deviation of the pixel values ​​in each sub-region, and the formula is as follows: ; in Indicates sub-area The signal strength, is the number of pixels in the subregion, is the average value of the sub-region, is the pixel in the sub-region Based on the calculated signal strength, an adaptive threshold sequence is dynamically generated for each sub-region. , set to ,in is a control parameter. Then a filtering operation is applied to each sub-region, using a median filter or a Gaussian filter, to suppress noise and smooth the regional content to obtain a denoised image sequence. A histogram statistical analysis is performed on the denoised image sequence to calculate the grayscale distribution of each frame. Grayscale histogram Indicates that the gray value in the image is The number of pixels is counted, and the gray value distribution of each frame of the image is counted to generate a histogram feature sequence. Based on these histogram features, the gamma correction coefficient of each frame of the image is calculated. Assume that the grayscale mean is , the target grayscale mean is , then the gamma correction coefficient is expressed as: ; Apply the gamma correction formula Perform nonlinear adjustment on the denoised image to obtain a contrast-enhanced image. Perform edge detection and detail enhancement on the contrast-enhanced image. Use edge detection algorithms (such as the Canny algorithm) to detect areas in the image where the grayscale value changes dramatically. Combined with detail enhancement methods (such as Laplace filtering or non-local mean enhancement), enhance high-frequency information to obtain the final standardized image sequence.

[0022] In one example, the standardized image sequence is input into the temporal attention encoder for temporal feature extraction to obtain a temporal feature tensor, including: Input the standardized image sequence into the first layer of the temporal attention network of the temporal attention encoder, where the first layer of the temporal attention network includes a first self-attention module and a first feedforward neural network; The first self-attention module performs attention calculation on the standardized image sequence to obtain the first layer of attention weighted features, and the first layer of attention weighted features are input into the first feedforward neural network for feature transformation, and are residually connected with the original features to obtain the first layer of temporal features; The first layer of temporal features is input into the second layer of temporal attention network of the temporal attention encoder, and features are extracted through the second self-attention module and the second feedforward neural network in the second layer of temporal attention network to obtain the second layer of temporal features; Perform position encoding on the second-layer temporal features, use sine and cosine functions to calculate the position embedding vector, and obtain temporal features with position information; Input the temporal features with position information into the third layer of the temporal attention network of the temporal attention encoder to perform multi-head attention calculation to obtain multi-head attention features; The multi-head attention features are layer-normalized and feature dimension reduction is performed through a fully connected layer to obtain compressed features. The compressed features are fused with the multi-head attention features to obtain a time series feature tensor.

[0023] In this example, the normalized image sequence is represented as a 3D tensor ,in Indicates the number of time frames, and are the height and width of the image respectively. In order to adapt the input to the computational requirements of the network, Reshape ,in is the dimension of each frame after flattening. Input the first layer of the temporal attention network of the temporal attention encoder. The first layer of the temporal attention network consists of the first self-attention module and the first feedforward neural network. The first self-attention module calculates the dependencies between the time frames in the input sequence through the attention mechanism. For the input sequence , first generate the query through linear transformation ( ),key( ) and value ( )Three matrices: ; in is the learned weight matrix, is the dimension after projection. By calculating The dot product with K obtains the attention weight: ; in Indicates the relevance of each time frame to other frames. The attention weight Applied to the value matrix : ; in is the first layer of attention weighted features, capturing the global temporal relationship of the input sequence. Input the first feedforward neural network, which consists of two fully connected layers, and the activation function is ReLU, the formula is: ; in is the weight matrix, is the bias, is the hidden layer dimension. In order to ensure gradient flow and feature stability, the input feature With the transformed features Make a residual connection: ; in is the first layer of temporal features, and LayerNorm is the layer normalization operation. The input is sent to the second layer of temporal attention network, which has a similar structure to the first layer, including the second self-attention module and the second feedforward neural network. Through a similar calculation process, the second layer of temporal features is obtained. In order to enhance the position information of the time frame, Perform position encoding and use sine and cosine functions to calculate the position embedding vector. Assume that the time frame The position code is , then it is expressed as: ; in is the index of the feature dimension, is the feature dimension size. Add position encoding to the second layer of temporal features: ; The time series features with position information Input to the third layer of temporal attention network, which uses a multi-head attention mechanism for calculation. Multi-head attention calculates multiple independent attention heads in parallel, concatenates the output of each head and projects it back to the original dimension: ; The calculation of each head is softmax is the output projection matrix. After multi-head attention calculation, we get the multi-head attention feature . For multi-head attention features Perform layer normalization and perform feature dimensionality reduction through the fully connected layer. The features after dimensionality reduction are: ; in is the dimensionality reduction weight, is the bias. With multi-head attention features Perform feature fusion, for example by element-wise weighted summation: ; in is the fusion coefficient. Finally, the time series feature tensor is obtained , used for subsequent image annotation tasks.

[0024] In one example, the temporal feature tensor is input into the adaptive convolution layer and the dynamic feature converter for spatial feature extraction to obtain spatiotemporal feature representation, including: The time series feature tensor is input into the first adaptive convolution layer for convolution operation. The first adaptive convolution layer uses a 3×3 convolution kernel, a convolution step size of 1, and a padding method of SAME. It is nonlinearly transformed through the ReLU activation function to obtain the first scale feature map. The time series feature tensor is input into the second adaptive convolution layer for convolution operation. The second adaptive convolution layer uses a 5×5 convolution kernel, a convolution step size of 1, and a padding method of SAME. It is nonlinearly transformed through the ReLU activation function to obtain the second scale feature map. The time series feature tensor is input into the third adaptive convolution layer for convolution operation. The third adaptive convolution layer adopts a 7×7 convolution kernel, a convolution step size of 1, a padding method of SAME, and a nonlinear transformation through a ReLU activation function to obtain a third-scale feature map. The first scale feature map, the second scale feature map and the third scale feature map are subjected to feature similarity calculation by a dynamic feature converter to obtain a feature weight coefficient, and a feature pyramid is performed based on the first scale feature map, the second scale feature map and the third scale feature map to obtain a pyramid feature map; Perform channel attention calculation on the pyramid feature map to obtain the channel weight vector, and multiply the channel weight vector by each channel of the pyramid feature map element by element to obtain the channel weighted feature; The channel weighted features are channel compressed to obtain compressed spatial features, and the compressed spatial features are matrix multiplied with the feature weight coefficients to obtain the spatiotemporal feature representation.

[0025] In this example, the input time series feature tensor is represented as ,in is the number of time frames, and are the height and width of the image, is the number of feature channels. Input the first adaptive convolution layer for convolution operation, which uses The convolution kernel is 3, the convolution step is 1, and the padding mode is SAME, which means that the spatial resolution of the output after the convolution operation is the same as the input. The convolution calculation formula is: ; in is the first scale feature map, It is The weight of the convolution kernel. The output is transformed nonlinearly through the ReLU activation function to enhance the nonlinear expression ability of the feature map. The formula is: ; Likewise, Enter the second adaptive convolutional layer, which uses The convolution kernel of the first adaptive convolution layer is the same as the other settings, and the output is the second scale feature map. The formula is: ; And also use the ReLU activation function to perform nonlinear transformation. Enter the third adaptive convolution layer and use The convolution kernel generates the third scale feature map , the calculation method is similar to the first two layers, but the convolution kernel has a larger coverage and captures more global spatial information. Through these three layers of convolution operations, local, mid-scale and global features are obtained respectively. In order to integrate these three features, the first scale feature map , the second scale feature map And the third scale feature map Input the dynamic feature converter to calculate the feature similarity. The dynamic feature converter generates feature weight coefficients by calculating the similarity between each pair of feature maps. , the weight calculation formula is: ; in Represents a feature similarity function, such as calculated by cosine similarity: ; Based on these weight coefficients, the three feature maps are weighted fused to generate a feature pyramid: ; in is a pyramid feature map. Perform channel attention calculation and generate a weight vector for each channel through a fully connected layer , the weight calculation formula is: ; in is the sigmoid activation function. Multiply the channel weight vector by the pyramid feature map channel by channel to obtain the channel weighted feature: ; Channel compression is performed on the channel weighted features, and the feature dimension is reduced by dimensionality reduction technology. The formula is: ; in , is the dimensionality reduction weight. and feature weight coefficient Perform matrix multiplication to obtain the final spatiotemporal feature representation: ; in It is the final spatiotemporal feature representation.

[0026] In one example, the spatiotemporal feature representation is input into a bi-fold teacher-student network for semi-supervised training, and the parameters are optimized through cross entropy loss and consistency loss to obtain an initial image annotation model, including: The spatiotemporal feature representation is divided into labeled training data and unlabeled training data to obtain a training data set; Input the labeled training data in the training data set into the teacher network in the bi-fold teacher-student network. The teacher network includes a feature extraction layer, an attention layer, and a classification layer. The feature extraction layer includes four convolution blocks, each of which includes two 3×3 convolution layers and a maximum pooling layer to obtain the teacher features. The unlabeled training data in the training data set is input into the teacher network, the category probability distribution is calculated based on the output of the teacher network, and the prediction results with probability values ​​greater than the threshold are used as pseudo labels to obtain pseudo labeled data; The pseudo-labeled data and the labeled training data are combined and input into the student network. The structure of the student network is the same as that of the teacher network, but the parameters are initialized independently to obtain the student features. Align the teacher features and the student features, and calculate the Euclidean distance between the normalized feature vectors as the consistency loss to obtain the feature consistency measure; A joint loss function is constructed based on feature consistency measurement and classification cross entropy loss, where the consistency loss weight coefficient is α and the cross entropy loss weight coefficient is 1-α, and the overall loss function is obtained; The parameters of the teacher network and the student network are optimized according to the overall loss function to obtain an optimized two-fold teacher-student network, and the teacher network in the optimized two-fold teacher-student network is used as the initial image annotation model.

[0027] In this example, the spatiotemporal features are represented as Divide into is the number of samples, is the number of time frames, and are the image height and width, is the number of feature channels. By random division, Divided into labeled training data and unlabeled training data ,in Contains the true label of the sample ,and There is no corresponding label. Assume that the labeled data accounts for ,but , . The training data will be labeled The teacher network is input into the bi-fold teacher-student network. The teacher network consists of a feature extraction layer, an attention layer, and a classification layer. The feature extraction layer consists of four convolutional blocks, each of which contains two Convolutional layer and a maximum pooling layer. Assume that the input data dimension is ,in , after the After convolution blocks, the feature map calculation formula is: ; in It is The output of the layer, * indicates the convolution operation, is the activation function (such as ReLU), and are the convolution kernel weight and bias, respectively. and Represents the feature size after pooling. After passing through four convolution blocks, the feature is input to the attention layer. The attention layer enhances the temporal correlation of the feature by calculating the temporal weight to obtain the teacher feature ,in is the final number of feature channels. Input the teacher network, the teacher network predicts the unlabeled data based on its current parameters and generates a category probability distribution ,in is the number of categories, The probability distribution is calculated by the Softmax function: ; in It is The classification score vector of unlabeled samples. When , the corresponding category is used as a pseudo label ,in is the confidence threshold. Through this process, pseudo-annotated data is generated Combine pseudo-annotated data with annotated training data to form an enhanced training dataset , and input into the student network for training. The structure of the student network is the same as that of the teacher network, but the parameters are initialized independently. The student network generates student features through the feature extraction layer and the attention layer. In order to ensure the consistency of the characteristics of the teacher network and the student network, the teacher characteristics and student characteristics Perform feature alignment and calculate the Euclidean distance between the normalized feature vectors as the consistency loss: ; At the same time, the classification cross entropy loss is calculated on the labeled data : ; in is the predicted probability distribution of the student network on the labeled data. By weighted combination of consistency loss and classification cross entropy loss, a joint loss function is constructed: ; in is a weight coefficient used to control the relative importance of consistency loss and cross entropy loss. The gradient descent method is used to optimize the parameters of the teacher network and the student network according to the overall loss function L. During the optimization process, the teacher network gradually improves the reliability of the pseudo-label, while the student network continuously learns feature representations from the teacher network. After multiple rounds of iterations, the optimized teacher network is used as the initial image annotation model. Through this process, the model can effectively improve the annotation performance using a small amount of labeled data and a large amount of unlabeled data.

[0028] In one example, the target image data stream collected in real time is input into the initial image annotation model for feature extraction and annotation prediction, and quality assessment is performed based on the predicted probability distribution and temporal consistency to obtain an annotation result with a quality score, including: The target image data stream collected in real time is segmented according to a fixed time window, and each image sequence is preprocessed and feature extracted to obtain a target image feature sequence; Input the target image feature sequence into the feature extraction layer of the initial image annotation model, perform feature mapping through the four convolution blocks in the feature extraction layer, and obtain the deep features of the target image; The spatial attention is calculated on the deep features of the target image through the attention layer of the initial image annotation model to obtain the attention weighted features; The attention-weighted features are input into the classification layer of the initial image annotation model for category prediction. The probability distribution of each category is calculated through the Softmax function to obtain the predicted probability vector. Compare the maximum probability value in the prediction probability vector with the preset confidence threshold, and take the prediction results higher than the threshold as high confidence annotations to obtain preliminary annotation results; Based on the preliminary annotation results, the annotation consistency between adjacent frames is calculated, and the temporal correlation of the annotation results is measured by cosine similarity to obtain the temporal consistency score; The maximum value of the predicted probability vector is taken as the confidence score and weightedly combined with the temporal consistency score, with weights of β and 1-β respectively, to obtain the comprehensive quality score. The preliminary annotation result and the comprehensive quality score are then associated to generate an annotation result with a quality score.

[0029] In this example, real-time target image data stream is collected from the ISP vision sensor, assuming that the data stream is a continuous frame ,in Indicates at time The image frames collected at each moment, is the total number of frames. For ease of processing and analysis, the data stream is divided into fixed time windows Each segment contains Frame image, get Segment image sequence block ,in For each image sequence Perform preprocessing operations, including normalization, image enhancement and noise suppression, to obtain a preprocessed image sequence. After the preprocessing is completed, each image sequence Input into a feature extraction module to extract the feature representation of each frame of the image and generate the target image feature sequence ,in and Represents the height and width of the image respectively, is the number of feature channels. Input the feature extraction layer of the initial image annotation model, which consists of four convolutional blocks, each of which includes two Convolutional layer and a maximum pooling layer. Assume the input feature is , after the After convolution blocks, the output features are: ; in It is The output features of the layer, is the weight matrix of the convolution kernel, is the bias vector, * represents the convolution operation, is the activation function (such as ReLU), These are the height, width, and number of channels after pooling. Through four convolution blocks, the deep features of the target image are generated. . The attention layer of the input model performs spatial attention calculations. The attention mechanism highlights important features by dynamically allocating spatial weights. Each channel of is weighted to generate attention weighted features The attention calculation formula is: ; in is the weight matrix used to generate queries and keys, and softmax ensures the normalization of weights. Input into the classification layer of the initial image annotation model, calculate the probability distribution of each category through the fully connected layer and Softmax function, and obtain the predicted probability vector ,in is the number of categories. The calculation formula of Softmax is: ; where z It is Frame in category The maximum probability value in the predicted probability vector With the preset confidence threshold For comparison, if , the corresponding category is marked as a high confidence label, and the preliminary labeling result is obtained In order to enhance the temporal consistency of annotation, the similarity of annotation results between adjacent frames is calculated. and The annotation consistency is measured by cosine similarity: ; The similarity of all frame pairs is averaged to obtain the temporal consistency score. . The maximum value of the predicted probability vector As confidence score , and the temporal consistency score By weight and Weighted combination to calculate the overall quality score: ; Initial marking results and comprehensive quality score Association, generate annotation results with quality scores.

[0030] In one example, high-quality samples in the annotation results with quality scores are stored in a sample buffer pool, sample screening is performed based on entropy and distance metrics, and the parameters of the initial image annotation model are updated through an incremental learning strategy to obtain a target image annotation model, including: The annotation results with quality scores are sorted in descending order according to the comprehensive quality scores, and the screening threshold is determined by the dynamic threshold selection method. Samples above the screening threshold are selected as high-quality samples and stored in the sample buffer pool; Normalize the predicted probability distribution of each sample in the sample buffer pool, calculate the entropy value of each category based on the Shannon entropy formula and sum them up to obtain a standardized sample entropy value sequence; The initial image annotation model is used to extract the deep features corresponding to the samples in the sample buffer pool, and the cosine distance metric is used to calculate the similarity between sample pairs to obtain the sample distance matrix. The distance is then transformed through the Gaussian kernel function to obtain the sample similarity distribution. Construct a sample importance score based on the standardized sample entropy value sequence and sample similarity distribution to obtain a sample weight vector; Performing stratified sampling on samples in the sample buffer pool according to the sample weight vector to obtain an incremental training set, and performing data enhancement on the incremental training set to obtain enhanced training samples; The enhanced training samples are input into the initial image annotation model, and the parameters are fine-tuned using knowledge distillation. The teacher network generates soft labels to guide the student network learning, and the updated model parameters are obtained. The updated model parameters are evaluated on the validation set to obtain a comprehensive performance score. The optimal model parameters are selected based on the comprehensive performance score, and the optimal model parameters are updated to the initial image annotation model to obtain the target image annotation model.

[0031] In this example, assume that the annotation results with comprehensive quality scores are represented as ,in Indicates samples, is the comprehensive quality score of the sample. In order to give priority to high-quality annotation results, the annotation results are sorted according to the comprehensive quality score. Sort in descending order. After sorting, the sample set becomes ,in . Determine the screening threshold by dynamic threshold selection method The threshold is adjusted dynamically according to the distribution of samples. The samples are taken as high-quality samples to generate a high-quality sample set . Store these high-quality samples in sample buffer pool B for subsequent processing. The predicted probability distribution of each sample in the sample buffer pool is Normalization is performed, where The probability distribution after normalization of the number of categories is: ; in It is a sample In the Normalized probability of the class. Based on the normalized probability distribution, calculate the Shannon entropy of each sample : ; The entropy value reflects the prediction uncertainty of the sample. The lower the value, the more certain the prediction. The entropy values ​​of all samples are standardized to obtain the standardized sample entropy value sequence : ; in and are the minimum and maximum entropy values, respectively. The initial image annotation model is used to extract the deep features of the samples in the sample buffer pool. Assume that the samples The deep features are ,in is the feature dimension. The feature similarity between sample pairs is calculated using the cosine distance metric: ; in Representation characteristics of Norm. Store the similarity results of all sample pairs in the sample distance matrix In order to enhance the usability of similarity, the distance matrix Apply Gaussian kernel function to transform and get sample similarity distribution : ; in is the bandwidth parameter of the Gaussian kernel function. Based on the standardized sample entropy value sequence Similarity distribution with samples , construct sample importance score , the formula is: ; in is the weighted coefficient of entropy and similarity. Sample importance score The higher the value, the greater the value of the sample in training. According to the sample importance score, the samples in the sample buffer pool are stratified and sampled. The samples are divided into high importance layer and low importance layer, and samples are drawn from each layer in proportion to generate incremental training sets. Perform data augmentation on the incremental training set, applying methods such as rotation, scaling, flipping, and noise addition to generate enhanced training samples . The enhanced training samples Input the initial image annotation model and use knowledge distillation to fine-tune the parameters. In knowledge distillation, the teacher network generates soft labels. To guide student network learning, the soft label is calculated by the Softmax function: ; in is the temperature parameter used to smooth the probability distribution. The student network learns by minimizing the KL divergence loss with the soft label, as follows: ; After the model is fine-tuned, the updated model is evaluated on the validation set to calculate the overall performance score. , for example, calculated by a weighted combination of accuracy and F1 score: ; The model parameters with the highest comprehensive performance score are selected as the optimal parameters, and they are updated to the initial image annotation model to finally generate the target image annotation model.

[0032] Reference Figure 2 This embodiment provides an automatic data labeling device for an ISP image signal processing visual sensor, including: Noise reduction module 1, used for performing noise reduction and signal enhancement on the original image data stream collected by the ISP image signal processing visual sensor to obtain a standardized image sequence; The first extraction module 2 is used to input the standardized image sequence into the temporal attention encoder to extract temporal features and obtain a temporal feature tensor; The second extraction module 3 is used to input the temporal feature tensor into the adaptive convolution layer and the dynamic feature converter to extract spatial features and obtain spatiotemporal feature representation; Training module 4 is used to input the spatiotemporal feature representation into the bi-fold teacher-student network for semi-supervised training, and optimize the parameters through cross entropy loss and consistency loss to obtain the initial image annotation model; Evaluation module 5, used to input the real-time collected target image data stream into the initial image annotation model for feature extraction and annotation prediction, and perform quality evaluation based on the predicted probability distribution and temporal consistency to obtain an annotation result with a quality score; The updating module 6 is used to store high-quality samples in the annotation results with quality scores into the sample buffer pool, screen the samples based on entropy and distance metrics, and update the parameters of the initial image annotation model through an incremental learning strategy to obtain the target image annotation model.

[0033] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the above method embodiment, which will not be repeated here.

[0034] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0035] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0036] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0037] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0038] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0039] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An automatic data annotation method for ISP image signal processing visual sensor, characterized in that: The following steps are involved: The original image data stream collected by the ISP image signal processing visual sensor is subjected to noise reduction and signal enhancement to obtain a standardized image sequence; Inputting the standardized image sequence into a temporal attention encoder to extract temporal features, thereby obtaining a temporal feature tensor; Inputting the temporal feature tensor into an adaptive convolutional layer and a dynamic feature converter to extract spatial features to obtain spatiotemporal feature representation; Inputting the spatiotemporal feature representation into a bi-fold teacher-student network for semi-supervised training, and optimizing parameters through cross entropy loss and consistency loss to obtain an initial image annotation model; Inputting the target image data stream collected in real time into the initial image annotation model for feature extraction and annotation prediction, and performing quality assessment based on the predicted probability distribution and temporal consistency to obtain an annotation result with a quality score; The high-quality samples in the annotation results with quality scores are stored in a sample buffer pool, the samples are screened based on entropy and distance metrics, and the parameters of the initial image annotation model are updated through an incremental learning strategy to obtain a target image annotation model.

2. The automatic data labeling method of the ISP image signal processing visual sensor according to claim 1, characterized in that: The ISP image signal processing visual sensor performs noise reduction and signal enhancement on the original image data stream collected to obtain a standardized image sequence, including: The ISP image signal processing visual sensor collects the original image data stream, and divides the original image data stream into time series to obtain image sequence blocks; Normalizing and mapping the pixel values ​​of each frame of the image in the image sequence block to obtain a normalized image sequence, and performing regional block division on the normalized image sequence to divide each frame of the image into a plurality of sub-regions to obtain an image sub-region sequence; Calculating the local signal strength of each sub-region in the image sub-region sequence to obtain an adaptive threshold sequence, and filtering the image sub-region sequence based on the adaptive threshold sequence to obtain a noise-reduced image sequence; Performing histogram statistics on the denoised image sequence, calculating the grayscale distribution of each frame of the image, obtaining a histogram feature sequence, calculating a gamma correction coefficient based on the histogram feature sequence, performing gamma correction processing on the denoised image sequence, and obtaining a contrast enhanced image; Edge detection and detail enhancement are performed on the contrast-enhanced image to obtain a standardized image sequence.

3. The automatic data labeling method of the ISP image signal processing visual sensor according to claim 2 is characterized in that: The step of inputting the standardized image sequence into a temporal attention encoder to extract temporal features and obtain a temporal feature tensor comprises: Inputting the standardized image sequence into a first layer of temporal attention network of a temporal attention encoder, wherein the first layer of temporal attention network comprises a first self-attention module and a first feedforward neural network; Performing attention calculation on the standardized image sequence through the first self-attention module to obtain a first layer of attention weighted features, and inputting the first layer of attention weighted features into the first feedforward neural network for feature transformation, and performing residual connection with the original features to obtain a first layer of temporal features; Inputting the first layer of temporal features into the second layer of temporal attention network of the temporal attention encoder, and extracting features through the second self-attention module and the second feedforward neural network in the second layer of temporal attention network to obtain the second layer of temporal features; Position encoding is performed on the second layer of time series features, and the position embedding vector is calculated using sine function and cosine function to obtain time series features with position information; Inputting the temporal features with position information into the third layer of the temporal attention network of the temporal attention encoder to perform multi-head attention calculation to obtain multi-head attention features; The multi-head attention features are layer-normalized, and feature dimension reduction is performed through a fully connected layer to obtain compressed features, and the compressed features are fused with the multi-head attention features to obtain a time series feature tensor.

4. The automatic data labeling method of the ISP image signal processing visual sensor according to claim 3 is characterized in that: The step of inputting the temporal feature tensor into an adaptive convolution layer and a dynamic feature converter to extract spatial features to obtain a spatiotemporal feature representation includes: Inputting the time series feature tensor into the first adaptive convolution layer for convolution operation, the first adaptive convolution layer adopts a 3×3 convolution kernel, a convolution step size of 1, a padding mode of SAME, and performs nonlinear transformation through a ReLU activation function to obtain a first scale feature map; Inputting the time series feature tensor into the second adaptive convolution layer for convolution operation, the second adaptive convolution layer adopts a 5×5 convolution kernel, a convolution step size of 1, a padding mode of SAME, and performs nonlinear transformation through a ReLU activation function to obtain a second scale feature map; Inputting the time series feature tensor into the third adaptive convolution layer for convolution operation, the third adaptive convolution layer adopts a 7×7 convolution kernel, a convolution step size of 1, a padding mode of SAME, and performs nonlinear transformation through a ReLU activation function to obtain a third scale feature map; Performing feature similarity calculation on the first scale feature map, the second scale feature map, and the third scale feature map through a dynamic feature converter to obtain a feature weight coefficient, and performing a feature pyramid based on the first scale feature map, the second scale feature map, and the third scale feature map to obtain a pyramid feature map; Performing channel attention calculation on the pyramid feature map to obtain a channel weight vector, and performing element-by-element multiplication of the channel weight vector with each channel of the pyramid feature map to obtain a channel weighted feature; Channel compression is performed on the channel weighted features to obtain compressed space features, and matrix multiplication operation is performed on the compressed space features and the feature weight coefficients to obtain spatiotemporal feature representation.

5. The automatic data labeling method of the ISP image signal processing visual sensor according to claim 4 is characterized in that: The spatiotemporal feature representation is input into a bi-fold teacher-student network for semi-supervised training, and parameters are optimized through cross entropy loss and consistency loss to obtain an initial image annotation model, including: Dividing the spatiotemporal feature representation into labeled training data and unlabeled training data to obtain a training data set; Inputting the labeled training data in the training data set into the teacher network in the bi-fold teacher-student network, the teacher network comprises a feature extraction layer, an attention layer and a classification layer, wherein the feature extraction layer comprises four convolution blocks, each convolution block comprises two 3×3 convolution layers and one maximum pooling layer, to obtain teacher features; Inputting the unlabeled training data in the training data set into the teacher network, calculating the category probability distribution based on the output of the teacher network, and taking the prediction results with probability values ​​greater than a threshold as pseudo labels to obtain pseudo labeled data; Combining the pseudo-annotated data and the annotated training data into a student network, wherein the structure of the student network is the same as that of the teacher network but the parameters are initialized independently, to obtain student features; Performing feature alignment on the teacher feature and the student feature, and obtaining a feature consistency measure by calculating the Euclidean distance between the normalized feature vectors as a consistency loss; Constructing a joint loss function based on the feature consistency measure and the classification cross entropy loss, wherein the consistency loss weight coefficient is α and the cross entropy loss weight coefficient is 1-α, and obtaining an overall loss function; The parameters of the teacher network and the student network are optimized according to the overall loss function to obtain an optimized two-fold teacher-student network, and the teacher network in the optimized two-fold teacher-student network is used as an initial image annotation model.

6. The automatic data labeling method of the ISP image signal processing visual sensor according to claim 5, characterized in that: The target image data stream collected in real time is input into the initial image annotation model for feature extraction and annotation prediction, and quality assessment is performed based on the predicted probability distribution and temporal consistency to obtain an annotation result with a quality score, including: The target image data stream collected in real time is segmented according to a fixed time window, and each image sequence is preprocessed and feature extracted to obtain a target image feature sequence; Inputting the target image feature sequence into the feature extraction layer of the initial image annotation model, performing feature mapping through four convolution blocks in the feature extraction layer, and obtaining deep features of the target image; Performing spatial attention calculation on the deep features of the target image through the attention layer of the initial image annotation model to obtain attention weighted features; Input the attention weighted features into the classification layer of the initial image annotation model for category prediction, calculate the probability distribution of each category through the Softmax function, and obtain a predicted probability vector; Compare the maximum probability value in the prediction probability vector with a preset confidence threshold, and mark the prediction results higher than the threshold as high confidence values ​​to obtain a preliminary marking result; Calculate the annotation consistency between adjacent frames based on the preliminary annotation results, measure the temporal correlation of the annotation results by cosine similarity, and obtain a temporal consistency score; The maximum value of the predicted probability vector is taken as the confidence score, and is weightedly combined with the temporal consistency score, with weights of β and 1-β respectively, to obtain a comprehensive quality score, and the preliminary annotation result and the comprehensive quality score are associated to generate an annotation result with a quality score.

7. The automatic data labeling method of the ISP image signal processing visual sensor according to claim 6, characterized in that: The method stores high-quality samples in the annotation results with quality scores into a sample buffer pool, screens samples based on entropy and distance metrics, and updates parameters of the initial image annotation model through an incremental learning strategy to obtain a target image annotation model, including: Sorting the annotation results with quality scores in descending order according to the comprehensive quality scores, determining a screening threshold by a dynamic threshold selection method, selecting samples above the screening threshold as high-quality samples, and storing the high-quality samples in a sample buffer pool; Normalizing the predicted probability distribution of each sample in the sample buffer pool, calculating the entropy value of each category based on the Shannon entropy formula and summing them up to obtain a standardized sample entropy value sequence; Extracting deep features corresponding to samples in the sample buffer pool by using the initial image annotation model, calculating similarities between sample pairs by using cosine distance metric to obtain a sample distance matrix, and performing distance transformation by using a Gaussian kernel function to obtain a sample similarity distribution; Constructing a sample importance score based on the standardized sample entropy value sequence and the sample similarity distribution to obtain a sample weight vector; Performing stratified sampling on the samples in the sample buffer pool according to the sample weight vector to obtain an incremental training set, and performing data enhancement on the incremental training set to obtain enhanced training samples; Inputting the enhanced training samples into the initial image annotation model, fine-tuning parameters by using knowledge distillation, generating soft labels through the teacher network to guide the student network learning, and obtaining updated model parameters; The updated model parameters are performance evaluated on the validation set to obtain a comprehensive performance score, the optimal model parameters are selected based on the comprehensive performance score, and the optimal model parameters are updated to the initial image annotation model to obtain the target image annotation model.

8. An automatic data labeling device for ISP image signal processing visual sensor, characterized in that: For implementing the steps of the method according to any one of claims 1 to 7, the device comprises: A noise reduction module is used to reduce noise and enhance signals on the original image data stream collected by the ISP image signal processing visual sensor to obtain a standardized image sequence; A first extraction module, configured to input the standardized image sequence into a temporal attention encoder to extract temporal features and obtain a temporal feature tensor; A second extraction module is used to input the temporal feature tensor into an adaptive convolution layer and a dynamic feature converter to extract spatial features and obtain a spatiotemporal feature representation; A training module, used for inputting the spatiotemporal feature representation into a bi-fold teacher-student network for semi-supervised training, and performing parameter optimization through a cross entropy loss and a consistency loss to obtain an initial image annotation model; An evaluation module is used to input the target image data stream collected in real time into the initial image annotation model for feature extraction and annotation prediction, and to perform quality evaluation based on the predicted probability distribution and temporal consistency to obtain an annotation result with a quality score; The updating module is used to store high-quality samples in the annotation results with quality scores into a sample buffer pool, screen samples based on entropy and distance metrics, and update the parameters of the initial image annotation model through an incremental learning strategy to obtain a target image annotation model.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images