A streaming perception method based on future feature self-supervised learning

Abstract

The application discloses a kind of based on future feature self-supervised learning streaming perception method, belong to computer vision field.The application implementation method is: the streaming perception model of future feature self-supervised learning is constructed, and streaming perception model is referred to as FFSSL model.FFSSL model includes StreamYOLO submodel, YOLOX submodel, self-supervised module.The appearance feature of future frame is used to enhance the performance of streaming perception, improve the capture ability of FFSSL model to the continuity and time sequence change in time series data, ensure the continuity and accuracy of prediction;In addition, the application combines the characteristics of StreamYOLO and YOLOX, realizes the deep analysis of each frame data in video sequence, obtains more rich, more varied feature representation, through comprehensive analysis to history, current and future frame, ensure that model can fully utilize the information of each frame at each step, through self-supervised learning strategy, effectively integrate the features of target at different time points, realize the high-precision, high-stability prediction of target.

Description

Technical Field

[0001] This invention relates to a streaming perception method based on self-supervised learning of future features for object detection, belonging to the field of computer vision. Background Technology

[0002] In the rapidly evolving field of computer vision, accurate and timely object detection is crucial, especially in practical applications such as autonomous driving and drone surveillance. Traditional object detection methods, whether image-based or video-based, primarily operate offline. They cannot keep up with the continuous and rapid changes in the real-world environment, thus compromising their effectiveness. The latency between data acquisition and processing allows environmental conditions to evolve, highlighting the urgent need for predictive methods capable of anticipating future states. To address this challenge, a new paradigm called streaming perception has been established and is gaining widespread application in current research.

[0003] Streaming sensing generates instantaneous predictions by processing continuous data streams. Unlike image object detection, which focuses solely on object detection in static images, and video object detection, which integrates historical data from consecutive frames, streaming sensing utilizes both current and historical frame data to predict the future location of objects. Given the constraints of object detection tasks, latency becomes a core challenge, impacting the accuracy and reliability of detection results. The latency between data acquisition and processing allows environmental conditions to evolve, highlighting the importance of streaming sensing, which can predict and rapidly respond to environmental changes, effectively addressing the latency issue.

[0004] Streaming sensing has spurred the proposal of numerous innovative methods. One early solution was the Streamer, a meta-detector. Streamers distinguish themselves through unique strategies such as decision-theoretic scheduling, asynchronous tracking, and Kalman filter-based prediction. However, a significant limitation of Streamers is their reliance on predefined scheduling. To address the scheduling problem of Streamers, adaptive Streamers were introduced. By using deep reinforcement learning, adaptive Streamers achieve more accurate predictions. Despite improved performance, the training process for adaptive Streamers is complex and computationally demanding. To address the challenges of adaptive Streamers, some new methods employ end-to-end strategies. Specifically, StreamYOLO reduces the complexity of the training process by utilizing temporal information contained in consecutive frames to predict future states. Similarly, LongShortNet, in its unique dual-path network, combines long-term temporal motion with short-term spatial semantics, facilitating streaming sensing.

[0005] However, existing end-to-end methods face the challenge of feature alignment, namely, a temporal mismatch between the input data of the current and past frames and the supervision data of the next frame. This mismatch often leads to inaccurate predictions when the target moves significantly from one frame to the next. In dynamic environments, rapid changes in the target are common, making this inaccuracy even more pronounced, thus requiring more adaptive methods. Summary of the Invention

[0006] To address the problem that current stream sensing algorithms rely heavily on coordinate information and cannot provide accurate predictions, this invention aims to provide a stream sensing method based on self-supervised learning of future features. This method utilizes the appearance features of future frames to enhance the performance of stream sensing, significantly improving the FFSSL model's ability to capture continuity and temporal changes in time-series data, ensuring the continuity and accuracy of predictions. Furthermore, this invention combines the features of StreamYOLO and YOLOX to achieve deep analysis of each frame in the video sequence, obtaining richer and more diverse feature representations. By comprehensively analyzing historical, current, and future frames, it ensures that the model fully utilizes the information of each frame at every step. Through a self-supervised learning strategy, it effectively integrates the features of the target at different time points, thereby achieving high-precision and high-stability predictions of the target.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] This invention discloses a streaming perception method based on self-supervised learning of future features, implemented using the FFSSL streaming perception model based on self-supervised learning of future features. By constructing input data, consisting of historical frames, the current frame, and the next frame, information about the spatiotemporal continuity of the FFSSL model is provided. A StreamYOLO sub-model is applied for initial processing; this structure includes a YOLOX sub-model, a feature fusion module, and a specific loss function to capture the dynamic characteristics of the video stream. The outputs of the StreamYOLO and YOLOX sub-models are used as inputs to the self-supervised module; these two inputs are the feature fusion output of the StreamYOLO sub-model and the feature output generated by the YOLOX sub-model in the next frame. To perform accurate feature alignment in this stage, target coordinate position labels are used to extract features from the two inputs. The input image is compared with the output features of the YOLOX and StreamYOLO sub-model feature fusion modules to obtain a scaling factor and a new ground truth label. The sample pair construction step systematically constructs anchor points and positive / negative sample pairs to ensure that the FFSSL model can learn more robust features from the data. The StreamYOLO sub-model's feature fusion output is used to extract anchor points containing target regions based on ground truth labels. The YOLOX sub-model is then used to process the features of the next frame, and positive samples with target features are extracted using scaled ground truth labels. Negative samples are constructed using background regions of the same size as the scaled ground truth labels. A feature consistency loss is introduced to ensure good consistency of features extracted from consecutive frames. This loss is constructed using anchor points and positive / negative samples to ensure feature alignment and consistency between consecutive frames in the video stream. Through object detection techniques, feature fusion strategies, and a self-supervised learning module, the accuracy of stream awareness is improved, providing strong technical support for practical applications such as video stream processing.

[0009] This invention discloses a streaming sensing method based on self-supervised learning of future features, comprising the following steps:

[0010] Step 1: Input the RGB data of the target to be detected, which consists of historical frames, current frame and next frame data. RGB data not only provides information on the spatiotemporal continuity of the target prediction, but also helps the streaming perception model to accurately capture the appearance features of the target in consecutive frames.

[0011] Step 2: Construct a self-supervised learning streaming perceptron model for future features, referred to as the FFSSL model. The FFSSL model includes a StreamYOLO sub-model, a YOLOX sub-model, and a self-supervised module. The StreamYOLO sub-model processes continuous frame data to predict the future state of the target. StreamYOLO mainly consists of a YOLOX sub-model, feature fusion, and a loss function. The self-supervised module trains the model using unlabeled RGB data, leveraging the consistency and continuity of the target in consecutive frames to enhance the learning ability of the streaming perceptron model without relying on a large amount of labeled data. Feature fusion combines features from historical frames and the current frame in consecutive frames, enhancing the FFSSL model's ability to predict future states. The loss function reduces the difference between the predicted and actual future states. When RGB data enters the model, the YOLOX sub-module extracts target features. These target features are then input into the StreamYOLO sub-model, where feature fusion combines features from multiple frames. The fused features are then fed into the self-supervised module to enhance the FFSSL model's predictive ability. The FFSSL model is optimized through a loss function to ensure prediction accuracy, enabling it to make comprehensive use of multi-frame data and achieve efficient and accurate target prediction.

[0012] Step 3: Input the RGB data of the detected target obtained in Step 1. Based on the current frame F in the video frame sequence... t This is defined as the center frame, and its preceding and following adjacent frames F are further determined. t-1 and F t+1 As the historical frame and the next frame. After constructing the training sample data set, the historical frame F t-1 and the current frame F t The information is processed by the feature fusion module, and the next frame F t+1 Feature extraction is performed using YOLOX. Historical frames F t-1 Current frame F t and the next frame F t+1 Three frames are used as input to the FFSSL model.

[0013] Step 4: Process the RGB data using the StreamYOLO sub-model. Train the model using the training sample data set generated in Step 3, and obtain the feature fusion output through the StreamYOLO structure.

[0014] Step 5: Calculate the classification loss, regression loss, and target degree loss using the output of the StreamYOLO sub-model. The loss function is used for label-supervised training in frame t+1.

[0015] Classification loss

[0016]

[0017] It is calculated based on cross-entropy loss. For each bounding box, this loss measures the difference between the predicted class probability and the actual class label. Where y o,c The actual label for target o belonging to category c, p o,c It is the probability that target o is predicted to be category c, where C is the total number of categories.

[0018] Regression loss

[0019]

[0020] Calculate the difference between the predicted bounding box coordinates and the actual coordinates. In the FFSSL model, the regression loss is calculated using the mean squared error (MSE). Where N... pos b is the number of positive sample bounding boxes. i These are the predicted bounding box coordinates. These are the actual coordinates.

[0021] Targeted loss

[0022]

[0023] This is used to distinguish between the target and the background in an image. Considering the probability that a predicted bounding box contains the target, this loss is calculated using binary cross-entropy loss, where N is the number of bounding boxes, and y... i This indicates whether bounding box i contains the actual label of the target; 1 indicates the target is included, 0 indicates the background, and p... i It is the probability that the predicted bounding box i contains the target.

[0024] Step 6: The feature fusion output of StreamYOLO and the feature output of the next frame obtained through YOLOX are fed into the self-supervised module. The self-supervised module uses target coordinate position labels to extract features from the two inputs. For feature extraction, both inputs use target coordinate position labels, and a scaling factor is obtained by comparing the input image with the features output by the YOLOX and StreamYOLO sub-model feature fusion modules, thus acquiring a new ground truth label. The self-supervised module compares the input image with the output features of the YOLOX and StreamYOLO feature fusion modules and calculates a new ground truth label.

[0025] To calculate the scaling ratio based on the input image and output features, the width and height of the two images are defined: I input The input image has a width and height of w. input and h input ; and I feature The output image of feature fusion has a width and height of w.feature and h feature .

[0026] The scaling ratios are calculated separately for width and height:

[0027]

[0028]

[0029] Based on the scaling ratio between the input image and the features, the self-supervised module calculates a new ground truth label.

[0030] Step 7: Using the extracted features and ground truth labels, perform sample pair construction, i.e., construct anchor points and positive / negative sample pairs: Use the features output by the StreamYOLO sub-model feature fusion module, and extract anchor points with target regions using ground truth labels; use the features extracted from the next frame by the YOLOX sub-model, and extract positive samples with target features using scaled ground truth labels; construct negative samples using a background region of the same size as the scaled ground truth label region. Constructing sample pairs ensures that the FFSSL model can capture more robust features from the data. Using the fused features from the StreamYOLO sub-model in Step 4, obtain the feature map f. t-1:t Calculate F based on the truth label of the target. t The scaling ratio between the target and the fused features is used to calculate the precise location of the target in the feature map and obtain the target's anchor point region.

[0031] Similarly, using the features from the next frame processed by the YOLOX sub-model in step 4, and combining them with the scaled ground truth labels, we obtain a positive sample f with the target features. t+1 This refers to the features of frame t+1. From this feature map, based on the position of the anchor point in the first feature map, a target region corresponding to the anchor point is selected. This positive sample is similar in appearance to the anchor point, and therefore is considered an effective learning target in self-supervised learning.

[0032] Using the features processed by YOLOX in step 4 as input, and extracting a background region of the same size as the anchor point using the scaled ground truth labels, negative samples are obtained. These negative samples provide a different perspective for comparison with positive samples, thus enhancing the discriminative power of the FFSSL model. To maintain a balance between positive and negative samples, a negative sample is randomly selected for each positive sample. In the FFSSL model, the sample pair construction process promotes self-supervised learning in the StreamYOLO sub-model, accurately capturing the feature information of the target between consecutive frames. The FFSSL model enhances the consistency of foreground features between the fused features of the previous two frames and the features of the next frame through a self-supervised learning module, achieving deep analysis of each frame's data in the video sequence, resulting in richer and more diverse feature representations. These feature representations not only reflect changes in the target's position but also capture changes in its appearance, size, shape, and other related attributes, ensuring that the FFSSL model fully utilizes the information of each frame at every step.

[0033] Step 8: Construct Feature Consistency Loss using Anchor Points and Positive / Negative Samples. To ensure good consistency of features extracted from consecutive frames, the feature consistency loss (FCL) is constructed using the anchor points and positive / negative samples from Step 4.

[0034]

[0035] s = s(f t-1:t ,f t+1 )

[0036]

[0037] Among them, f t-1:t f represents the feature obtained by fusing the features of frame (t-1) and frame (t). t+1 Let represent the features of the (t+1)th frame, and s(f t-1:t ,f t+1 ) is f t-1:t and f t+1 A measure of similarity between feature representations. Variables This represents the negative samples selected from the non-target region of frame t+1. τ is a temperature parameter used to adjust the sharpness of the distribution.

[0038] The goal of FCL (Fast-Locked Classification) is to maximize the mutual information between positive samples (anchor points and their corresponding targets in the next frame) and negative samples, thereby improving the effectiveness of the self-supervised learning process. Implementing the FCL mechanism optimizes the effectiveness of the self-supervised learning strategy, contributing to improved streaming perception performance. FCL, combined with other key losses mentioned in step 2, constitutes the loss function during training. The self-supervised learning strategy effectively integrates the features of the target at different time points, thus achieving high-precision and high-stability predictions of the target.

[0039] Step 9: To ensure the predictive performance of the FFSSL model, construct a comprehensive loss function by combining Feature Consistency Loss (FCL) and Supervision Loss.

[0040]

[0041] By combining object detection technology, feature fusion strategies, and self-supervised learning modules, the accuracy of stream perception is improved, and strong technical support is provided for practical application scenarios such as video stream processing.

[0042] Step 10: The self-supervised module is no longer used in the inference phase of the FFSSL model, but the feature representation formed by self-supervised learning in the training phase still significantly enhances the prediction ability, thereby improving the prediction accuracy and stability of the target and effectively solving the problem that the current flow-aware algorithm mainly relies on coordinate information and cannot provide accurate prediction.

[0043] Beneficial effects:

[0044] 1. This invention discloses a streaming perception method based on self-supervised learning of future features. It utilizes the appearance features of future frames to enhance the performance of streaming perception, significantly improving the ability of the FFSSL model to capture continuity and temporal changes in time-series data. By considering the appearance features of future frames, this method not only improves the ability to predict short-term continuity but also possesses strong identification and processing capabilities for long-term dependencies. Especially when the target is partially occluded or disappears in the current frame, the information from future frames not only provides the model with additional spatiotemporal continuity information but also provides strong support for continuous prediction, ensuring the continuity and accuracy of the predictions.

[0045] 2. This invention discloses a streaming perception method based on self-supervised learning of future features. This method does not rely entirely on coordinate information for perception, enabling the FFSSL model to capture all possible attributes and features of an object in greater detail, not just its position. Combining the features of StreamYOLO and YOLOX, this method achieves deep analysis of each frame of data in a video sequence, resulting in richer and more diverse feature representations. These feature representations not only reflect changes in the target's position but also capture changes in its appearance, size, shape, and other related attributes. Through comprehensive analysis of historical, current, and future frames, this method ensures that the FFSSL model fully utilizes the information of each frame at every step and effectively integrates the target's features at different time points through a self-supervised learning strategy, thereby achieving high-precision and high-stability prediction of the target. Attached Figure Description

[0046] Figure 1This is a flowchart illustrating a streaming perception method based on self-supervised learning of future features according to the present invention.

[0047] Figure 2 This is a system block diagram of a streaming perception method based on self-supervised learning of future features according to the present invention;

[0048] Figure 3 This is a schematic diagram comparing the perception results of a streaming perception method based on self-supervised learning of future features according to the present invention. Detailed Implementation

[0049] To better illustrate the purpose and advantages of the present invention, the invention will be further described below in conjunction with the accompanying drawings and examples.

[0050] Example 1:

[0051] like Figure 1 As shown, this embodiment describes a streaming perceptual method (FFSSL) based on self-supervised learning of future features. First, input data is constructed, consisting of historical frames, the current frame, and the next frame. A StreamYOLO structure is introduced for processing, which includes YOLOX, a feature fusion module, and a loss function. The self-supervised module is designed to receive the output of the StreamYOLO feature fusion and the feature output of the next frame processed by YOLOX. Both inputs use target coordinate location labels. By comparing the input image with the output features of YOLOX and the StreamYOLO feature fusion module, a scaling ratio is obtained, and a new ground truth label is obtained accordingly. Using this extracted feature and ground truth label, sample pairs are constructed. Using the output of the StreamYOLO feature fusion module, anchor points with target regions are extracted using the ground truth labels. Then, using the features of the next frame extracted by YOLOX, positive samples with target features are extracted using the scaled ground truth labels, and negative samples are constructed using a background region of the same size as the scaled ground truth label region. Finally, feature consistency loss is constructed using anchor points and positive / negative sample pairs. The flowchart of this embodiment is shown below. Figure 1 As shown, the system block diagram is as follows: Figure 2 As shown.

[0052] Step 1: Input the RGB data of the target to be detected, which consists of historical frames, current frame and next frame data. RGB data not only provides information on the spatiotemporal continuity of the target prediction, but also helps the model to accurately capture the appearance features of the target in consecutive frames.

[0053] Step 2: Construct a self-supervised streaming perceptron model for future features, referred to as the FFSSL model. The FFSSL model includes a StreamYOLO sub-model, a YOLOX sub-model, and a self-supervised module. The StreamYOLO sub-model processes continuous frame data to predict the future state of the target. StreamYOLO mainly consists of a YOLOX sub-model, feature fusion, and a loss function. The self-supervised module trains the model using unlabeled RGB data, leveraging the consistency and continuity of the target in consecutive frames to enhance the model's learning ability without relying on a large amount of labeled data. Feature fusion combines features from historical and current frames in consecutive frames, enhancing the model's ability to predict future states. The loss function reduces the difference between the predicted and actual future states. When RGB data enters the model, the YOLOX sub-module extracts target features. These target features are then input into the StreamYOLO sub-model, where feature fusion combines features from multiple frames. The fused features are then fed into the self-supervised module to enhance the FFSSL model's predictive ability. Throughout all these processes, the FFSSL model is optimized using the loss function to ensure prediction accuracy. In this way, the FFSSL model can make comprehensive use of multiple frames of data to achieve efficient and accurate target prediction.

[0054] Step 3: Input the RGB data of the detected target obtained in Step 1, and process the RGB data into a video frame sequence of size 960x600. Define the current frame Ft in the video frame sequence as the center frame, and further determine its preceding and following frames F. t-1 and F t+1 As the historical frame and the next frame. After constructing the training sample data set, the historical frame F t-1 and the current frame F t The information is processed by the feature fusion module, and the next frame F t+1 Feature extraction was performed using YOLOX. These three frames: historical frame F t-1 Current frame F t , and the next frame F t+1 Use a size of 960x600 as input for the FFSSL model for the next step of processing.

[0055] Step 4: Process the data using the StreamYOLO structure, which integrates the features of YOLOX, combining feature fusion and a specific loss function. Train the dataset using the training samples generated in Step 1, and obtain the feature fusion output through the StreamYOLO structure.

[0056] Step 5: Calculate the classification loss, regression loss, and target degree loss using the output of the StreamYOLO sub-model, and use them for label-supervised training at frame t+1.

[0057] Classification loss

[0058]

[0059] It is calculated based on cross-entropy loss. For each bounding box, this loss measures the difference between the predicted class probability and the actual class label. Where y o,c The actual label for target o belonging to category c, p o,c It is the probability that target o is predicted to be category c, where C is the total number of categories.

[0060] Regression loss

[0061]

[0062] Calculate the difference between the predicted bounding box coordinates and the actual coordinates. In FFSSL, the regression loss is calculated using the mean squared error (MSE). Where N... pos b is the number of positive sample bounding boxes. i These are the predicted bounding box coordinates. These are the actual coordinates.

[0063] Targeted loss

[0064]

[0065] This is used to distinguish between the target and the background in an image. The loss is calculated using binary cross-entropy loss, considering the probability that the predicted bounding box contains the target. Here, N is the number of bounding boxes, and y... i This indicates whether bounding box i contains the actual label of the target; 1 indicates the target is included, 0 indicates the background, and p... i It is the probability that the predicted bounding box i contains the target.

[0066] Step 6: The feature fusion output of StreamYOLO and the feature output of the next frame obtained through YOLOX are fed into the self-supervised module. The self-supervised module uses the target coordinate position label to extract features from the two inputs. The self-supervised module compares the input image with the output features of the YOLOX and StreamYOLO feature fusion modules. Based on this comparison, the module calculates a scaling factor to generate a new ground truth label.

[0067] Step 7: The main purpose of constructing sample pairs is to ensure that the model can capture more robust features from the data. This step constructs anchor points and positive and negative sample pairs. Using the fused features from StreamYOLO in Step 4, the feature map f is obtained. t-1:t Calculate F based on the truth label of the target. tThe scaling ratio between the target and the fused features is used to calculate the precise location of the target in the feature map and obtain the target's anchor point region.

[0068] Similarly, using the features from the next frame processed by YOLOX in step 4, and combining them with the scaled ground truth labels, we obtain a positive sample f with target features. t+1 This refers to the features of frame t+1. From this feature map, based on the position of the anchor point in the first feature map, a target region corresponding to the anchor point is selected. This positive sample is similar in appearance to the anchor point, and therefore is considered an effective learning target in self-supervised learning.

[0069] Using the features processed by YOLOX in step 4 as input, and extracting a background region of the same size as the anchor point using the scaled ground truth labels, negative samples are obtained. These negative samples provide a different perspective for comparison with positive samples, thus enhancing the model's discriminative ability. To maintain a balance between positive and negative samples, a negative sample is randomly selected for each positive sample; in this implementation example, only one pair of positive and negative samples is selected. Within the FFSSL framework, samples play a role in promoting self-supervised learning during the construction process. This process constructs a pair of positive and negative samples, accurately capturing the feature information of the target between consecutive frames. The FFSSL framework enhances the consistency of foreground features between the fused features of the previous two frames and the features of the next frame through this strategy, thereby optimizing overall performance.

[0070] Step 8: To ensure good consistency of features extracted from consecutive frames, use the anchor points and positive and negative samples from Step 4 to construct the Feature Consistency Loss (FCL).

[0071]

[0072] s = s(f t-1:t ,f t+1 )

[0073]

[0074] Among them, f t-1:t f represents the feature obtained by fusing the features of frame (t-1) and frame (t). t+1 Let represent the features of the (t+1)th frame, and s(f t-1:t ,f t+1 ) is f t-1:t and f t+1 A measure of similarity between feature representations. Variables This represents the negative samples selected from the non-target region of frame t+1. Finally, τ is a temperature parameter used to adjust the sharpness of the distribution.

[0075] The goal of FCL (Fast Closed Array) is to maximize the mutual information between positive samples (anchor points and their corresponding targets in the next frame) and negative samples, thereby improving the effectiveness of the self-supervised learning process. Implementing this mechanism optimizes the effectiveness of the self-supervised learning strategy, contributing to improved streaming perception performance. FCL, combined with other key losses mentioned in step 2, constitutes the loss function during training.

[0076] Step 9: To ensure the model's predictive performance, this step employs a comprehensive loss function that combines Feature Consistency Loss (FCL) with traditional supervised losses. These supervised losses include classification loss, regression loss, and target loss.

[0077] The total loss is:

[0078]

[0079] By designing and combining these loss components in this way, FFSSL achieves more accurate predictions of targets in video streams. The Adam optimizer is used for network optimization.

[0080] Step 10: Self-supervised modules are no longer used in the inference phase of the model, but the feature representations formed by self-supervised learning during the training phase still significantly enhance the prediction ability, thereby improving the prediction accuracy and stability of the target and effectively solving the problem that the current flow perception algorithm mainly relies on coordinate information and cannot provide accurate prediction.

[0081] This embodiment will illustrate the effects of the present invention from two aspects.

[0082] 1. Experimental conditions

[0083] This embodiment discloses a streaming perception method based on self-supervised learning of future features. The method is trained and predicted on the Argoverse-HD dataset using the PyTorch deep learning framework in an environment with four NVIDIA RTX 3090 GPUs.

[0084] 2. Experimental Results

[0085] The Argoverse-HD dataset presents realistic urban driving scenarios, providing a challenging and diverse testing environment for tool evaluation. The method proposed in this embodiment has been rigorously tested on this dataset.

[0086] Figure 3 This provides an intuitive demonstration and in-depth comparison of the effectiveness of different flow sensing methods in handling complex traffic scenarios. Specifically, Figure 3(a) Presents the prediction results of the LongShortNet method on three different video frame sequences. Although the method can give good predictions in some scenarios, its prediction performance is not ideal in other situations, especially when faced with complex traffic conditions. Figure 3 (b) Further demonstration of the performance of the proposed method in the same scenario. Compared to LongShortNet, this method not only provides more accurate predictions in simple scenarios but also demonstrates superior stability and accuracy in more complex scenarios. For example, when a vehicle is turning, the method of this embodiment still provides high-quality prediction results, highlighting its strong adaptability and robustness. The figure illustrates the differences between the method of this embodiment and other methods, further demonstrating its superiority in handling flow perception tasks in real-world traffic scenarios.

[0087] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is merely a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A streaming perception method based on self-supervised learning of future features, characterized in that: Includes the following steps, Step 1: Input the RGB data of the target to be detected, which consists of historical frame, current frame and next frame data. The RGB data provides information on the spatiotemporal continuity of the target prediction, which helps the streaming perception model to accurately capture the appearance features of the target in consecutive frames. Step 2: Construct a streaming perceptron model for self-supervised learning of future features, referred to as the FFSSL model. The FFSSL model includes a StreamYOLO sub-model, a YOLOX sub-model, and a self-supervised module. The StreamYOLO sub-model is used to process continuous frame data and predict the future state of the target. The StreamYOLO sub-model consists of a YOLOX sub-model, a feature fusion module, and a loss function. The self-supervised module is used to train the model using unlabeled RGB data, leveraging the consistency and continuity of the target in continuous frames to enhance the learning ability of the streaming perceptron model without relying on a large amount of labeled data. The feature fusion module is used to fuse features from historical frames and the current frame in consecutive frames to enhance the FFSSL model's ability to predict future states; the loss function is used to reduce the difference between the prediction and the actual future state; when RGB data enters the model, the YOLOX submodule extracts the target features from it; The target features are then input into the StreamYOLO sub-model, where the feature fusion module fuses the features from multiple frames together. The fused features are then fed into the self-supervised module to enhance the predictive power of the FFSSL model. The FFSSL model is optimized through a loss function to ensure the accuracy of the predictions, enabling the FFSSL model to make comprehensive use of data from multiple frames. Step 3: Input the RGB data of the detected target obtained in Step 1; extract the current frame F from the video frame sequence. t Defined as the center frame, and further determine its preceding and following adjacent frames F. t-1 and F t+1 As the historical frame and the next frame; after the training sample data set is constructed, the historical frame F t-1 and the current frame F t The information is processed by the feature fusion module, and the next frame F t+1 Feature extraction is performed using YOLOX; historical frames F t-1 Current frame F t and the next frame F t+1 Three frames are used as input to the FFSSL model; Step 4: Process the RGB data based on the StreamYOLO sub-model; train using the training sample data set generated in Step 3, and obtain the output of the feature fusion module through the StreamYOLO structure; Step 5: Calculate the classification loss, regression loss, and target degree loss using the output of the StreamYOLO sub-model. The loss function is used for label-supervised training in frame t+1. Step 6: The output of the StreamYOLO feature fusion module and the feature output of the next frame obtained through YOLOX are fed into the self-supervised module; the self-supervised module uses the target coordinate position label to extract features from the two inputs; both inputs use the target coordinate position label, and based on the comparison of the input image with the features output by the YOLOX sub-model and the StreamYOLO sub-model feature fusion module, a scaling ratio is obtained, thereby obtaining a new ground truth label; the self-supervised module compares the input image with the output features of the YOLOX and StreamYOLO feature fusion modules, and calculates a new ground truth label; Step 7: Construct sample pairs using the extracted features and ground truth labels, i.e., construct anchor points and positive / negative sample pairs: Use the features output by the StreamYOLO sub-model feature fusion module to extract anchor points with target regions through ground truth labels; Use the features of the next frame extracted by the YOLOX sub-model to extract positive samples with target features through scaled ground truth labels. Negative samples are constructed using a background region of the same size as the scaled ground truth label region; constructing sample pairs ensures that the FFSSL model can capture more robust features from the data; Using the fused features from the StreamYOLO sub-model in step 4, we obtain the feature map f. t-1:t ; Calculate F based on the truth label of the target. t The scaling ratio between the target and the fused features is used to calculate the precise location of the target in the feature map and obtain the target's anchor point region. Step 8: Construct a feature consistency loss using anchor points and positive and negative samples. Step 9: To ensure the predictive performance of the FFSSL model, incorporate feature consistency loss. Construct a comprehensive loss function with the supervised loss By combining object detection technology, feature fusion strategies, and self-supervised learning modules, the accuracy of flow perception is improved. Step 10: The self-supervised module is no longer used in the inference phase of the FFSSL model, but the feature representation formed by self-supervised learning in the training phase still significantly enhances the prediction ability, thereby improving the prediction accuracy and stability of the target.

2. The streaming perception method based on self-supervised learning of future features as described in claim 1, characterized in that: In step 5, Classification loss It is calculated based on cross-entropy loss; for each bounding box, this loss measures the difference between the predicted class probability and the actual class label; where y o,c The actual label for target o belonging to category c, p o,c It is the probability that target o is predicted to be category c, where C is the total number of categories; Regression loss Calculate the difference between the predicted bounding box coordinates and the actual coordinates; in the FFSSL model, the regression loss is calculated using the mean squared error (MSE); where N pos b is the number of positive sample bounding boxes. i These are the predicted bounding box coordinates. These are the actual coordinates; Target degree loss In an image, this loss is used to distinguish between the target and the background; it considers the probability that the predicted bounding box contains the target, and is calculated using the binary cross-entropy loss, where N is the number of bounding boxes, and y... i This indicates whether bounding box i contains the actual label of the target; 1 indicates the target is included, 0 indicates the background, and p... i It is the probability that the predicted bounding box i contains the target.

3. The streaming perception method based on self-supervised learning of future features as described in claim 2, characterized in that: In step 6, Calculate the scaling ratio based on the input image and output features, and define the width and height of the two images: I input The input image has a width and height of w. input and h input ; and I feature The output image of feature fusion has a width and height of w. feature and h feature ; The scaling ratios for width and height are as follows: Based on the scaling ratio between the input image and the features, the self-supervised module calculates a new ground truth label.

4. The streaming perception method based on self-supervised learning of future features as described in claim 3, characterized in that: In step 7, Using the features from the next frame processed by the YOLOX sub-model in step 4, and combining them with the scaled ground truth labels, we obtain a positive sample f with the target features. t+1 The first feature is the feature of the (t+1)th frame; based on the position of the anchor point in the first feature, the target region corresponding to the anchor point is selected; the positive sample and the anchor point are similar in appearance and are regarded as effective learning targets in self-supervised learning. The features of the next frame processed by YOLOX in step 4 are used as input, and the background region with the same size as the anchor point is extracted using the scaled ground truth label to obtain negative samples. To maintain the balance between the number of positive and negative samples, a negative sample is randomly selected for each positive sample. In the FFSSL model, the sample pair construction process promotes self-supervised learning for the StreamYOLO sub-model, accurately capturing the feature information of the target between consecutive frames. The FFSSL model enhances the consistency of foreground features between the fused features of the first two frames and the features of the next frame through the self-supervised learning module, realizing deep analysis of the data of each frame in the video sequence, thereby obtaining richer and more diverse feature representations. The feature representation not only reflects changes in the target's position, but also captures changes in its appearance, size, shape, and other related attributes, ensuring that the FFSSL model can fully utilize the information of each frame at every step.

5. The streaming perception method based on self-supervised learning of future features as described in claim 4, characterized in that: In step 8, s=s(f t-1:t ,f t+1 ) Among them, f t-1:t f represents the feature obtained by fusing the features of frame (t-1) and frame (t). t+1 Let represent the features of frame t+1, and s(f) t-1:t ,f t+1 ) is f t-1:t and f t+1 Similarity measure between feature representations; variables This represents the negative samples selected from the non-target region of frame t+1; τ is a temperature parameter used to adjust the sharpness of the distribution. The goal is to maximize the mutual information between positive and negative samples, thereby improving the effectiveness of the self-supervised learning process. Combined with other key losses mentioned in step 5, this forms the loss function during the training process. The self-supervised learning strategy effectively integrates the features of the target at different time points, thereby achieving high-precision and high-stability prediction of the target.

6. The streaming perception method based on self-supervised learning of future features as described in claim 5, characterized in that: In step 9, Comprehensive loss function 。

Images