A video processing method and system based on edge computing and confidence evaluation

Abstract

The present application improves the efficiency and accuracy of video processing through region of interest prediction and confidence assessment. The ROI region is identified from each frame of video, reducing redundant information and significantly reducing data processing. Preprocessing is performed on each frame of video to extract feature vectors, including object class, size and location information, to support subsequent model selection. In terms of confidence assessment, the confidence score of each model is calibrated to ensure that the score accurately reflects the correctness of the classification result. According to the calibrated score, the model with high confidence and low processing delay is preferentially selected for processing; if the confidence score is low or the delay is high, the video frame is uploaded to the cloud for further processing. To further optimize the efficiency and accuracy of video processing, the present application introduces a dynamic programming algorithm to construct a solution graph and find the optimal path to determine the processing model for each video frame, achieving efficient and accurate video processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and edge computing, and more particularly, to a video processing method and system based on edge computing and confidence evaluation. BACKGROUND

[0002] With the rapid development of mobile devices and edge computing technology, more and more applications require efficient video processing on mobile devices. These applications include but are not limited to intelligent monitoring, unmanned aerial vehicle navigation, virtual reality (VR) and augmented reality (AR), etc. Traditional video processing methods usually rely on central servers to upload video data to the cloud for processing. Although this method can take advantage of powerful computing resources, it has significant limitations. First, high network latency is one of the main problems, especially in wireless or unstable network conditions, the uploading and downloading process of video data will cause significant delay. Second, the transmission of large amounts of video data requires a large amount of network bandwidth, increasing the transmission cost and possibly causing network congestion.

[0003] In order to overcome the limitations of traditional methods, mobile devices and edge computing technology have gradually become a research hotspot. Mobile devices have the characteristics of portability and low power consumption, while edge computing can perform calculations close to the data source, reducing data transmission delay and cost. Specifically, edge computing can perform processing close to the data source, significantly reducing data transmission time and thus reducing processing delay. By performing preliminary processing on edge devices, the amount of data that needs to be transmitted can be reduced, reducing transmission costs. At the same time, edge computing can take advantage of the local computing resources of mobile devices, reducing energy consumption caused by data transmission and remote processing, and improving energy efficiency. Although mobile devices and edge computing technology have many advantages, they still face some challenges in practical applications. For example, mobile devices have limited computing resources and cannot run complex deep learning models; the network conditions of edge devices are unstable, affecting the reliability and efficiency of data transmission. SUMMARY

[0004] The present application provides a video processing method and system based on edge computing and confidence evaluation to overcome the low efficiency and low precision of existing video processing methods, for efficient processing of two-dimensional video on mobile devices, improving processing accuracy and meeting delay constraints.

[0005] To solve the above technical problems, the technical solution of the present application is as follows:

[0006] The present application provides a video processing method based on edge computing and confidence evaluation, comprising:

[0007] Obtain a video stream and divide the video stream into independent video frames;

[0008] preprocessing each video frame to obtain a plurality of preprocessed video frames;

[0009] performing feature extraction based on the plurality of preprocessed video frames to generate corresponding feature vectors;

[0010] determining a key ROI region in the corresponding video frame according to the plurality of preprocessed video frames and the corresponding feature vectors;

[0011] selecting a corresponding classification model according to the corresponding feature vectors and the key ROI region, and calibrating a confidence score of the model;

[0012] monitoring the current network bandwidth and delay in real time, setting network bandwidth threshold and delay threshold, and adjusting the selection of the classification model according to the network bandwidth threshold and the delay threshold;

[0013] further adjusting the classification model of the current video frame by using a dynamic programming algorithm to obtain the best classification model of the current video frame, and processing the current video frame by using the best classification model.

[0014] Preferably, preprocessing each video frame to obtain a plurality of preprocessed video frames comprises:

[0015] normalizing each video frame to a uniform resolution to obtain a plurality of normalized video frames;

[0016] performing denoising processing on the plurality of normalized video frames to obtain a plurality of preprocessed video frames.

[0017] Preferably, determining a key ROI region in the corresponding video frame according to the plurality of preprocessed video frames and the corresponding feature vectors comprises:

[0018] creating a common ROI region set and a special ROI region set, wherein the common ROI region set and the special ROI region set are used to store different types of ROI regions;

[0019] obtaining an ROI region set of the plurality of preprocessed video frames, and calculating the number of objects in the ROI region set;

[0020] obtaining the class, size and position of the object in the preprocessed video frame according to the corresponding feature vector;

[0021] If the object size in the ROI region set of the plurality of pre-processed video frames is less than or equal to a preset size of a field of view, and a merging condition is met, the object is merged into a common ROI region set; if the merging condition is not met, a new common ROI region is created to contain the object and is merged into the common ROI region set; if the object size is greater than the preset size of the field of view, a special ROI region is created, the special ROI region is adjusted according to the object size and a preset scaling factor, the adjusted special ROI region is used to contain the object, and the object is merged into a special ROI set;

[0022] After processing the ROI region set of all pre-processed video frames, the merged common ROI region and the special ROI region are merged to obtain a final ROI region set;

[0023] The weight of each ROI region in the final ROI region set is determined, and a key ROI region is selected according to a preset weight.

[0024] Preferably, the formula for adjusting the special ROI region according to the object size and the preset scaling factor is:

[0025] S i ′ =y×Area(O i )

[0026] wherein y is a preset scaling factor, and Area(O i ) is the area of the object.

[0027] Preferably, a corresponding classification model is selected according to the corresponding feature vector and the key ROI region, and a confidence score of the model is calibrated, including:

[0028] A corresponding classification model is selected according to the corresponding feature vector and the key ROI region, and an original confidence score of the key ROI region is obtained through the model;

[0029] A calibration data set is constructed according to the feature vector, the true label, and the original confidence score of the key ROI region;

[0030] The calibration data set is input into a trained calibration model to obtain a calibrated confidence score;

[0031] The calibrated confidence score is used to replace the original confidence score of the classification model.

[0032] Preferably, the trained calibration model is obtained, including:

[0033] A calibration model is constructed, the calibration data set is input into the trained calibration model to obtain a calibrated confidence score;

[0034] A log-likelihood loss function is set, a log-likelihood loss function value is calculated according to the original confidence score and the calibrated confidence score, and the constructed calibration model is adjusted in parameters;

[0035] When the log-likelihood loss function value reaches a preset value or reaches a preset training number of times, a trained calibration model is obtained.

[0036] Preferably, the log-likelihood loss function is:

[0037]

[0038] Wherein, y i is a real label; is a calibrated confidence score; p i is an original confidence score.

[0039] Preferably, the constructed calibration model is adjusted in parameters by using a gradient descent method, specifically:

[0040] The loss function is minimized using the gradient descent method, and the formula is:

[0041]

[0042] Wherein, y i is a real label; is a calibrated confidence score; p i is an original confidence score; is a gradient of the loss function of the first model parameter; is a gradient of the loss function of the second model parameter.

[0043] The model parameters are updated by the gradient descent, and the formula is:

[0044]

[0045] Wherein, A is the first model parameter; B is the second model parameter; η is a learning rate, which controls the step size of parameter update in each iteration.

[0046] Preferably, the classification model of the current video frame is further adjusted by using a dynamic programming algorithm to obtain the best classification model of the current video frame, including:

[0047] A scheme graph is constructed, and nodes and edges of the scheme graph are defined, each node of the scheme graph representing a frame number and a total delay number of processing the current video frame; each edge representing a certain classification model processing the current video frame;

[0048] Set total delay budget, network bandwidth threshold and network delay threshold and create three-dimensional data for storing maximum precision corresponding to each delay, index of last model, index of last video frame and calibrated confidence score;

[0049] According to the feature vector of each pre-processed video frame and network condition, judge whether model selection needs to be adjusted, initialize DP table and perform dynamic programming filling;

[0050] Find the model index of the current video frame when the precision is maximum and the total delay is minimum, and determine the best classification model of the current video frame according to the model index.

[0051] The application provides a video processing system based on edge computing and confidence evaluation, which is used to realize the above method, comprising:

[0052] The data acquisition module acquires a video stream and divides the video stream into independent video frames;

[0053] The preprocessing module pre-processes each video frame to obtain a plurality of pre-processed video frames;

[0054] The feature vector acquisition module extracts features based on the plurality of pre-processed video frames to generate corresponding feature vectors;

[0055] The key ROI region determination module determines the key ROI region in the corresponding video frame according to the plurality of pre-processed video frames and the corresponding feature vectors;

[0056] The calibration module selects the corresponding classification model according to the corresponding feature vectors and the key ROI region, and calibrates the confidence score of the model;

[0057] The model adjustment module monitors the current network bandwidth and delay in real time, sets the network bandwidth threshold and delay threshold, and adjusts the selection of the classification model according to the network bandwidth threshold and the delay threshold;

[0058] The video frame processing module further adjusts the classification model of the current video frame by using a dynamic programming algorithm, obtains the best classification model of the current video frame, and processes the current video frame by using the best classification model.

[0059] Compared with the prior art, the technical scheme of the application has the beneficial effects that:

[0060] The present application improves the efficiency and accuracy of video processing through region of interest prediction and confidence assessment. The ROI region is identified from each frame of video, reducing redundant information and significantly reducing data processing volume. Each frame of video is preprocessed to extract feature vectors, including object class, size and position information, to support subsequent model selection. In terms of confidence assessment, the confidence score of each model is calibrated to ensure that the score accurately reflects the correctness of the classification result. According to the calibrated score, the model with high confidence and low processing delay is preferentially selected for processing; if the confidence score is low or the delay is high, the video frame is uploaded to the cloud for further processing. To further optimize the efficiency and accuracy of video processing, the present application introduces a dynamic programming algorithm to determine the processing model of each video frame by constructing a solution graph and finding the optimal path, achieving efficient and accurate video processing. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 Flowchart of the video processing method based on edge computing and confidence assessment described in Example 1;

[0062] Figure 2 Structure diagram of the video processing method based on edge computing and confidence assessment described in Example 2;

[0063] Figure 3 Structure diagram of the solution graph described in Example 2;

[0064] Figure 4 Structure diagram of the video processing system based on edge computing and confidence assessment described in Example 3. DETAILED DESCRIPTION

[0065] The drawings are only used for illustrative purposes and should not be construed as limiting the present patent;

[0066] In order to better illustrate the present embodiment, some components in the drawings may be omitted, enlarged or reduced, and do not represent the actual size of the product;

[0067] It is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0068] The technical solutions of the present application will be further described below in conjunction with the drawings and examples.

[0069] Example 1

[0070] The present embodiment provides a video processing method based on edge computing and confidence assessment, as shown in Figure 1 , comprising:

[0071] Obtain a video stream and divide the video stream into independent video frames;

[0072] preprocessing each video frame to obtain a plurality of preprocessed video frames;

[0073] performing feature extraction based on the plurality of preprocessed video frames to generate corresponding feature vectors;

[0074] determining a key ROI region in the corresponding video frame based on the plurality of preprocessed video frames and the corresponding feature vectors;

[0075] selecting a corresponding classification model according to the corresponding feature vectors and the key ROI region, and calibrating a confidence score of the model;

[0076] monitoring the current network bandwidth and delay in real time, setting network bandwidth threshold and delay threshold, and adjusting the selection of the classification model according to the network bandwidth threshold and the delay threshold;

[0077] further adjusting the classification model of the current video frame by using a dynamic programming algorithm to obtain the best classification model of the current video frame, and processing the current video frame by using the best classification model.

[0078] The embodiment improves the efficiency and accuracy of video processing through region of interest prediction and confidence assessment. The ROI region is identified from each video frame to reduce redundant information, thereby significantly reducing the data processing amount. Each video frame is preprocessed to extract feature vectors, including object category, size and position information, to support subsequent model selection. In terms of confidence assessment, the confidence score of each model is calibrated to ensure that the score accurately reflects the correctness of the classification result. According to the calibrated score, the model with high confidence and low processing delay is preferentially selected for processing; if the confidence score is low or the delay is high, the video frame is uploaded to the cloud for further processing. To further optimize the efficiency and accuracy of video processing, the dynamic programming algorithm is introduced to determine the processing model of each video frame by constructing a solution graph and finding the optimal path, thereby achieving efficient and accurate video processing.

[0079] Embodiment 2

[0080] The embodiment provides a video processing method based on edge computing and confidence assessment, comprising:

[0081] obtaining a video stream and dividing the video stream into independent video frames;

[0082] preprocessing each video frame to obtain a plurality of preprocessed video frames;

[0083] performing feature extraction based on the plurality of preprocessed video frames to generate corresponding feature vectors;

[0084] Determine a key ROI region in a corresponding video frame according to a plurality of preprocessed video frames and corresponding feature vectors;

[0085] Select a corresponding classification model according to the corresponding feature vectors and the key ROI region, and calibrate a confidence score of the model;

[0086] Real-time monitor current network bandwidth and delay, set network bandwidth threshold and delay threshold, adjust the selection of the classification model according to the network bandwidth threshold and the delay threshold;

[0087] Further adjust the classification model of the current video frame by using a dynamic programming algorithm, obtain the best classification model of the current video frame, and process the current video frame by using the best classification model.

[0088] Preprocess each video frame to obtain a plurality of preprocessed video frames, including:

[0089] Normalize each video frame to a uniform resolution to obtain a plurality of normalized video frames;

[0090] De-noise the plurality of normalized video frames to obtain a plurality of preprocessed video frames.

[0091] Determine a key ROI region in a corresponding video frame according to a plurality of preprocessed video frames and corresponding feature vectors, including:

[0092] Create a common ROI region set and a special ROI region set, wherein the common ROI region set and the special ROI region set are used to store different types of ROI regions;

[0093] Obtain an ROI region set of the plurality of preprocessed video frames, and calculate the number of objects in the ROI region set;

[0094] Obtain the class, size and position of an object in the preprocessed video frame according to the corresponding feature vector;

[0095] If the size of the object in the ROI region set of the plurality of preprocessed video frames is less than or equal to a preset size of a field of view, and the merging condition is met, the object is merged into the common ROI region set; if the merging condition is not met, a new common ROI region is created to contain the object and is merged into the common ROI region set; if the size of the object is greater than the preset size of the field of view, a special ROI region is created, the special ROI region is adjusted according to the size of the object and a preset scaling factor, the adjusted special ROI region is used to contain the object, and the object is merged into the special ROI set;

[0096] After processing all the ROI region sets of the pre-processed video frames, the merged normal ROI region and the special ROI region are merged to obtain a final ROI region set;

[0097] The weight of each ROI region in the final ROI region set is determined, and a key ROI region is selected according to a preset weight.

[0098] The formula for adjusting the special ROI region according to the object size and a preset scaling factor is:

[0099] S i ′ =y×Area(O i )

[0100] Wherein, y is a preset scaling factor; Area(O i ) is the area of the object.

[0101] According to the corresponding feature vector and the key ROI region, a corresponding classification model is selected, and the confidence score of the model is calibrated, including:

[0102] According to the corresponding feature vector and the key ROI region, a corresponding classification model is selected, and the confidence score of the model is calibrated, including:

[0103] According to the feature vector, the true label and the original confidence score of the key ROI region, a calibration data set is constructed;

[0104] The calibration data set is input into the trained calibration model to obtain the calibrated confidence score;

[0105] The calibrated confidence score is used to replace the original confidence score of the classification model.

[0106] The trained calibration model is obtained, including:

[0107] The calibration model is constructed, the calibration data set is input into the trained calibration model, and the calibrated confidence score is obtained;

[0108] The log-likelihood loss function is set, the log-likelihood loss function value is calculated according to the original confidence score and the calibrated confidence score, and the parameters of the constructed calibration model are adjusted;

[0109] When the log-likelihood loss function value reaches a preset value or reaches a preset training number of times, the trained calibration model is obtained.

[0110] The log-likelihood loss function is:

[0111]

[0112] wherein y i is the true label; is the calibrated confidence score; p i is the original confidence score.

[0113] The constructed calibration model is adjusted in parameters by using the gradient descent method, specifically:

[0114] The loss function is minimized by using the gradient descent method, and the formula is:

[0115]

[0116] wherein y i is the true label; is the calibrated confidence score; p i is the original confidence score; is the gradient of the loss function of the first model parameter; is the gradient of the loss function of the second model parameter.

[0117] The model parameters are updated by the gradient descent, and the formula is:

[0118]

[0119] wherein A is the first model parameter; B is the second model parameter; η is the learning rate, which controls the step size of parameter update in each iteration.

[0120] The classification model of the current video frame is further adjusted by using the dynamic programming algorithm to obtain the best classification model of the current video frame, including:

[0121] A scheme graph is constructed, and the nodes and edges of the scheme graph are defined, wherein each node of the scheme graph represents the frame number and the total delay number of processing the current video frame; and each edge represents that a certain classification model processes the current video frame.

[0122] The total delay budget, the network bandwidth threshold and the network delay threshold are set and three-dimensional data is created, wherein the three-dimensional data is used to store the maximum accuracy corresponding to each delay, the index of the last model, the index of the last video frame and the calibrated confidence score;

[0123] According to the feature vector of each preprocessed video frame and the network condition, it is judged whether the model selection needs to be adjusted, the DP table is initialized and the dynamic programming filling is performed.

[0124] The model index of the current video frame when the accuracy is maximum and the total delay number is minimum is found, and the best classification model of the current video frame is determined according to the model index.

[0125] In one specific embodiment, as Figure 2The system is designed to efficiently process two-dimensional videos on mobile devices, incorporating Region of Interest Prediction (RoI Prediction) technology and confidence assessment methods to effectively address the challenges of limited computing resources and unstable network conditions on edge devices. Here are the specific details:

[0126] S1: Video frame preprocessing and feature extraction: The input two-dimensional video stream is segmented into independent video frames, each of which undergoes preprocessing steps including normalization of resolution, noise removal, and other preprocessing steps to ensure data consistency and quality. Feature extraction is performed on each video frame to generate a feature vector.

[0127] S2: Region of Interest Prediction: Based on the results of preprocessing and feature extraction, a lightweight Region of Interest Prediction algorithm is applied to identify key regions of interest (RoI) in the video frames.

[0128] S3: Confidence Assessment and Calibration: For each predicted RoI, an appropriate classification model is selected based on its feature vector, and the Platt Calibration technique is used to calibrate the confidence score of the model.

[0129] S4: Network Condition Monitoring and Dynamic Adjustment: Real-time monitoring of network bandwidth and delay, dynamic adjustment of model selection and processing strategy. If network conditions deteriorate, simplify model selection and prioritize low-latency local processing.

[0130] S5: Dynamic Programming Algorithm Optimized Path Selection: Initialize parameters including total delay budget, network bandwidth threshold, and network delay threshold. Create a three-dimensional array to store the maximum precision of each delay, the last model index, the last frame index, and the corrected confidence score. When processing each video frame, obtain the feature vector of the current video frame and the network condition, determine whether to simplify model selection, and perform dynamic programming filling. Real-time backtracking finds the optimal path to determine the best model for the current frame and records and outputs the results.

[0131] Further, in step S1, video frame preprocessing and feature extraction.

[0132] S1.1: First, preprocess the input video frame. Let the input video frame be I, normalize it to a uniform resolution, denoted as I norm .

[0133] S1.2: Next, perform noise removal on the normalized video frame to reduce noise interference in subsequent steps, obtaining the preprocessed video frame I pre .

[0134] S1.3: Then, use a Convolutional Neural Network (CNN) to process the preprocessed video frame I preFeature extraction is performed. Let f be the feature extraction function of the CNN CNN The feature vector F can be expressed as:

[0135] F = f cnn (I pre )

[0136] The feature vector F contains the key information of the object in the video frame, such as the class, size, and position, which provides a basis for the RoI prediction in the subsequent steps.

[0137] Further, in the RoI prediction and post-processing in step S2, the prediction of the RoI (Region of Interest) is performed.

[0138] S2.1: First, initialize two RoI sets: the normal RoI set S and the special RoI set S ′ , which are used to store different types of interest regions.

[0139] S2.2: Then, obtain the historical detection result set O and calculate the number N of objects in it. For each object o i (i = 1, 2, …, N) in the historical detection result set O, perform the following operations:

[0140] Check whether the object O i can be covered by the field of view (FoV) of f x f. If it can be covered, try to merge it into an existing RoI. The merging rule can be set according to the actual situation, such as the FoV of the merged RoI does not exceed the preset threshold. If the merging fails, create a new normal RoI to contain the object.

[0141] S2.3: If the size of the object O i exceeds the FoV of f x f, create a special RoI S i ′ . The area of the special RoI can be adjusted according to the size of the object and the preset scaling factor y, i.e.:

[0142] S i ′ = y x Area(O i )

[0143] where Area(O i ) represents the area of the object O i . After predicting all the RoIs, merge the normal RoI set S and the special RoI set S ′ into the final RoI set R. Finally, perform post-processing on the RoI set R. Use the non-maximum suppression (NMS) algorithm to remove overlapping RoIs to ensure that each RoI is independent. Let the overlap degree of two RoIs be IOU(Si S j ), if IOU(S i ,S j )>θ (θ is a preset threshold), the RoI with larger weight is retained and the RoI with smaller weight is removed. At the same time, it is checked whether the boundary of each RoI exceeds the range of the video frame, and if it exceeds, it is adjusted appropriately. After the above steps, the processed RoI set R ′ is obtained as the output of the algorithm. The set contains the key interest regions in the video frame. The position information of each RoI is extracted from the RoI set R, including the center position (X c, Y c ) and the field of view angle (w, h). The bounding box coordinates are calculated: the center position and the field of view angle are converted into the coordinates of the upper left corner and the lower right corner of the bounding box (x1, y1, x2, y2). Format conversion: the bounding box coordinates are converted into a format suitable for processing by the existing model.

[0144] Further, in step S3, the confidence evaluation is performed.

[0145] S3.1: In order to achieve confidence evaluation and calibration, first, a data set containing feature vectors, true labels and original confidence scores is prepared for training and testing x i represents the input feature vector, y i represents the true label, and p i represents the original confidence score output by the model. For each region of interest (RoI), the feature vector x i is extracted from each RoI, the model is used to predict each RoI to obtain the original confidence score p i , and the feature vector x i , the true label y i and the original confidence score p i of each RoI are recorded. Then, the Platt Calibration method is used to calibrate the confidence score. Platt Calibration is a method based on logistic regression, which converts the original confidence score into calibrated probability by fitting a mapping function. Assuming that the original confidence score is p i , the calibrated confidence score is

[0146] S3.2: Platt Calibration is calibrated by the following formula:

[0147]

[0148] where A and B are parameters that need to be fitted by training data.

[0149] S3.3: Train the Platt Calibration model using the training dataset D. The specific steps are as follows:

[0150] Define the loss function: Use the log-likelihood loss function to optimize parameters A and B

[0151]

[0152] Optimize parameters: Use gradient descent or other optimization algorithms to minimize the loss function L(A, B)

[0153]

[0154] Update parameters: Update parameters A and B through gradient descent

[0155]

[0156] where η is the learning rate.

[0157] Evaluate the calibration effect: Evaluate the performance of the calibrated model using the test dataset. Use the Brier Score index to evaluate the calibration effect, which measures the squared difference between the predicted probability and the actual label.

[0158] Brier Score formula:

[0159] S3.4: Performance evaluation of the model, using the prepared dataset Specifically, we list all the models to be evaluated M1, M2,..., M k . Record its accuracy and delay under different feature vectors. Extract the feature vector X i from the dataset, use each model M j to predict the feature vector X i , record the prediction result of the model and the prediction time t i . Then, calculate the accuracy of each model under different feature vectors, which can be measured by accuracy (Accuracy), precision (Precision), recall (Recall) and F1 score, etc., and record the prediction time t iNext, we establish a model performance matrix to store the performance indicators of each model. We initialize a matrix P, where each row corresponds to a model and each column corresponds to a performance indicator (such as accuracy, latency, etc.). We record the accuracy and latency of each model under different feature vectors into the performance matrix. Through these steps, we can effectively record and evaluate the performance of each model, ensuring that the performance indicators of the models can be accurately recorded and compared. This also provides reliable data support for subsequent confidence calibration and model selection.

[0160] Further, in step S4, the current network bandwidth and latency are monitored in real time. Thresholds for network bandwidth and latency are set to determine whether the current network conditions are suitable for model selection adjustment. It is assumed that when the network bandwidth is below a certain value or the latency is above a certain value, a simpler model is selected or the amount of data transmission is reduced. Suppose the threshold values are B threshold and L threshold .

[0161] Further, in step S5, a dynamic programming algorithm is used.

[0162] S5.1: Before that, a scheme graph is constructed. As shown in Figure 3 , nodes and edges are defined. Nodes: Each node V i (j, t) represents a state, i.e., a state of processing the jth video frame and a total latency of t milliseconds. Edges: Each edge represents an operation of transitioning from one state to another, i.e., selecting a model to process the current video frame. The graph is initialized. Start point: V start represents the initial state, i.e., no video frame has been processed and the total latency is 0 milliseconds. End point: V end Since it is stream processing, we do not have a fixed end point, but rather continuously update the optimal path. Intermediate nodes: V(j, t) represent a state of processing the jth video frame and a total latency of t milliseconds. The graph is constructed and processed online. For each video frame j, when it arrives: For each possible latency t, add an edge from node V(j-1, t) to node V(j, t+latency), where latency is the latency of selecting a model to process the current video frame. Immediately calculate and update the best path so that a decision is ready before the next frame arrives.

[0163] S5.2: Dynamic programming algorithm for video stream processing. First, input: Feature vector list F: Feature vector of each video frame. Model performance matrix M: Accuracy and latency of each model under different feature vectors. Network status N: Including current network bandwidth B, network latency L, and total latency budget T budget . Network bandwidth threshold B threshold : When the network bandwidth is below this value, a simpler model is selected. Network latency threshold L thresholdWhen network latency exceeds this value, choose the simpler model. Initialize parameters: Total latency budget T budget Network bandwidth B, network bandwidth threshold B threshold Network latency threshold L threshold Create a three-dimensional array dp to store the maximum precision for each delay t, the previous model index, the previous frame index, and the calibrated confidence score.

[0164] S5.3: Process each video frame j. For each frame, obtain the feature vector (feature_vector) containing key information such as object category, size, and location. Obtain the current network status (B and L). Based on the current network status, determine whether to select a simpler model set (M). simplified If the current network bandwidth B is lower than the bandwidth threshold B... threshold Or the network latency L is higher than the latency threshold L. threshold Then choose the simpler model set M. simplified Otherwise, use all available models M. Create a new dp table new_dp, initialized to default values, so that it can be updated when processing the next frame. Dynamically populate the table, iterating through every possible delay t and every available model m. i This involves checking whether the model supports the feature vectors of the current frame, and calculating the new latency, new accuracy, and calibrated confidence score. The formula is as follows:

[0165] calibrated_confident=calibrate_confidence(m i [raw_confidence])new_accuracy=dp[t][o]+accuracy

[0166] If new_accuracy > new_dp[t+latency][0]:

[0167] new_dp[t+latency]=[new_accuracy,m i [j-1]

[0168] For each possible delay t and each available model m i If model m i Supports the feature vectors of the current frame and the new delay does not exceed the total delay budget T. budgetMeanwhile, if the network bandwidth is sufficient, calculate the new accuracy new_accuracy and the calibrated confidence score calibrated_confidence. Update the DP table new_dp from the DP table dp for use in processing the next frame. Find the maximum accuracy and the corresponding latency for the current state, and determine the optimal model for the current frame.

[0169] best_accuracy = max t (dp[t][o]), the optimal latency best_latency = argmax t (dp[t][o]), the optimal calibrated confidence best_calibrated_confidence = dp[best_latency][3] Traverse all latencies t to find the maximum accuracy best_accuracy and the corresponding latency best_latency. Determine the optimal model optimal_model and the optimal calibrated confidence score best_calibrated_confidence based on the found maximum accuracy. Process the current frame j using the optimal model optimal_model. Record the optimal model for the current frame for subsequent analysis or output. Output the optimal model, the maximum accuracy, and the optimal latency calibrated confidence score for the current frame for debugging and logging. This dynamic programming algorithm realizes how to select the optimal model in video stream processing to maximize processing time, taking into account the impact of network conditions (bandwidth and latency) and directly using the corrected confidence score. Specifically:

[0170] Initialize parameters: set the total latency budget, network bandwidth threshold, and network latency threshold.

[0171] Initialize DP table: create a three-dimensional array to store the maximum accuracy, last model index, last frame index, and corrected confidence score for each latency.

[0172] Process each frame of video: obtain the feature vector and network conditions, determine whether to simplify model selection, initialize a new DP table, and perform dynamic programming filling.

[0173] Find the optimal path in real time: find the maximum accuracy and the corresponding latency for the current state, determine the optimal model for the current frame, and record and output the results.

[0174] Embodiment 3

[0175] This embodiment also provides a video processing system based on edge computing and confidence evaluation, which is used to implement the method described in Embodiment 1 or Embodiment 2, as shown in Figure 4 , comprising:

[0176] a data acquisition module, acquiring a video stream and dividing the video stream into independent video frames;

[0177] a preprocessing module, preprocessing each video frame to obtain a plurality of preprocessed video frames;

[0178] a feature vector acquisition module, performing feature extraction based on the plurality of preprocessed video frames to generate corresponding feature vectors;

[0179] a key ROI region determination module, determining a key ROI region in a corresponding video frame according to the plurality of preprocessed video frames and the corresponding feature vectors;

[0180] a calibration module, selecting a corresponding classification model according to the corresponding feature vectors and the key ROI region, and calibrating a confidence score of the model;

[0181] a model adjustment module, monitoring a current network bandwidth and delay in real time, setting a network bandwidth threshold and a delay threshold, and adjusting the selection of the classification model according to the network bandwidth threshold and the delay threshold;

[0182] a video frame processing module, further adjusting the classification model of a current video frame by using a dynamic programming algorithm, obtaining a best classification model of the current video frame, and processing the current video frame by using the best classification model.

[0183] The same or similar reference numerals correspond to the same or similar components;

[0184] The terms describing the positional relationship in the drawings are only used for illustrative description, and should not be understood as a limitation on the patent;

[0185] Obviously, the above embodiments of the present application are only examples for clearly illustrating the present application, and are not intended to limit the implementation modes of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all the implementation modes are not required or can not be exhausted. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the claims of the present application.

Claims

1. A method for video processing based on edge computing and confidence evaluation, characterized in that, The method comprises the following steps: acquiring a video stream and dividing the video stream into independent video frames; preprocessing each video frame to obtain a plurality of preprocessed video frames; extracting features based on the plurality of preprocessed video frames to generate corresponding feature vectors; determining a key ROI region in a corresponding video frame according to the plurality of preprocessed video frames and the corresponding feature vectors; selecting a corresponding classification model according to the corresponding feature vectors and the key ROI region, and calibrating a confidence score of the model; monitoring a current network bandwidth and delay in real time, setting a network bandwidth threshold and a delay threshold, and adjusting the selection of the classification model according to the network bandwidth threshold and the delay threshold; further adjusting the classification model of the current video frame by using a dynamic programming algorithm to obtain an optimal classification model of the current video frame, and processing the current video frame by using the optimal classification model; determining a key ROI region in a corresponding video frame according to a plurality of preprocessed video frames and corresponding feature vectors, comprising: creating a common ROI region set and a special ROI region set, wherein the common ROI region set and the special ROI region set are used to store different types of ROI regions; acquiring a ROI region set of a plurality of preprocessed video frames, and calculating the number of objects in the ROI region set; obtaining the category, size and position of an object in a preprocessed video frame according to a corresponding feature vector; if the size of the object in the ROI region set of the plurality of preprocessed video frames is less than or equal to a preset size of a field of view, and a merging condition is met, the object is merged into the common ROI region set; if the merging condition is not met, a new common ROI region is created to contain the object and is merged into the common ROI region set; if the size of the object is greater than the preset size of the field of view, a special ROI region is created, the special ROI region is adjusted according to the size of the object and a preset scaling factor, the adjusted special ROI region is used to contain the object, and the object is merged into the special ROI set; after processing the ROI region set of all preprocessed video frames, the merged common ROI region and the special ROI region are merged to obtain a final ROI region set; determining the weight of each ROI region in the final ROI region set, and screening a key ROI region according to a preset weight.

2. The edge computing and confidence evaluation based video processing method of claim 1, wherein, preprocessing each video frame to obtain a plurality of preprocessed video frames, comprising: normalizing each video frame to a uniform resolution to obtain a plurality of normalized video frames; performing denoising processing on the plurality of normalized video frames to obtain a plurality of preprocessed video frames. 3.The edge computing and confidence evaluation based video processing method of claim 1, wherein, The formula for adjusting the special ROI region according to the size of the object and the preset scaling factor is: S i ′ = y x Area (O i ) where y is a preset scaling factor; Area(O i ) is the area of the object. 4.The edge computing and confidence evaluation based video processing method of claim 1, wherein, selecting a corresponding classification model according to the corresponding feature vectors and the key ROI region, and calibrating a confidence score of the model, comprising: selecting a corresponding classification model according to the corresponding feature vectors and the key ROI region, and obtaining an original confidence score of the key ROI region by using the model; constructing a calibration data set according to the feature vectors, the real label and the original confidence score of the key ROI region; Input the calibration dataset into the trained calibration model to obtain a calibrated confidence score; Replace the original confidence score of the classification model with the calibrated confidence score.

5. The edge computing and confidence evaluation based video processing method of claim 4, wherein, Obtaining the trained calibration model comprises: Building a calibration model, inputting the calibration dataset into the trained calibration model to obtain a calibrated confidence score; Setting a log-likelihood loss function, calculating the log-likelihood loss function value according to the original confidence score and the calibrated confidence score, and adjusting the parameters of the built calibration model; When the log-likelihood loss function value reaches a preset value or reaches a preset training number of times, the trained calibration model is obtained.

6. The edge computing and confidence evaluation based video processing method of claim 5, wherein, The log-likelihood loss function is: where y i is the true label; is the calibrated confidence score; p i is the original confidence score.

7. The edge computing and confidence evaluation based video processing method of claim 5, wherein, Adjusting the parameters of the built calibration model using the gradient descent method, specifically: Minimizing the loss function using the gradient descent method, the formula is: where y i is the true label; is the calibrated confidence score; p i is the original confidence score; is the gradient of the loss function for the first model parameters; is the gradient of the loss function for the second model parameters; Updating the model parameters by gradient descent. 8.The edge computing and confidence evaluation based video processing method of claim 1, wherein, Further adjusting the classification model of the current video frame using the dynamic programming algorithm to obtain the best classification model of the current video frame, comprising: Building a scheme graph, defining the nodes and edges of the scheme graph, each node of the scheme graph representing the frame number and total delay number of processing the current video frame; each edge represents a certain classification model processing the current video frame; Setting a total delay budget, a network bandwidth threshold and a network delay threshold and creating three-dimensional data, the three-dimensional data being used to store the maximum accuracy corresponding to each delay, the index of the last model, the index of the last video frame and the calibrated confidence score; According to the feature vector of each preprocessed video frame and the network condition, it is judged whether the model selection needs to be adjusted, the DP table is initialized and dynamic programming filling is performed; Finding the model index when the accuracy of the current video frame is maximum and the total delay number is minimum, and determining the best classification model of the current video frame according to the model index.

9. A video processing system based on edge computing and confidence evaluation, configured to implement the method of any one of claims 1-8, characterized in that, Comprise: A data acquisition module acquires a video stream and divides the video stream into independent video frames; A preprocessing module pre-processes each video frame to obtain a plurality of pre-processed video frames; A feature vector acquisition module extracts features based on the plurality of pre-processed video frames to generate corresponding feature vectors; A key ROI region determination module determines the key ROI region in the corresponding video frame according to the plurality of pre-processed video frames and the corresponding feature vectors; A calibration module selects a corresponding classification model according to the corresponding feature vectors and the key ROI region, and calibrates the confidence score of the model; A model adjustment module monitors the current network bandwidth and delay in real time, sets a network bandwidth threshold and a delay threshold, and adjusts the selection of the classification model according to the network bandwidth threshold and the delay threshold; A video frame processing module further adjusts the classification model of the current video frame using the dynamic programming algorithm to obtain the best classification model of the current video frame, and processes the current video frame using the best classification model.

Images