Video processing method and system based on edge calculation and confidence evaluation

Abstract

According to the invention, through region-of-interest prediction and confidence evaluation, the video processing efficiency and precision are improved. The ROI is identified from each frame of video, redundant information is reduced, and therefore the data processing amount is remarkably reduced. Each frame of video is preprocessed, feature vectors are extracted, the feature vectors comprise information such as object types, sizes and positions, and support is provided for subsequent model selection. In the aspect of confidence evaluation, the confidence score of each model is calibrated to ensure that the score accurately reflects the correctness of the classification result. According to the calibrated score, a model with high confidence and low processing delay is preferentially selected for processing; and if the confidence score is low or the delay is high, considering to upload the video frame to the cloud for further processing. In order to further optimize the video processing efficiency and precision, a dynamic programming algorithm is introduced, a processing model of each video frame is determined by constructing a solution graph and searching an optimal path, and high efficiency and precision of video processing are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and edge computing, and more specifically, to a video processing method and system based on edge computing and confidence assessment. Background Art

[0002] With the rapid development of mobile devices and edge computing technology, more and more applications require efficient video processing on mobile devices. These applications include but are not limited to intelligent monitoring, drone navigation, virtual reality (VR) and augmented reality (AR). Traditional video processing methods usually rely on central servers to upload video data to the cloud for processing. Although this method can utilize powerful computing resources, it has significant limitations. First, high network latency is one of the main problems, especially in wireless networks or unstable network conditions, the upload and download process of video data will cause significant delays. Secondly, the transmission of large amounts of video data requires a lot of network bandwidth, which increases transmission costs and may cause network congestion.

[0003] In order to overcome the limitations of traditional methods, mobile devices and edge computing technology have gradually become research hotspots. Mobile devices are characterized by portability and low power consumption, while edge computing can perform calculations close to the data source, reducing the delay and cost of data transmission. Specifically, edge computing can perform processing close to the data source, significantly reducing the time of data transmission, thereby reducing processing delays. By performing preliminary processing on edge devices, the amount of data that needs to be transmitted can be reduced, reducing transmission costs. At the same time, edge computing can utilize the local computing resources of mobile devices, reduce the energy consumption caused by data transmission and remote processing, and improve energy efficiency. Although mobile devices and edge computing technology have many advantages, they still face some challenges in practical applications. For example, the computing resources of mobile devices are limited and cannot run complex deep learning models; the network conditions of edge devices are unstable, affecting the reliability and efficiency of data transmission. Summary of the invention

[0004] In order to overcome the defects of low efficiency and low precision of video processing in the prior art, the present invention provides a video processing method and system based on edge computing and confidence assessment, which is used to efficiently process two-dimensional video on a mobile device, improve processing accuracy and meet delay constraints.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] The present invention provides a video processing method based on edge computing and confidence evaluation, comprising:

[0007] Get the video stream and split it into independent video frames;

[0008] Preprocess each video frame to obtain a number of preprocessed video frames;

[0009] Perform feature extraction based on several preprocessed video frames to generate corresponding feature vectors;

[0010] Determine a key ROI region in a corresponding video frame according to a plurality of preprocessed video frames and corresponding feature vectors;

[0011] Selecting a corresponding classification model according to the corresponding feature vector and key ROI area, and calibrating the confidence score of the model;

[0012] Monitor the current network bandwidth and delay in real time, set the network bandwidth threshold and delay threshold, and adjust the selection of the classification model according to the network bandwidth threshold and delay threshold;

[0013] The classification model of the current video frame is further adjusted using a dynamic programming algorithm to obtain an optimal classification model for the current video frame, and the current video frame is processed using the optimal classification model.

[0014] Preferably, each video frame is preprocessed to obtain a plurality of preprocessed video frames, including:

[0015] Normalize each video frame to a uniform resolution to obtain a number of normalized video frames;

[0016] De-noising is performed on a number of normalized video frames to obtain a number of pre-processed video frames.

[0017] Preferably, determining a key ROI region in a corresponding video frame according to a plurality of preprocessed video frames and corresponding feature vectors includes:

[0018] Creating a common ROI region set and a special ROI region set, wherein the common ROI region set and the special ROI region set are used to store different types of ROI regions;

[0019] Obtaining a plurality of ROI region sets of preprocessed video frames, and calculating the number of objects in the ROI region sets;

[0020] Obtain the category, size and position of the object in the preprocessed video frame according to the corresponding feature vector;

[0021] If the size of the object in the ROI area set of the pre-processed video frames is less than or equal to the field of view of the preset size and meets the merging condition, the object is merged into the common ROI area set; if the merging condition is not met, a new common interest area is created to include the object and merged into the common interest area set; if the size of the object is larger than the field of view of the preset size, a special ROI area is created, the special ROI area is adjusted according to the object size and the preset scaling factor, the object is included in the adjusted special ROI area, and merged into the special ROI set;

[0022] After processing all the ROI region sets of the preprocessed video frames, the merged common ROI regions and special ROI regions are merged to obtain a final ROI region set;

[0023] The weight of each ROI region in the final ROI region set is determined, and key ROI regions are screened out according to the preset weights.

[0024] Preferably, the formula for adjusting the special ROI area according to the object size and the preset scaling factor is:

[0025] S′ i =y×Area(O i )

[0026] Where y is the preset scaling factor; Area(O i ) is the area of ​​the object.

[0027] Preferably, selecting a corresponding classification model according to the corresponding feature vector and the key ROI area, and calibrating the confidence score of the model, includes:

[0028] Selecting a corresponding classification model according to the corresponding feature vector and the key ROI area, and obtaining an original confidence score corresponding to the key ROI area through the model;

[0029] Construct a calibration dataset based on feature vectors, true labels, and original confidence scores of key ROI regions;

[0030] Input the calibration data set into the trained calibration model to obtain the confidence score after calibration;

[0031] The original confidence scores of the classification model are replaced with the calibrated confidence scores.

[0032] Preferably, obtaining a trained calibration model includes:

[0033] Build a calibration model, input the calibration data set into the trained calibration model, and obtain the confidence score after calibration;

[0034] Set the log-likelihood loss function, calculate the log-likelihood loss function value according to the original confidence score and the calibrated confidence score, and adjust the parameters of the constructed calibration model;

[0035] When the log-likelihood loss function value reaches a preset value or reaches a preset number of training times, a trained calibration model is obtained.

[0036] Preferably, the log-likelihood loss function is:

[0037]

[0038] Among them, yi is the true label; is the calibrated confidence score; pi is the original confidence score.

[0039] Preferably, the constructed calibration model is adjusted by using the gradient descent method, specifically:

[0040] The loss function is minimized using the gradient descent method, and the formula is:

[0041]

[0042] Among them, yi is the true label; is the confidence score after calibration; pi is the original confidence score; is the gradient of the loss function of the first model parameter; is the gradient of the loss function with respect to the second model parameter.

[0043] The model parameters are updated by gradient descent, and the formula is:

[0044]

[0045] Among them, A is the first model parameter; B is the second model parameter; η is the learning rate, which controls the step size of parameter update in each iteration.

[0046] Preferably, the classification model of the current video frame is further adjusted using a dynamic programming algorithm to obtain an optimal classification model of the current video frame, including:

[0047] Constructing a scheme graph, defining nodes and edges of the scheme graph, wherein each node of the scheme graph represents the number of frames and the total delay number of the processed current video frame; and each edge represents that a certain classification model processes the current video frame;

[0048] Setting a total delay budget, a network bandwidth threshold, and a network delay threshold and creating three-dimensional data, wherein the three-dimensional data is used to store a maximum accuracy corresponding to each delay, an index of a previous model, an index of a previous video frame, and a calibrated confidence score;

[0049] According to the feature vector and network status of each preprocessed video frame, determine whether the model selection needs to be adjusted, initialize the DP table and perform dynamic programming filling;

[0050] Find the model index when the accuracy of the current video frame is the highest and the total delay is the lowest, and determine the best classification model for the current video frame based on the model index.

[0051] The present invention provides a video processing system based on edge computing and confidence assessment, which is used to implement the above method, including:

[0052] A data acquisition module acquires a video stream and divides the video stream into independent video frames;

[0053] A preprocessing module preprocesses each video frame to obtain a number of preprocessed video frames;

[0054] A feature vector acquisition module performs feature extraction based on a number of preprocessed video frames to generate corresponding feature vectors;

[0055] A key ROI region determination module determines a key ROI region in a corresponding video frame according to a number of pre-processed video frames and corresponding feature vectors;

[0056] A calibration module, selecting a corresponding classification model according to the corresponding feature vector and key ROI area, and calibrating the confidence score of the model;

[0057] The model adjustment module monitors the current network bandwidth and delay in real time, sets the network bandwidth threshold and delay threshold, and adjusts the selection of the classification model according to the network bandwidth threshold and delay threshold;

[0058] The video frame processing module further adjusts the classification model of the current video frame using a dynamic programming algorithm to obtain an optimal classification model for the current video frame, and processes the current video frame using the optimal classification model.

[0059] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0060] The present invention improves the efficiency and accuracy of video processing through region of interest prediction and confidence assessment. The ROI region is identified from each frame of video to reduce redundant information, thereby significantly reducing the amount of data processing. Each frame of video is preprocessed to extract feature vectors, including information such as object category, size and position, to provide support for subsequent model selection. In terms of confidence assessment, the confidence score of each model is calibrated to ensure that the score accurately reflects the correctness of the classification result. According to the calibrated score, models with high confidence and low processing delay are preferentially selected for processing; if the confidence score is low or the delay is high, consider uploading the video frame to the cloud for further processing. In order to further optimize the efficiency and accuracy of video processing, the present invention introduces a dynamic programming algorithm, which determines the processing model of each video frame by constructing a solution graph and finding the optimal path, thereby achieving efficient and accurate video processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flowchart of the video processing method based on edge computing and confidence assessment described in Example 1;

[0062] Figure 2 This is a schematic diagram of the structure of the video processing method based on edge computing and confidence assessment described in Example 2;

[0063] Figure 3 It is a schematic diagram of the structure of the scheme diagram described in Example 2;

[0064] Figure 4 This is a schematic diagram of the structure of the video processing system based on edge computing and confidence assessment described in Example 3. DETAILED DESCRIPTION

[0065] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0066] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0067] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0068] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0069] Example 1

[0070] This embodiment provides a video processing method based on edge computing and confidence evaluation, such as Figure 1 As shown, including:

[0071] Get the video stream and split it into independent video frames;

[0072] Preprocess each video frame to obtain a number of preprocessed video frames;

[0073] Perform feature extraction based on several preprocessed video frames to generate corresponding feature vectors;

[0074] Determine a key ROI region in a corresponding video frame according to a plurality of preprocessed video frames and corresponding feature vectors;

[0075] Selecting a corresponding classification model according to the corresponding feature vector and key ROI area, and calibrating the confidence score of the model;

[0076] Monitor the current network bandwidth and delay in real time, set the network bandwidth threshold and delay threshold, and adjust the selection of the classification model according to the network bandwidth threshold and delay threshold;

[0077] The classification model of the current video frame is further adjusted using a dynamic programming algorithm to obtain an optimal classification model for the current video frame, and the current video frame is processed using the optimal classification model.

[0078] This embodiment improves the efficiency and accuracy of video processing by predicting regions of interest and evaluating confidence. The ROI region is identified from each frame of video to reduce redundant information, thereby significantly reducing the amount of data processing. Each frame of video is preprocessed to extract feature vectors, including information such as object category, size, and position, to provide support for subsequent model selection. In terms of confidence assessment, the confidence score of each model is calibrated to ensure that the score accurately reflects the correctness of the classification result. According to the calibrated score, models with high confidence and low processing delay are preferentially selected for processing; if the confidence score is low or the delay is high, consider uploading the video frame to the cloud for further processing. In order to further optimize the efficiency and accuracy of video processing, the present invention introduces a dynamic programming algorithm, which determines the processing model of each video frame by constructing a solution graph and finding the optimal path, thereby achieving efficient and accurate video processing.

[0079] Example 2

[0080] This embodiment provides a video processing method based on edge computing and confidence evaluation, including:

[0081] Get the video stream and split it into independent video frames;

[0082] Preprocess each video frame to obtain a number of preprocessed video frames;

[0083] Perform feature extraction based on several preprocessed video frames to generate corresponding feature vectors;

[0084] Determine a key ROI region in a corresponding video frame according to a plurality of preprocessed video frames and corresponding feature vectors;

[0085] Selecting a corresponding classification model according to the corresponding feature vector and key ROI area, and calibrating the confidence score of the model;

[0086] Monitor the current network bandwidth and delay in real time, set the network bandwidth threshold and delay threshold, and adjust the selection of the classification model according to the network bandwidth threshold and delay threshold;

[0087] The classification model of the current video frame is further adjusted using a dynamic programming algorithm to obtain an optimal classification model for the current video frame, and the current video frame is processed using the optimal classification model.

[0088] Each video frame is preprocessed to obtain several preprocessed video frames, including:

[0089] Normalize each video frame to a uniform resolution to obtain a number of normalized video frames;

[0090] De-noising is performed on a number of normalized video frames to obtain a number of pre-processed video frames.

[0091] According to a number of pre-processed video frames and corresponding feature vectors, a key ROI region in a corresponding video frame is determined, including:

[0092] Creating a common ROI region set and a special ROI region set, wherein the common ROI region set and the special ROI region set are used to store different types of ROI regions;

[0093] Obtaining a plurality of ROI region sets of preprocessed video frames, and calculating the number of objects in the ROI region sets;

[0094] Obtain the category, size and position of the object in the preprocessed video frame according to the corresponding feature vector;

[0095] If the size of the object in the ROI area set of the pre-processed video frames is less than or equal to the field of view of the preset size and meets the merging condition, the object is merged into the common ROI area set; if the merging condition is not met, a new common interest area is created to include the object and merged into the common interest area set; if the size of the object is larger than the field of view of the preset size, a special ROI area is created, the special ROI area is adjusted according to the object size and the preset scaling factor, the object is included in the adjusted special ROI area, and merged into the special ROI set;

[0096] After processing all the ROI region sets of the preprocessed video frames, the merged common ROI regions and special ROI regions are merged to obtain a final ROI region set;

[0097] The weight of each ROI region in the final ROI region set is determined, and key ROI regions are screened out according to the preset weights.

[0098] The formula for adjusting the special ROI area according to the object size and the preset scaling factor is:

[0099] S i i =y×Area(O i )

[0100] Where y is the preset scaling factor; Area(O i ) is the area of ​​the object.

[0101] Select a corresponding classification model according to the corresponding feature vector and key ROI area, and calibrate the confidence score of the model, including:

[0102] Selecting a corresponding classification model according to the corresponding feature vector and the key ROI area, and obtaining an original confidence score corresponding to the key ROI area through the model;

[0103] Construct a calibration dataset based on feature vectors, true labels, and original confidence scores of key ROI regions;

[0104] Input the calibration data set into the trained calibration model to obtain the confidence score after calibration;

[0105] The original confidence scores of the classification model are replaced with the calibrated confidence scores.

[0106] Get the trained calibration model, including:

[0107] Build a calibration model, input the calibration data set into the trained calibration model, and obtain the confidence score after calibration;

[0108] Set the log-likelihood loss function, calculate the log-likelihood loss function value according to the original confidence score and the calibrated confidence score, and adjust the parameters of the constructed calibration model;

[0109] When the log-likelihood loss function value reaches a preset value or reaches a preset number of training times, a trained calibration model is obtained.

[0110] The log-likelihood loss function is:

[0111]

[0112] Among them, y i is the true label; is the confidence score after calibration; p i Score the raw confidence.

[0113] The gradient descent method is used to adjust the parameters of the constructed calibration model, specifically:

[0114] The loss function is minimized using the gradient descent method, and the formula is:

[0115]

[0116] Among them, y i is the true label; is the confidence score after calibration; p i Score the raw confidence; is the gradient of the loss function of the first model parameter; is the gradient of the loss function with respect to the second model parameter.

[0117] The model parameters are updated by gradient descent, and the formula is:

[0118]

[0119] Among them, A is the first model parameter; B is the second model parameter; η is the learning rate, which controls the step size of parameter update in each iteration.

[0120] The classification model of the current video frame is further adjusted using a dynamic programming algorithm to obtain the optimal classification model of the current video frame, including:

[0121] Constructing a scheme graph, defining nodes and edges of the scheme graph, wherein each node of the scheme graph represents the number of frames and the total delay number of the processed current video frame; and each edge represents that a certain classification model processes the current video frame;

[0122] Setting a total delay budget, a network bandwidth threshold, and a network delay threshold and creating three-dimensional data, wherein the three-dimensional data is used to store a maximum accuracy corresponding to each delay, an index of a previous model, an index of a previous video frame, and a calibrated confidence score;

[0123] According to the feature vector and network status of each preprocessed video frame, determine whether the model selection needs to be adjusted, initialize the DP table and perform dynamic programming filling;

[0124] Find the model index when the accuracy of the current video frame is the highest and the total delay is the lowest, and determine the best classification model for the current video frame based on the model index.

[0125] In a specific embodiment, Figure 2As shown in the figure, it is designed for efficient processing of two-dimensional videos on mobile devices, integrating the region of interest prediction (RoIPrediction) technology and confidence assessment methods, effectively addressing the challenges of limited computing resources on mobile devices and unstable network conditions of edge devices. The following are the specific contents:

[0126] S1: Video frame preprocessing and feature extraction: The input two-dimensional video stream is segmented into independent video frames, and each video frame undergoes preprocessing steps, including normalized resolution, denoising, etc., to ensure data consistency and quality. Feature extraction is performed on each frame of video to generate a feature vector.

[0127] S2: Region of Interest Prediction: Based on the results of preprocessing and feature extraction, a lightweight region of interest prediction algorithm is applied to identify key regions of interest (RoIs) in video frames.

[0128] S3: Confidence evaluation and calibration: For each predicted RoI, an appropriate classification model is selected based on its feature vector, and the confidence score of the model is calibrated using the Platt Calibration technique.

[0129] S4: Network condition monitoring and dynamic adjustment: Real-time monitoring of network bandwidth and latency, dynamic adjustment of model selection and processing strategies. If network conditions deteriorate, simplify model selection and prioritize low-latency local processing.

[0130] S5: Dynamic programming algorithm to optimize path selection: Initialize parameters, including total delay budget, network bandwidth threshold, and network delay threshold. Create a three-dimensional array to store the maximum accuracy of each delay, the previous model index, the previous frame index, and the corrected confidence score. When processing each frame of video, obtain the feature vector and network status of the current video frame, determine whether to simplify the model selection, and perform dynamic programming filling. Real-time backtracking to find the optimal path, determine the best model for the current frame, and record and output the results.

[0131] Furthermore, in step S1, video frames are preprocessed and features are extracted.

[0132] S1.1: First, the input video frame is preprocessed. Let the input video frame be I, and normalize it to a uniform resolution, denoted as I norm .

[0133] S1.2: Next, the normalized video frame is subjected to denoising to reduce the interference of noise on subsequent steps, and the preprocessed video frame I is obtained. pre .

[0134] S1.3: Then, a convolutional neural network (CNN) is used to process the preprocessed video frame I prePerform feature extraction. Suppose the feature extraction function of CNN is f CNN , then the eigenvector F can be expressed as:

[0135] F=f cnn (I pre )

[0136] The feature vector F contains key information such as the category, size, and position of the object in the video frame, which provides a basis for RoI prediction in subsequent steps.

[0137] Further, after the RoI prediction and post-processing in step S2, the prediction of RoI (selective region of interest) is performed.

[0138] S2.1: First, initialize two RoI sets: the normal RoI set S and the special RoI set S ′ , which are used to store different types of regions of interest.

[0139] S2.2: Then, obtain the historical detection result set O and calculate the number of objects N in it. For each object o in the historical detection result set O i (i=1,2,…,N), perform the following operations:

[0140] Inspection object O i Can it be covered by the f×f field of view (FoV). If it can be covered, try to merge it into the existing RoI. The merging rules can be set according to the actual situation, such as the merged FoV does not exceed the preset threshold. If the merge fails, create a new normal RoI to contain the object.

[0141] S2.3: If object O i If the size of the FoV exceeds f×f, a special RoIS is created i ′ The area of ​​a special RoI can be adjusted according to the size of the object and the preset scaling factor y, that is:

[0142] S i i =y×Area(O i )

[0143] Among them, Area(O i ) represents object O i After predicting all RoIs, the common RoI set S and the special RoI set S ′ Merge into the final RoI set R. Finally, post-process the RoI set R. Use the non-maximum suppression (NMS) algorithm to remove overlapping RoIs to ensure that each RoI is independent. Let the overlap between two RoIs be IOU (Si ,S j ), if IOU(S i ,S j )>θ (θ is the preset threshold), retain the RoI with larger weight and remove the RoI with smaller weight. At the same time, check whether the boundary of each RoI exceeds the range of the video frame. If it exceeds, make appropriate adjustments. After the above steps, the processed RoI set R is obtained ′ , as the output of the algorithm. This set contains the key regions of interest in the video frame. Extract the location information of each RoI from the RoI set R, including the center position (X c, Y c ) and field of view angle (w,h). Calculate bounding box coordinates: Convert the center position and field of view angle to the upper left and lower right coordinates of the bounding box (x1,y1,x2,y2). Format conversion: Convert the bounding box coordinates to a format suitable for processing by the existing model.

[0144] Furthermore, confidence evaluation is performed in step S3.

[0145] S3.1: To implement confidence evaluation and calibration, you first need to prepare a dataset for training and testing that contains feature vectors, true labels, and original confidence scores. x i represents the input feature vector, y i represents the true label, p i Represents the raw confidence score output by the model. Each region of interest (RoI) is evaluated and a feature vector x is extracted from each RoI. i , use the model to predict each RoI and get the original confidence score p i , record the feature vector x of each RoI i , the true label y i and the original confidence score p i Next, the confidence score is calibrated using the Platt Calibration method. Platt Calibration is a method based on logistic regression that converts the original confidence score into a calibrated probability by fitting a mapping function. Assume that the original confidence score is p i , the calibrated confidence score is

[0146] S3.2: Platt Calibration is calibrated by the following formula:

[0147]

[0148] Where A and B are the parameters that need to be fitted by the training data.

[0149] S3.3: Use the training data set D to train the Platt Calibration model. The specific steps are as follows:

[0150] Define the loss function: Use the log-likelihood loss function to optimize parameters A and B

[0151]

[0152] Optimize parameters: Use gradient descent or other optimization algorithms to minimize the loss function L(A,B)

[0153]

[0154] Update parameters: Update parameters A and B by gradient descent

[0155]

[0156] Where η is the learning rate.

[0157] Evaluate calibration effect: Use the test dataset to evaluate the performance of the calibrated model. Use the Brier Score metric to evaluate the calibration effect, which measures the squared difference between the predicted probability and the actual label.

[0158] Brier Score formula:

[0159] S3.4: Performance evaluation of the model using the prepared dataset Specifically, we list all models M1,M2,...,M k . Record its accuracy and latency under different feature vectors. Extract feature vector X from the dataset i , using each model M j For the eigenvector X i Make predictions and record the model's prediction results and prediction time t i Then, calculate the accuracy of each model under different feature vectors, which can be measured using indicators such as accuracy, precision, recall, and F1 score, and record the prediction time t of each model iNext, we build a model performance matrix to store the performance indicators of each model. We initialize a matrix P, where each row corresponds to a model and each column corresponds to a performance indicator (such as accuracy, latency, etc.). The accuracy and latency of each model under different feature vectors are recorded in the performance matrix. Through these steps, the performance of each model can be effectively recorded and evaluated, ensuring that the performance indicators of the model can be accurately recorded and compared. It can also provide reliable data support for subsequent confidence calibration and model selection.

[0160] Furthermore, in step S4, the current network bandwidth and delay are monitored in real time. Thresholds for network bandwidth and delay are set to determine whether the current network condition is suitable for model selection adjustment. It is assumed that when the network bandwidth is lower than a certain value or the delay is higher than a certain value, a simpler model is selected or the amount of data transmission is reduced. Assume that the thresholds are B threshold and L threshold .

[0161] Furthermore, in step S5, a dynamic programming algorithm is adopted.

[0162] S5.1: Before this, build a solution diagram. Figure 3 As shown, define nodes and edges. Node: Each node V i (j, t) represents a state, that is, the state where the jth frame of video is processed and the total delay is t milliseconds. Edge: Each edge represents the operation of transferring from one state to another, that is, selecting a certain model to process the current video frame. Initialization graph. Starting point: V start Indicates the initial state, that is, no video frame is processed and the total delay is 0 milliseconds. End point: V end Since it is stream processing, we do not have a fixed end point, but constantly update the optimal path. Intermediate node: V(j,t) represents the state of processing up to the jth frame of video with a total delay of t milliseconds. Build a graph and process online, for each video frame j, when it arrives: for each possible delay t, add an edge from node V(j-1,t) to node V(j,t+latency), where latency is the delay of selecting a model to process the current video frame. Calculate and update the optimal path immediately so that the decision is ready before the next frame arrives.

[0163] S5.2: Dynamic programming algorithm for video stream processing. First, input: feature vector list F: feature vector of each frame of video. Model performance matrix M: accuracy and delay of each model under different feature vectors. Network status N: including current network bandwidth B, network delay L, and total delay budget T. budget . Network bandwidth threshold B threshold : When the network bandwidth is lower than this value, a simpler model is selected. Network delay threshold L threshold:When the network delay is higher than this value, a simpler model is selected. Initialization parameter total delay budget T budget , network bandwidth B, network bandwidth threshold B threshold , network delay threshold L threshold Create a three-dimensional array dp to store the maximum accuracy, the previous model index and the previous frame index at each delay t, and the calibrated confidence score.

[0164] S5.3: Process each frame of video j, and obtain the feature vector feature_vector of the current frame for each frame, which contains key information of the frame such as object category, size, position, etc. Obtain the current network status B and L. Determine whether to select a simpler model set M based on the current network status simplified If the current network bandwidth B is lower than the bandwidth threshold B threshold Or the network delay L is higher than the delay threshold L threshold , then choose a simpler model set M simplified . Otherwise, use all available models M. Create a new dp table new_dp, initialized to default values, so that it can be updated when processing the next frame. Dynamically plan to fill, traversing every possible delay t and every available model m i , check whether the model supports the feature vector of the current frame, and calculate the new latency, new accuracy and calibrated confidence score calibrated_confidence. The formula is:

[0165] calibrated_confident=calibrate_confidence(m i [raw_confidence])

[0166] new_accuracy=dp[t][o]+accuracy

[0167] If new_accuracy>new_dp[t+latency][0]:

[0168] new_dp[t+latency]=[new_accuracy,m i ,j-1]

[0169] For every possible delay t and every available model m i , if the model m i The feature vector of the current frame is supported and the new delay does not exceed the total delay budget T budget, and if the network bandwidth is sufficient, calculate the new accuracy new_accuracy and the calibrated confidence score calibrated_confidence. Update the DP table dp with the new DP table new_dp for use in processing the next frame. Find the maximum accuracy and corresponding delay of the current state and determine the best model for the current frame.

[0170] Best accuracy best_accuracy = max t (dp[t][o]), best latency best_latency = argmax t (dp[t][o]), best calibration confidence best_calibrated_confidence = dp[best_latency][3] Traverse all delays t and find the maximum accuracy best_accuracy and the corresponding delay best_latency. Determine the optimal model optimal_model and the best calibration confidence score best_calibrated_confidence based on the maximum accuracy found. Use the optimal model optimal_model to process the current frame j. Record the best model of the current frame for subsequent analysis or output. Output the best model, maximum accuracy and best latency calibrated confidence score of the current frame to facilitate debugging and logging. This dynamic programming algorithm implements how to select the best model in video stream processing to maximize processing. Consider the impact of network conditions (bandwidth and latency) and directly use the corrected confidence score. Specifically:

[0171] Initialization parameters: Set the total delay budget, network bandwidth threshold, and network delay threshold.

[0172] Initialize the DP table: Create a three-dimensional array to store the maximum accuracy, previous model index, previous frame index, and corrected confidence score for each delay.

[0173] Process each frame of video: obtain feature vectors and network status, determine whether model selection needs to be simplified, initialize a new DP table and perform dynamic programming to fill it.

[0174] Real-time backtracking to find the optimal path: find the maximum accuracy and corresponding delay of the current state, determine the best model for the current frame, and record and output the results.

[0175] Example 3

[0176] This embodiment also provides a video processing system based on edge computing and confidence evaluation, which is used to implement the method described in Embodiment 1 or Embodiment 2, such as Figure 4 As shown, including:

[0177] A data acquisition module acquires a video stream and divides the video stream into independent video frames;

[0178] A preprocessing module preprocesses each video frame to obtain a number of preprocessed video frames;

[0179] A feature vector acquisition module performs feature extraction based on a number of preprocessed video frames to generate corresponding feature vectors;

[0180] A key ROI region determination module determines a key ROI region in a corresponding video frame according to a number of pre-processed video frames and corresponding feature vectors;

[0181] A calibration module, selecting a corresponding classification model according to the corresponding feature vector and key ROI area, and calibrating the confidence score of the model;

[0182] The model adjustment module monitors the current network bandwidth and delay in real time, sets the network bandwidth threshold and delay threshold, and adjusts the selection of the classification model according to the network bandwidth threshold and delay threshold;

[0183] The video frame processing module further adjusts the classification model of the current video frame using a dynamic programming algorithm to obtain an optimal classification model for the current video frame, and processes the current video frame using the optimal classification model.

[0184] The same or similar reference numerals correspond to the same or similar components;

[0185] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0186] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A video processing method based on edge computing and confidence assessment, characterized in that: include: Get the video stream and split it into independent video frames; Preprocess each video frame to obtain a number of preprocessed video frames; Perform feature extraction based on several preprocessed video frames to generate corresponding feature vectors; Determine a key ROI region in a corresponding video frame according to a plurality of preprocessed video frames and corresponding feature vectors; Selecting a corresponding classification model according to the corresponding feature vector and key ROI area, and calibrating the confidence score of the model; Monitor the current network bandwidth and delay in real time, set the network bandwidth threshold and delay threshold, and adjust the selection of the classification model according to the network bandwidth threshold and delay threshold; The classification model of the current video frame is further adjusted using a dynamic programming algorithm to obtain an optimal classification model for the current video frame, and the current video frame is processed using the optimal classification model.

2. The video processing method based on edge computing and confidence assessment according to claim 1, characterized in that: Each video frame is preprocessed to obtain several preprocessed video frames, including: Normalize each video frame to a uniform resolution to obtain a number of normalized video frames; De-noising is performed on a number of normalized video frames to obtain a number of pre-processed video frames.

3. The video processing method based on edge computing and confidence assessment according to claim 1, characterized in that: According to a number of pre-processed video frames and corresponding feature vectors, a key ROI region in a corresponding video frame is determined, including: Creating a common ROI region set and a special ROI region set, wherein the common ROI region set and the special ROI region set are used to store different types of ROI regions; Obtaining a plurality of ROI region sets of preprocessed video frames, and calculating the number of objects in the ROI region sets; Obtain the category, size and position of the object in the preprocessed video frame according to the corresponding feature vector; If the size of the object in the ROI area set of the pre-processed video frames is less than or equal to the field of view of the preset size and meets the merging condition, the object is merged into the common ROI area set; if the merging condition is not met, a new common interest area is created to include the object and merged into the common interest area set; if the size of the object is larger than the field of view of the preset size, a special ROI area is created, the special ROI area is adjusted according to the object size and the preset scaling factor, the object is included in the adjusted special ROI area, and merged into the special ROI set; After processing all the ROI region sets of the preprocessed video frames, the merged common ROI regions and special ROI regions are merged to obtain a final ROI region set; The weight of each ROI region in the final ROI region set is determined, and key ROI regions are screened out according to the preset weights.

4. The video processing method based on edge computing and confidence assessment according to claim 3, characterized in that: The formula for adjusting the special ROI area according to the object size and the preset scaling factor is: S i ′ =y×Area(O i ) Where y is the preset scaling factor; Area(O i ) is the area of ​​the object.

5. The video processing method based on edge computing and confidence assessment according to claim 1, characterized in that: Select a corresponding classification model according to the corresponding feature vector and key ROI area, and calibrate the confidence score of the model, including: Selecting a corresponding classification model according to the corresponding feature vector and the key ROI area, and obtaining an original confidence score corresponding to the key ROI area through the model; Construct a calibration dataset based on feature vectors, true labels, and original confidence scores of key ROI regions; Input the calibration data set into the trained calibration model to obtain the confidence score after calibration; The original confidence scores of the classification model are replaced with the calibrated confidence scores.

6. The video processing method based on edge computing and confidence assessment according to claim 5, characterized in that: Get the trained calibration model, including: Build a calibration model, input the calibration data set into the trained calibration model, and obtain the confidence score after calibration; Set the log-likelihood loss function, calculate the log-likelihood loss function value according to the original confidence score and the calibrated confidence score, and adjust the parameters of the constructed calibration model; When the log-likelihood loss function value reaches a preset value or reaches a preset number of training times, a trained calibration model is obtained.

7. The video processing method based on edge computing and confidence assessment according to claim 6, characterized in that: The log-likelihood loss function is: Among them, y i is the true label; is the confidence score after calibration; p i Score the raw confidence.

8. The video processing method based on edge computing and confidence assessment according to claim 6, characterized in that: The gradient descent method is used to adjust the parameters of the constructed calibration model, specifically: The loss function is minimized using the gradient descent method, and the formula is: Among them, y i is the true label; is the confidence score after calibration; p i Score the raw confidence; is the gradient of the loss function of the first model parameter; is the gradient of the loss function with respect to the second model parameter. The model parameters are updated by gradient descent, and the formula is: Among them, A is the first model parameter; B is the second model parameter; η is the learning rate, which controls the step size of parameter update in each iteration.

9. The video processing method based on edge computing and confidence assessment according to claim 1, characterized in that: The classification model of the current video frame is further adjusted using a dynamic programming algorithm to obtain the optimal classification model of the current video frame, including: Constructing a scheme graph, defining nodes and edges of the scheme graph, wherein each node of the scheme graph represents the number of frames and the total delay number of the processed current video frame; and each edge represents that a certain classification model processes the current video frame; Setting a total delay budget, a network bandwidth threshold, and a network delay threshold and creating three-dimensional data, wherein the three-dimensional data is used to store a maximum accuracy corresponding to each delay, an index of a previous model, an index of a previous video frame, and a calibrated confidence score; According to the feature vector and network status of each preprocessed video frame, determine whether the model selection needs to be adjusted, initialize the DP table and perform dynamic programming filling; Find the model index when the accuracy of the current video frame is the highest and the total delay is the lowest, and determine the best classification model for the current video frame based on the model index.

10. A video processing system based on edge computing and confidence assessment, used to implement the method according to any one of claims 1 to 9, characterized in that: include: A data acquisition module acquires a video stream and divides the video stream into independent video frames; A preprocessing module preprocesses each video frame to obtain a number of preprocessed video frames; A feature vector acquisition module performs feature extraction based on a number of preprocessed video frames to generate corresponding feature vectors; A key ROI region determination module determines a key ROI region in a corresponding video frame according to a number of pre-processed video frames and corresponding feature vectors; A calibration module, selecting a corresponding classification model according to the corresponding feature vector and key ROI area, and calibrating the confidence score of the model; The model adjustment module monitors the current network bandwidth and delay in real time, sets the network bandwidth threshold and delay threshold, and adjusts the selection of the classification model according to the network bandwidth threshold and delay threshold; The video frame processing module further adjusts the classification model of the current video frame using a dynamic programming algorithm to obtain an optimal classification model for the current video frame, and processes the current video frame using the optimal classification model.

Images