Multi-computing-power collaborative video structured analysis method

Abstract

This invention relates to the fields of computer vision and edge computing technology, and discloses a multi-computing-power collaborative video structured analysis method. The method includes: acquiring multiple raw video streams and synchronously decoding them; using gradient analysis to assess the quality of the decoded raw video streams and selecting a combination of preprocessing algorithms based on quality feature vectors; establishing a data transmission channel using an asynchronous communication protocol and a multi-level caching system; performing content complexity analysis and obtaining scheduling decisions through model matching based on the analysis; distributing the model and performing pipelined inference on the deployed model; fusing the pipelined inference results and verifying the consistency of the fused results; and optimizing system configuration parameters based on the video structured analysis results to achieve video structured analysis. This invention improves the real-time performance, accuracy, and system stability of video analysis.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and edge computing technology, and more specifically, to a method for collaborative video structured analysis using multi-computing power. Background Technology

[0002] With the rapid development of industry and intelligent manufacturing, the demand for video structured analysis technology in fields such as industrial quality inspection, intelligent monitoring, and autonomous driving is increasing. Traditional video analysis methods mainly rely on a single processor architecture, which often suffers from insufficient computing power, high processing latency, and high energy consumption when facing the real-time processing requirements of multiple high-resolution video streams. Especially in industrial production environments, higher requirements are placed on the real-time performance, accuracy, and stability of video analysis systems.

[0003] Existing multi-computing power collaboration technologies mainly focus on heterogeneous computing between general-purpose GPUs and CPUs, lacking in-depth optimization techniques for collaboration between dedicated video processing units (VPUs) and ARM processors. Patent CN115578694A proposes a video analytics computing power scheduling method, but it has shortcomings in cross-processor collaboration efficiency and real-time performance guarantees. While edge-cloud collaborative intelligent computing has theoretical advantages, it faces technical challenges in practical industrial applications, such as data transmission latency and privacy security.

[0004] The main shortcomings of existing technical solutions include: low efficiency in utilizing computing resources and a lack of in-depth utilization of VPU characteristics; insufficient cross-processor collaboration efficiency, with problems such as high data transmission overhead and long synchronization waiting times; inadequate collaborative inference capabilities of multiple AI models, failing to fully utilize the parallel capabilities of heterogeneous computing platforms; and imperfect real-time guarantee mechanisms, lacking predictive task scheduling and dynamic optimization capabilities. These technical issues limit the promotion and application effectiveness of video structured analysis technology in industrial applications. Summary of the Invention

[0005] This invention provides a video structured analysis method that utilizes multiple computing resources collaboratively, addressing technical problems such as low efficiency of computing resource utilization, insufficient collaborative reasoning capabilities of multiple AI models, and imperfect real-time guarantee mechanisms in related technologies.

[0006] This invention provides a multi-computing-power collaborative video structured analysis method, comprising the following steps:

[0007] Multiple raw video streams are acquired and synchronously decoded. Gradient analysis is used to evaluate the quality of the decoded raw video streams, and quality feature vectors are obtained. Based on the quality feature vectors, a combination of preprocessing algorithms is selected to obtain quality-enhanced video frame data.

[0008] Based on the enhanced video frame data, a data transmission channel is established, and an asynchronous communication protocol and a multi-level caching system are used to obtain a cross-processor data communication stream.

[0009] Perform content complexity analysis on cross-processor data communication flows, and obtain scheduling decisions based on model matching according to the content complexity analysis;

[0010] Based on the scheduling decision, the model is distributed and deployed, and pipelined inference is performed on the deployed model to obtain the pipelined inference results.

[0011] The pipeline inference results are fused, and the consistency constraint verification of the fused results is performed to obtain the video structured analysis results;

[0012] The system configuration parameters are optimized based on the video structured analysis results to achieve millisecond-level detection response and micron-level defect identification through video structured analysis.

[0013] In a preferred embodiment, the gradient analysis quality assessment of the decoded multi-channel original video streams includes:

[0014] Grayscale images are obtained by converting the original video frame sequence to grayscale.

[0015] Calculate the horizontal and vertical gradients of a grayscale image, and calculate the sharpness score based on the gradient variance;

[0016] Perform frequency domain transformation on grayscale images, analyze the energy distribution of high-frequency components, and calculate noise level scores;

[0017] Obtain the brightness distribution of the grayscale image, calculate the brightness histogram entropy value, and obtain the illumination quality score;

[0018] The sharpness score, noise score, and illumination score are weighted and summed according to preset weighting coefficients to obtain the quality feature vector.

[0019] In a preferred embodiment, the step of obtaining the scheduling decision by model matching based on content complexity analysis includes:

[0020] The video content complexity index is calculated using an integral image texture analysis algorithm;

[0021] The AI ​​model type and computational requirements are determined by using a decision table lookup based on the content complexity index.

[0022] A weighted scoring algorithm is used to calculate the task processor matching degree;

[0023] Obtain the real-time load status of multiple computing nodes and calculate the comprehensive load index of each node;

[0024] Based on the comprehensive load index and the computational complexity of the AI ​​inference model, a load balancing strategy is adopted to distribute inference tasks to each computing node.

[0025] In a preferred embodiment, the combination of preprocessing algorithms based on quality feature vector selection includes:

[0026] Obtain the noise score from the quality feature vector;

[0027] Based on noise scoring, a Gaussian filtering noise reduction algorithm was selected to process the video frame data;

[0028] Obtain the illumination score from the quality feature vector;

[0029] Based on the illumination score, a contrast-limited adaptive histogram equalization algorithm is selected for illumination compensation.

[0030] Detect the overall contrast of video frame data, and select a histogram equalization algorithm to enhance contrast based on the contrast score;

[0031] The selected preprocessing algorithms are combined in order of priority to form a preprocessing algorithm sequence for the current video frame.

[0032] In a preferred embodiment, the step of using an asynchronous communication protocol and a multi-level caching system to obtain a cross-processor data communication stream includes:

[0033] A dedicated DMA data transfer channel is established between the 18TOPS VPU and the 6TOPS RK3588 using a PCIe high-speed channel and shared memory mechanism.

[0034] Establish a dedicated DMA channel between processors and use hardware-level data transfer to avoid CPU copy overhead;

[0035] The shared memory region is divided into multiple circular buffers, and the producer-consumer pattern is used to realize data exchange between computing nodes.

[0036] It adopts an event-driven asynchronous communication protocol and coordinates data transmission and task execution through a message queue mechanism;

[0037] Establish a multi-level caching system, including internal processor cache, inter-processor shared cache, and system-level data cache.

[0038] In a preferred embodiment, the distributed deployment of the model based on the scheduling decision includes:

[0039] Based on AI model structure analysis, a computational graph segmentation algorithm is used to determine the model's distribution and deployment scheme on VPU and RK3588;

[0040] Choose inference models of different precision based on the complexity of the video content; use lightweight models for simple scenarios and high-precision models for complex scenarios.

[0041] A strategy combining model parallelism and data parallelism is adopted to split large models into multiple computing nodes for parallel execution;

[0042] Establish a model caching mechanism to preload frequently used models into the high-speed cache and reduce model switching latency.

[0043] In a preferred embodiment, the pipelined inference of the deployed model includes:

[0044] Based on the principle of computational load balancing, the model partitioning point is determined, and computationally intensive operations are allocated to the 18 TOPS VPU, while control logic is allocated to the 6 TOPS RK3588 processor.

[0045] A double buffering mechanism with input and output buffers is adopted to enable each computing node to work continuously and avoid waiting time;

[0046] The large inference task is decomposed into fine-grained computational tasks at the millisecond level, and a strategy combining data parallelism and model parallelism is adopted.

[0047] A semaphore-based synchronization mechanism is adopted to ensure that parallel tasks are completed in the correct order through precise timing control.

[0048] In a preferred embodiment, the consistency constraint verification of the fusion result includes:

[0049] Based on the inference output of each AI model, a standardized set of detection results is obtained using the result aggregation module of the 6TOPS RK3588 processor.

[0050] A weighted average method is used to fuse the inference results of multiple computing nodes, with the weights determined based on the historical accuracy of each node;

[0051] Establish a confidence assessment mechanism to quantitatively evaluate the reliability of inference results;

[0052] When a conflicting result is detected, arbitration is carried out using a voting mechanism or an expert system.

[0053] The fused results are post-processed, including nonmaximum suppression and result smoothing.

[0054] In a preferred embodiment, optimizing the system configuration parameters based on the video structured analysis results includes:

[0055] Based on the system operation process, a multi-dimensional performance monitoring mechanism is adopted to obtain comprehensive performance statistics.

[0056] The buffer size is dynamically adjusted according to the video resolution and frame rate to ensure smooth data transmission;

[0057] Based on processor temperature and power consumption monitoring, dynamically adjust processor frequency and voltage;

[0058] An adaptive threshold adjustment mechanism is adopted to dynamically adjust the algorithm parameters according to the detection accuracy requirements;

[0059] Establish a performance monitoring and feedback mechanism to optimize the overall system performance in real time.

[0060] A computer-readable storage medium for storing computer-readable instructions that, when read by a computer, enable the execution of a multi-computing-power collaborative video structured analysis method.

[0061] The beneficial effects of this invention are as follows:

[0062] By establishing an intelligent collaborative mechanism between an 18-TOPS VPU and a 6-TOPS RK3588, the computational strengths of each processor are fully utilized to achieve optimal allocation of computing resources. A refined task scheduling strategy is employed, allocating computationally intensive neural network inference tasks to the VPU and tasks with complex control logic to the RK3588, thus avoiding resource waste and performance bottlenecks found in traditional solutions. Through cross-processor communication optimization, a multi-level caching system, and zero-copy data transfer technology, data transfer overhead and processing latency are reduced, resulting in a significant improvement in overall system performance compared to traditional solutions.

[0063] By employing model segmentation and distributed inference technologies, large-scale AI models are intelligently decomposed and distributed across different processors for parallel execution, overcoming the computational limitations of a single processor. A pipelined processing mechanism and double buffering technology enable different processors to work in parallel without waiting for each other, achieving true collaborative computing. Combined with adaptive model compression and quantization techniques, the model accuracy can be dynamically adjusted according to real-time requirements, meeting millisecond-level response requirements while ensuring detection quality, providing reliable technical support for the concurrent processing of multiple high-resolution video streams. Attached Figure Description

[0064] Figure 1 This is a flowchart of the multi-computing power collaborative video structured analysis method of the present invention;

[0065] Figure 2 This is a comparison chart of the system processing delays of the present invention;

[0066] Figure 3 This is a radar chart showing the computing power utilization rate of this invention;

[0067] Figure 4 This is a trend chart of the detection accuracy of the present invention. Detailed Implementation

[0068] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0069] At least one embodiment of the present invention discloses a video structured analysis method with multi-computing power collaboration, such as... Figure 1 As shown, it includes:

[0070] Step 1: Acquire multiple raw video streams and decode them synchronously; use gradient analysis to evaluate the quality of the decoded raw video streams, obtain quality feature vectors, and select a combination of preprocessing algorithms based on the quality feature vectors to obtain enhanced video frame data.

[0071] Specifically, it includes the following:

[0072] Based on 8-16 channels of 4K industrial camera input, a multi-channel video decoder using a 6TOPS RK3588 processor is employed to obtain a synchronized sequence of raw video frames. Specifically, the RK3588 processor uses its built-in hardware video decoding unit to perform parallel decoding processing on video streams from different industrial cameras. The decoding process employs a timestamp synchronization mechanism to ensure temporal consistency of video frames from different cameras, providing a reliable data foundation for subsequent multi-view collaborative analysis.

[0073] Based on the original video frame sequence, an image quality assessment algorithm based on gradient analysis is used to obtain the quality score and quality feature vector of each frame. Specifically, the input video frames are converted to grayscale; the gradient information of the image, including horizontal and vertical gradients, is calculated; and key features such as image sharpness, noise level, and illumination quality are extracted by analyzing the statistical characteristics of the gradient distribution.

[0074] The quality score is calculated using a weighted comprehensive evaluation method: three sub-indicators are calculated separately: image sharpness score, noise level score, and illumination quality score; these are then weighted and summed according to preset weights to obtain the comprehensive quality score. Specifically, the sharpness score is obtained by calculating the image gradient variance; a larger gradient variance indicates sharper image edges. The noise score is calculated by analyzing the energy distribution of high-frequency components; excessively high-frequency energy indicates severe noise. The illumination score is obtained by evaluating the uniformity of brightness distribution, using the entropy value of the brightness histogram to measure illumination uniformity. The weighting coefficients are determined based on the application scenario; typically, the sharpness weight is set to 0.5, the noise weight to 0.3, and the illumination weight to 0.2.

[0075] To ensure the comparability of different quality indicators, each score needs to undergo normalization preprocessing before weighted calculation: calculate the original value of each indicator; and use the min-maximum normalization method to map it to a standard range of 0 to 1. The specific steps are: obtain the historical minimum and maximum values ​​of each indicator as the normalization benchmark, subtract the minimum value from the current original value, and divide by the difference between the maximum and minimum values; for noise scores, since lower noise generally indicates better quality, a reverse processing is required, i.e., subtract the result of the forward normalization from 1.

[0076] Based on the quality feature vector, a decision tree classifier is used to obtain the optimal combination of preprocessing algorithms for the current frame. The decision tree classifier learns the mapping relationship between different combinations of quality features and the optimal preprocessing strategy in advance through training data. The preprocessing algorithm library includes Gaussian filtering for noise reduction, histogram equalization, contrast-limited adaptive histogram equalization (CLAHE), gamma correction, etc. The decision tree gradually narrows down the candidate algorithm range based on different dimensional values ​​of the quality feature vector through branching judgment in the tree structure, and finally selects the appropriate preprocessing algorithm sequence. For example, when the noise score is high, Gaussian filtering for noise reduction is preferred; when the illumination is uneven, the CLAHE algorithm is selected; when the contrast is insufficient, histogram equalization is selected to ensure that the quality of the output image meets the requirements of subsequent AI inference.

[0077] Based on the preprocessed video frames, a quality restoration algorithm with temporal consistency constraints is used to obtain enhanced video frame data. For frames with quality scores below a threshold, temporal interpolation restoration is performed using information from adjacent high-quality frames. The temporal interpolation algorithm is based on optical flow estimation, calculates motion vectors between adjacent frames, and uses motion compensation techniques to generate high-quality interpolated frames.

[0078] Furthermore, deep learning-based image super-resolution reconstruction techniques can be employed to enhance the quality of low-quality images through a pre-trained lightweight super-resolution network. This network uses a residual connection structure, which can significantly improve image quality while maintaining real-time performance, providing more reliable input data for subsequent defect detection.

[0079] Step 2: Based on the enhanced video frame data, establish a data transmission channel, and use an asynchronous communication protocol and a multi-level caching system to obtain a cross-processor data communication stream;

[0080] Specifically, it includes the following:

[0081] Based on the hardware interface of the 18TOPS VPU and 6TOPS RK3588, a low-latency data transfer channel is achieved using a high-speed PCIe channel and a shared memory mechanism. Specifically, a dedicated DMA (Direct Memory Access) channel is established between the two processors to avoid CPU copy overhead through hardware-level data transfer; the shared memory region is divided into multiple circular buffers, and a producer-consumer model is used to achieve efficient data exchange.

[0082] Based on DMA channel configuration, an event-driven asynchronous communication protocol is adopted to achieve a non-blocking data exchange mechanism between processors. The communication protocol uses a message queue mechanism. The VPU and RK3588 coordinate data transmission and task execution by sending and receiving messages. Each message contains information such as data type, data address, data length, and timestamp to ensure the correctness and timing of data transmission.

[0083] Based on data access pattern analysis, a hierarchical caching strategy is adopted to obtain an optimized data storage and access mechanism.

[0084] The cache hit rate is calculated using a statistical counting method: by recording the number of cache hits and the total number of accesses during the cache access process, the proportion of hits to total accesses is calculated. Specifically, the system maintains two counters, one recording the number of cache hits and the other the number of cache misses. Each time data is accessed, the corresponding counter is updated based on whether the required data is found in the cache. The hit rate equals the number of hits divided by the total number of accesses, where the total number of accesses is the sum of hits and misses.

[0085] The multi-level caching system includes L1 cache (internal processor cache), L2 cache (inter-processor shared cache), and L3 cache (system-level data cache). L1 cache stores frequently accessed small data blocks, L2 cache stores medium-sized datasets, and L3 cache stores large-scale historical data and model parameters.

[0086] Based on data feature analysis, an adaptive data compression algorithm is employed to obtain a data stream with optimized transmission bandwidth. For image data, a lossless compression algorithm based on wavelet transform is used, typically achieving a compression ratio of 2-4 times. For feature vector data, vector quantization technology is employed to reduce data transmission volume while maintaining accuracy.

[0087] Furthermore, predictive data prefetching technology can be used to predict future data needs based on historical access patterns and load potentially needed data into the cache in advance. This technology is based on the Markov chain model and predicts the data blocks that may be accessed at the next moment by analyzing the temporal patterns of data access, thereby reducing data transmission latency.

[0088] Step 3: Perform content complexity analysis on the cross-processor data communication flow, and obtain scheduling decisions based on model matching according to the content complexity analysis;

[0089] Specifically, it includes the following:

[0090] Based on quality-enhanced video frames, a fast texture analysis algorithm based on integral images is used to obtain video content complexity indicators. The implementation of integral image technology includes: constructing an integral image of the original image, where each position stores the cumulative sum of all pixel values ​​from the top left corner of the image to that position; and utilizing the properties of integral images, quickly calculating the pixel sum of any rectangular region through simple addition and subtraction operations.

[0091] Texture complexity calculation employs a gradient statistics-based approach: Sobel edge detection is performed on the image, and the gradient strengths in the horizontal and vertical directions are calculated separately; statistical features of the gradient magnitudes are then calculated. Specific steps include: performing convolution operations on the image using a 3×3 Sobel operator to obtain the horizontal and vertical gradients; calculating the gradient magnitude of each pixel, i.e., the square root of the sum of the squares of the horizontal and vertical gradients; and calculating the mean and standard deviation of the gradient magnitudes across the entire image as a quantitative indicator of texture complexity.

[0092] This algorithm utilizes integral image technology to quickly calculate gradient statistics for any rectangular region, thereby efficiently evaluating the texture complexity of an image. The complexity metric guides subsequent computational resource allocation; regions with high complexity require more computational resources, while regions with low complexity can employ lightweight processing strategies.

[0093] To ensure the comparability of texture complexity metrics across different images, the calculated raw complexity values ​​need to be standardized. The Z-score standardization method is used: the raw complexity value is subtracted from the historical mean, and then divided by the historical standard deviation to obtain the standardized texture complexity score. The standardized value reflects the texture complexity of the current image relative to historical samples; a positive value indicates a more complex texture, a negative value indicates a simpler texture, and the absolute value indicates the degree of deviation.

[0094] Based on the content complexity index and the type of detection task, a decision table lookup mechanism is used to obtain the required AI model type and computational requirements. The decision table pre-stores the optimal model configuration corresponding to different complexity levels, including model type (defect detection, dimensional measurement, assembly inspection), model size, accuracy requirements, etc.; the lookup process uses hash table technology to ensure that the decision time is in the microsecond range.

[0095] Based on the computational requirements of AI models and the performance characteristics of processors, a weighted scoring algorithm is used to obtain the task processor matching degree matrix.

[0096] The matching score calculation uses a weighted summation method, linearly combining the matching scores of each performance dimension according to preset weights. The specific calculation process is as follows: obtain the processor's capability value and the task's requirement value for each performance dimension; calculate the performance supply-demand ratio for each dimension, i.e., processor capability divided by task requirement; convert the supply-demand ratio into a matching score between 0 and 1; and perform a weighted average according to the importance weights of each dimension to obtain the comprehensive matching score. The weight allocation is usually determined based on the application scenario, with more important dimensions receiving higher weights.

[0097] The matching degree calculation adopts a multi-dimensional comprehensive evaluation method: assessing the performance supply capacity and task demand intensity of each computing unit separately, and then calculating the degree of supply and demand matching. The specific implementation steps are as follows: collecting real-time performance data of each computing unit, including key indicators such as processing speed, memory utilization, and power consumption; simultaneously analyzing the computing requirements of the tasks to be assigned, including algorithm complexity, data size, and real-time requirements; standardizing the performance data and demand data to eliminate differences in the units of measurement between different indicators; calculating the supply-demand ratio for each dimension to reflect the degree of matching of the computing unit in that dimension; and using a weighted average method to synthesize the matching degrees of each dimension to obtain the overall matching score.

[0098] To improve the accuracy of matching degree calculation, the following data preprocessing strategies are adopted: performance indicators and demand indicators are normalized separately to map the original values ​​to a unified scoring range; when calculating the supply-demand ratio, safe division is used to avoid division by zero errors; the sigmoid function is used to map the supply-demand ratio to a matching score between 0 and 1 to ensure the rationality and interpretability of the score; and the weight coefficients of each dimension are dynamically adjusted according to the task type and application scenario to improve the adaptability of the matching algorithm.

[0099] Performance characteristics include key metrics such as computational throughput, memory bandwidth, and inference latency. A higher match indicates that the processor is more suitable for performing the task.

[0100] Based on the matching degree matrix and the current system load status, a greedy scheduling algorithm is used to obtain the optimal task allocation scheme. The greedy algorithm prioritizes assigning tasks with the highest matching degree and the highest urgency to the corresponding processors, while considering load balancing constraints to avoid overloading any processor. The scheduling decision also includes strategies such as task execution order, priority settings, and timeout handling.

[0101] Furthermore, an adaptive scheduling strategy based on reinforcement learning can be adopted, continuously optimizing scheduling decisions through online learning. The reinforcement learning agent observes the current system state (including task queues, processor load, historical performance, etc.), selects the optimal task allocation action, and adjusts the strategy parameters based on system performance feedback, thereby achieving continuous optimization of the scheduling strategy.

[0102] Step 4: Distribute the model according to the scheduling decision, perform pipelined inference on the deployed model, and obtain the pipelined inference results;

[0103] Specifically, it includes the following:

[0104] Based on AI model structure analysis, a computational graph segmentation algorithm is used to obtain a distribution deployment scheme for the model on VPU and RK3588. The model segmentation considers the following factors: computational complexity distribution, memory access pattern, and data dependencies. Typically, computationally intensive operations such as convolutional layers and pooling layers are allocated to the 18 TOPS VPU, while data preprocessing, post-processing, and control logic are allocated to the 6 TOPS RK3588.

[0105] Load balancing employs a processing time ratio allocation strategy: by comparing the processing time per unit of computing power of different processors, tasks are allocated rationally. Specifically, the time required for each processor to complete a standard task is measured to obtain its respective processing efficiency index; task loads are allocated according to the ratio of processing efficiency to ensure that the relative load of each processor remains balanced; the system reaches optimal load balancing when the time efficiency ratio of the VPU and RK3588 processors approaches their computing power ratio. This allocation method fully utilizes the computing power of each processor, avoiding situations where some processors are overloaded while others are idle.

[0106] Based on the model distribution scheme, a double-buffering mechanism is adopted to obtain a continuous pipeline execution flow. The double-buffering mechanism includes an input buffer and an output buffer. When one processor processes the data in the current buffer, another processor prepares the data for the next buffer. This mechanism ensures that the processors can work continuously and avoids idle time waiting for data transmission.

[0107] Based on pipelined design, task slicing technology is employed to obtain fine-grained parallel execution units. Large inference tasks are decomposed into multiple smaller computational tasks, with the execution time of each task controlled at the millisecond level. Task slicing is based on a strategy combining data parallelism and model parallelism; data parallelism is used for batch processing scenarios, while model parallelism is used for large models.

[0108] Based on parallel execution units, a semaphore-based synchronization mechanism is employed to achieve a coordinated and consistent inference execution process. The synchronization mechanism includes task startup synchronization, data transmission synchronization, and result collection synchronization. Through precise timing control, it ensures that each parallel task completes in the correct order, avoiding data races and result errors.

[0109] Furthermore, a dynamic task scheduling mechanism based on a stream graph can be adopted, modeling the entire inference process as a directed acyclic graph, where nodes represent computational tasks and edges represent data dependencies. The scheduler dynamically determines the execution order and resource allocation of tasks based on the graph structure and resource status, achieving a more flexible parallel execution strategy.

[0110] Step 5: Fuse the pipeline inference results, verify the consistency constraints of the fused results, and obtain the video structured analysis results;

[0111] The inference results are integrated using a Bayesian fusion algorithm, and the results are verified using consistency constraints to obtain the video structured analysis results.

[0112] Specifically, it includes the following:

[0113] Based on the inference output of each AI model, the result aggregation module of the 6TOPS RK3588 processor is used to obtain a standardized set of detection results. The result set includes defect detection results (defect type, location, size), dimensional measurement results (critical dimensions, deviation values), assembly inspection results (assembly status, error type), etc. Each result is accompanied by a confidence score and timestamp information.

[0114] Based on the detection result set, a Bayesian inference fusion algorithm is used to obtain an optimized comprehensive detection result. Bayesian fusion employs a conditional probability inference method: based on observations from multiple evidence sources, the most probable hypothesis is calculated through probabilistic inference. The specific implementation process is as follows: Detection evidence from different computing units is collected, each piece of evidence containing detection results and confidence information; prior probabilities are determined based on the historical performance of each computing unit, with units that perform well receiving higher prior weights; likelihood probabilities are calculated based on the confidence of the current detection results, with results having higher confidence receiving greater likelihood weights; the likelihood probability is multiplied by the prior probability to obtain the joint probability; the joint probability of all possible hypotheses is normalized to obtain the posterior probability of each hypothesis; the hypothesis with the highest posterior probability is selected as the final fusion result. This method comprehensively considers historical experience and current evidence, improving the reliability of the fusion decision.

[0115] Bayesian fusion employs probabilistic statistical methods to integrate the inference results of multiple computing units: collecting the detection results and corresponding confidence scores output by each computing unit; and evaluating the reliability weight of each unit based on historical performance data. The specific fusion process includes: normalizing the confidence scores of each unit to ensure consistency in numerical range; calculating the prior probability based on the historical accuracy of each unit, with units having higher accuracy receiving greater prior weights; calculating the likelihood probability by combining the confidence of the current detection result, with results having higher confidence receiving greater likelihood weights; applying Bayes' theorem to calculate the posterior probability, comprehensively considering prior information and current evidence; and selecting the detection category with the highest posterior probability as the final fusion result.

[0116] To ensure the reliability of the fusion results, the following quality control measures are adopted: abnormally low confidence detection results are marked and specially processed; when the results of each unit differ significantly, a conflict resolution mechanism is activated, which may include re-detection or manual intervention; the historical performance statistics of each unit are dynamically updated to maintain the timeliness of the prior probability; a minimum confidence threshold is set for the fusion results, and results below the threshold are marked as uncertain.

[0117] This algorithm can comprehensively consider the detection results of multiple AI models and obtain a more reliable final conclusion through probabilistic reasoning.

[0118] Based on the Bayesian fusion results, a consistency constraint check algorithm is used to obtain validated detection results. The consistency check includes spatial consistency (different detection results at the same location should support each other), temporal consistency (detection results in consecutive frames should change smoothly), and logical consistency (detection results of different types should conform to physical laws).

[0119] Based on the validated test results, a standardized quality inspection report is generated using templated report generation technology. The report includes a test summary, detailed results, statistical analysis, and quality rating. The report format supports multiple formats such as XML, JSON, and database records, facilitating integration with upper-level management systems.

[0120] Furthermore, relational reasoning technology based on graph neural networks can be used to model various detected defects, dimensional deviations, assembly errors, and other information into graph structures. By learning the correlation between various quality problems through graph neural networks, potential systemic quality problems can be discovered, providing deeper insights for production process optimization.

[0121] Step 6: Optimize the system configuration parameters based on the video structured analysis results to achieve millisecond-level detection response and micron-level defect identification in video structured analysis;

[0122] Specifically, it includes the following:

[0123] Based on the system operation process, a multi-dimensional performance monitoring mechanism is adopted to obtain comprehensive performance statistics. The monitoring dimensions include: processing latency (end-to-end latency, latency of each step), throughput (frames processed per second, task completion rate), accuracy (detection accuracy, false alarm rate, false negative rate), and resource utilization (CPU utilization, memory utilization, cache hit rate).

[0124] The comprehensive performance evaluation adopts a multi-index weighted evaluation method: collect key performance data during system operation, including core indicators such as processing latency, throughput, detection accuracy, and resource utilization efficiency; standardize each indicator to eliminate differences in the dimensions and numerical ranges between different indicators; and perform weighted summation according to preset weights to obtain a comprehensive performance score.

[0125] The specific evaluation steps are as follows: Latency metrics are processed using inverse standardization, meaning lower latency results in a higher score; the inverse of the original latency value is normalized. Throughput metrics are processed using forward standardization, meaning higher throughput results in a higher score; the original throughput value is directly normalized. Accuracy metrics, already in percentage form, are directly converted to a score of 0 to 1 by dividing by 100. Efficiency metrics are obtained by calculating the proportion of effective processing time to total time. The weighting coefficients for each metric are determined based on application requirements; typically, latency is weighted at 0.3, throughput at 0.3, accuracy at 0.25, and efficiency at 0.15.

[0126] To ensure the objectivity and comparability of the evaluation results, standardized performance benchmarks were established: the latency benchmark is based on the processing time of a single frame, with a target value typically set to within 50 milliseconds; the throughput benchmark is based on the number of frames processed per second, with a target value typically set to more than 30 frames per second; the accuracy benchmark is based on the detection accuracy, with a target value typically set to more than 95%; and the efficiency benchmark is based on the resource utilization rate, with a target value typically set to more than 80%.

[0127] Based on performance statistics, the Bayesian optimization algorithm is used to obtain the optimal system configuration parameters, including task scheduling strategy parameters, cache configuration parameters, model compression parameters, and communication protocol parameters. Bayesian optimization constructs a Gaussian process model of the performance function, intelligently selects the next evaluation point, and finds the global optimal solution within a limited number of evaluations.

[0128] Based on the parameter optimization results, the gradient descent algorithm is used to obtain updated scheduling strategies and configuration parameters. The adaptive update mechanism can dynamically optimize system behavior according to changes in workload and adjustments in performance requirements. The update process adopts a gradual strategy to avoid drastic parameter changes from affecting system stability.

[0129] Based on the updated system configuration and performance optimization results, and employing system integration technology, the system achieves millisecond-level detection response and micron-level defect identification in structured video analysis. Through self-learning performance tuning, the system intelligently coordinates heterogeneous computing resources, establishes an adaptive task scheduling mechanism, and forms a complete industrial video analytics solution. The system possesses anomaly detection and automatic recovery capabilities, triggering early warnings and automatic optimization strategies when performance indicators exceed normal ranges.

[0130] Furthermore, digital twin-based system modeling technology can be used to construct a virtual mirror of the entire video analytics system. Simulations can then predict system performance under different configurations, providing more accurate guidance for parameter optimization. Digital twin models include hardware performance models, software behavior models, and workload models, enabling various optimization experiments to be conducted without affecting the actual system operation.

[0131] Application examples of this implementation method

[0132] To verify the effectiveness of the method of the present invention, a 30-day field test was conducted on the PCB board quality inspection production line of a large electronics manufacturing enterprise. During the test, the multi-computing power collaborative video analysis system of the present invention was deployed, equipped with 12 4K industrial cameras, covering the front, back and side inspection of PCB boards. The test area covered 3 production lines, with a daily processing volume of approximately 8,000 PCB boards. The test environment was a standard industrial workshop, with the temperature controlled at 22±2℃, the humidity controlled at 45%~65%RH, and the lighting conditions being flicker-free LED industrial lighting.

[0133] Examples of real-time video data acquisition are shown in Table 1:

[0134] Table 1: Example of real-time video data acquisition;

[0135]

[0136] Table 2 shows an example of how AI detection result data is obtained.

[0137] Table 2: Example of AI detection result data acquisition;

[0138]

[0139] Technical effect verification

[0140] like Figure 2 As shown, this invention demonstrates the significant advantages of its method over traditional solutions in terms of latency. Traditional single-processor solutions achieve an end-to-end latency of 156 milliseconds, which is insufficient for real-time detection requirements. Simple multi-processor solutions reduce the latency to 89 milliseconds through parallel processing, but still have considerable room for improvement. The multi-computing power collaborative method of this invention further reduces the end-to-end latency to 42 milliseconds, a 73% improvement over traditional solutions, fully meeting the stringent requirements of industrial real-time detection. The optimization effect is particularly significant in terms of inference latency and data transmission latency.

[0141] like Figure 3 As shown, the advantages of the method of this invention in various performance indicators are demonstrated. Compared with traditional solutions, the method of this invention shows significant improvements in VPU utilization, RK3588 utilization, memory utilization, cache hit rate, communication efficiency, and overall efficiency. In particular, the communication efficiency has increased from 40% to 92%, demonstrating the effectiveness of cross-processor optimization technology. The overall efficiency has increased from 52% to 87%, indicating that the multi-computing power collaboration mechanism can fully utilize the hardware potential.

[0142] like Figure 4 The figure shows the trend of the system's detection accuracy during 8 hours of continuous operation. It can be seen that with the continuous optimization of the self-learning mechanism, the accuracy of each detection task shows a steady upward trend. The defect detection accuracy increased from the initial 96.2% to 98.2%, the dimensional measurement accuracy increased from 94.5% to 97.6%, and the assembly inspection accuracy increased from 97.8% to 99.2%. This demonstrates that the adaptive optimization mechanism of this invention can continuously improve system performance and achieve a continuous improvement in accuracy during actual operation.

[0143] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A video structured analysis method using multi-computing power collaboration, characterized in that, Includes the following steps: Acquire multiple raw video streams and decode them synchronously. Gradient analysis is used to assess the quality of the decoded multi-channel original video streams, obtain quality feature vectors, and select a combination of preprocessing algorithms based on the quality feature vectors to obtain enhanced video frame data. Based on the enhanced video frame data, a data transmission channel is established, and an asynchronous communication protocol and a multi-level caching system are used to obtain a cross-processor data communication stream. Perform content complexity analysis on cross-processor data communication flows, and obtain scheduling decisions based on model matching according to the content complexity analysis; Based on the scheduling decision, the model is distributed and deployed, and pipelined inference is performed on the deployed model to obtain the pipelined inference results. The pipeline inference results are fused, and the consistency constraint verification of the fused results is performed to obtain the video structured analysis results; The system configuration parameters are optimized based on the video structure analysis results to achieve millisecond-level detection response and micron-level defect identification in video structure analysis. The method of using an asynchronous communication protocol and a multi-level caching system to obtain cross-processor data communication flow includes: A dedicated DMA data transfer channel is established between the 18TOPS VPU and the 6TOPS RK3588 using a PCIe high-speed channel and shared memory mechanism. Establish a dedicated DMA channel between processors and use hardware-level data transfer to avoid CPU copy overhead; The shared memory region is divided into multiple circular buffers, and the producer-consumer pattern is used to realize data exchange between computing nodes. It adopts an event-driven asynchronous communication protocol and coordinates data transmission and task execution through a message queue mechanism; Establish a multi-level caching system, including internal processor cache, inter-processor shared cache, and system-level data cache; The distributed deployment of the model based on scheduling decisions includes: Based on AI model structure analysis, a computational graph segmentation algorithm is used to determine the model's distribution and deployment scheme on VPU and RK3588; Choose inference models of different precision based on the complexity of the video content; use lightweight models for simple scenarios and high-precision models for complex scenarios. A strategy combining model parallelism and data parallelism is adopted to split large models into multiple computing nodes for parallel execution; Establish a model caching mechanism to preload frequently used models into the high-speed cache and reduce model switching latency; The pipelined inference process for the deployed model includes: Based on the principle of computational load balancing, the model partitioning point is determined, and computationally intensive operations are allocated to the 18 TOPS VPU, while control logic is allocated to the 6 TOPS RK3588 processor. A double buffering mechanism with input and output buffers is adopted to enable each computing node to work continuously and avoid waiting time; The large inference task is decomposed into fine-grained computational tasks at the millisecond level, and a strategy combining data parallelism and model parallelism is adopted. A semaphore-based synchronization mechanism is adopted to ensure that parallel tasks are completed in the correct order through precise timing control.

2. The video structured analysis method with multi-computing power collaboration according to claim 1, characterized in that, The multiple original video streams decoded using gradient analysis quality assessment include: Grayscale images are obtained by converting the original video frame sequence to grayscale. Calculate the horizontal and vertical gradients of a grayscale image, and calculate the sharpness score based on the gradient variance; Perform frequency domain transformation on grayscale images, analyze the energy distribution of high-frequency components, and calculate noise level scores; Obtain the brightness distribution of the grayscale image, calculate the brightness histogram entropy value, and obtain the illumination quality score; The sharpness score, noise score, and illumination score are weighted and summed according to preset weighting coefficients to obtain the quality feature vector.

3. The video structured analysis method with multi-computing power collaboration according to claim 1, characterized in that, The process of obtaining scheduling decisions through model matching based on content complexity analysis includes: The video content complexity index is calculated using an integral image texture analysis algorithm; The AI ​​model type and computational requirements are determined by using a decision table lookup based on the content complexity index. A weighted scoring algorithm is used to calculate the task processor matching degree; Obtain the real-time load status of multiple computing nodes and calculate the comprehensive load index of each node; Based on the comprehensive load index and the computational complexity of the AI ​​inference model, a load balancing strategy is adopted to distribute inference tasks to each computing node.

4. The video structured analysis method with multi-computing power collaboration according to claim 1, characterized in that, The combination of preprocessing algorithms based on quality feature vector selection includes: Obtain the noise score from the quality feature vector; Based on noise scoring, a Gaussian filtering noise reduction algorithm was selected to process the video frame data; Obtain the illumination score from the quality feature vector; Based on the illumination score, a contrast-limited adaptive histogram equalization algorithm is selected for illumination compensation. Detect the overall contrast of video frame data, and select a histogram equalization algorithm to enhance contrast based on the contrast score; The selected preprocessing algorithms are combined in order of priority to form a preprocessing algorithm sequence for the current video frame.

5. The video structured analysis method with multi-computing power collaboration according to claim 1, characterized in that, The consistency constraint verification of the fusion results includes: Based on the inference output of each AI model, a standardized set of detection results is obtained using the result aggregation module of the 6TOPS RK3588 processor. A weighted average method is used to fuse the inference results of multiple computing nodes, with the weights determined based on the historical accuracy of each node; Establish a confidence assessment mechanism to quantitatively evaluate the reliability of inference results; When a conflicting result is detected, arbitration is carried out using a voting mechanism or an expert system. The fused results are post-processed, including nonmaximum suppression and result smoothing.

6. The video structured analysis method with multi-computing power collaboration according to claim 1, characterized in that, The optimization of system configuration parameters based on video structured analysis results includes: Based on the system operation process, a multi-dimensional performance monitoring mechanism is adopted to obtain comprehensive performance statistics. The buffer size is dynamically adjusted according to the video resolution and frame rate to ensure smooth data transmission; Based on processor temperature and power consumption monitoring, dynamically adjust processor frequency and voltage; An adaptive threshold adjustment mechanism is adopted to dynamically adjust the algorithm parameters according to the detection accuracy requirements; Establish a performance monitoring and feedback mechanism to optimize the overall system performance in real time.

7. A computer-readable storage medium, characterized in that, It is used to store computer-readable instructions, which, when read by a computer, enable the execution of the multi-computing power collaborative video structured analysis method as described in any one of claims 1-6.

Images