An artificial intelligence application software development system for computer audiovisual software

Abstract

The application relates to the technical field of computer audio-visual software, in particular to an artificial intelligence application software development system. The system comprises a fusion unit, a scheduling unit, an optimization unit, a monitoring unit and a visual configuration unit. The fusion unit is used for acquiring multi-source audio-visual data, pre-processing and feature extraction, and determining a target perception area based on time sequence correlation analysis. The scheduling unit is used for task decomposition and adaptive load balancing scheduling of the perception task of the specified area. The optimization unit is used for pruning, quantization compression, operator adaptation and performance testing of an algorithm model. The monitoring unit is used for runtime monitoring, performance bottleneck positioning, abnormal behavior identification and generation of an optimization strategy for dynamic model updating. The visual configuration unit is used for visual display of a development process, a deployment topology, a running state and optimization records. The application solves the problems of limited multi-modal data fusion processing capability, poor cross-hardware platform adaptability, low model deployment automation degree and insufficient runtime monitoring and fault diagnosis capability.

Description

Technical Field

[0001] This invention relates to the field of computer audiovisual software development technology, and more specifically, to an artificial intelligence application software development system for computer audiovisual software. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, computer vision and audiovisual software, as an important carrier of human-computer interaction, has been widely used in fields such as intelligent monitoring, autonomous driving, medical image analysis, and industrial defect detection. By acquiring, processing, and analyzing multimodal data such as images, videos, and audio in real time, computer vision and audio software can achieve functions such as target recognition, behavior understanding, and scene perception, providing crucial perceptual capabilities for intelligent systems. In recent years, breakthroughs in deep learning technology have significantly improved the accuracy and real-time performance of vision and audiovisual algorithms, driving the large-scale deployment of AI application software across various industries.

[0003] At the software development level, to reduce development costs and improve R&D efficiency, the industry is gradually exploring the use of modular components, visual programming frameworks, and automated deployment tools to build artificial intelligence application software. By standardizing and encapsulating key elements such as algorithm models, hardware drivers, and data interfaces, developers can quickly integrate visual processing modules, auditory processing modules, and decision-making reasoning modules on a unified platform to form complete audiovisual application solutions. Meanwhile, the maturity of cloud-edge collaborative computing architecture provides more flexible computing power allocation schemes for artificial intelligence application software, enabling edge devices to undertake audiovisual reasoning tasks with high real-time requirements.

[0004] However, existing AI application software development systems for computer vision and hearing still have many shortcomings. Current software development platforms have limited capabilities for fusing and processing multimodal data, and the lack of efficient collaboration mechanisms between the visual and auditory modules makes it difficult for the system to fully utilize the complementarity of visual and auditory information for comprehensive perception and decision-making. In complex application scenarios, single-modal perception results often result in false detections or missed detections, and existing systems fail to provide effective multi-source information fusion and confidence verification methods, affecting overall perception reliability. Furthermore, existing AI application software development systems lack adaptability to different hardware platforms. When application scenarios migrate from the cloud to edge terminals, it is often necessary to redevelop or significantly modify the software code for specific hardware, leading to extended development cycles and increased maintenance costs. Simultaneously, existing development systems have low automation in the model deployment stage. Optimization operations such as model compression, quantization, and acceleration require manual completion by developers, demanding not only high levels of expertise but also difficulty in ensuring consistency and reproducibility across different projects.

[0005] Therefore, how to build an artificial intelligence application software development system that can support efficient fusion of multimodal data, adapt to multiple hardware platforms, automate model deployment, and intelligent operation and maintenance monitoring has become an urgent technical problem to be solved in the field of computer vision and audiovisual software. Summary of the Invention

[0006] In view of this, the present invention proposes an artificial intelligence application software development system for computer audiovisual software, which aims to solve the problems of limited multimodal data fusion processing capabilities, poor cross-hardware platform adaptability, low degree of automation in model deployment, and insufficient runtime monitoring and fault diagnosis capabilities in the existing technology.

[0007] This invention proposes an artificial intelligence application software development system for computer audiovisual software, comprising: The fusion unit is configured to acquire multi-source audiovisual raw data and perform preprocessing and feature extraction. The fusion unit is also configured to determine the target perception region based on multimodal feature data from different time phases.

[0008] The scheduling unit, electrically connected to the fusion unit, is configured to perform real-time scheduling of audiovisual perception tasks within a designated area.

[0009] The optimization unit is electrically connected to the fusion unit and the scheduling unit respectively. The optimization unit is configured to determine the target perception area as a specified task and send it to the scheduling unit. The optimization unit is also configured to perform compression quantization, operator adaptation, and performance testing on the adapted model.

[0010] The monitoring unit is electrically connected to the optimization unit. The monitoring unit is configured to perform runtime monitoring, performance bottleneck location, and abnormal behavior identification based on the model's running status information. The monitoring unit is also configured to generate model tuning strategies based on the monitoring results and dynamically adjust the algorithm model according to the tuning strategies.

[0011] The visualization configuration unit is electrically connected to the monitoring unit. The visualization configuration unit is configured to visualize the development process, deployment configuration, running status, and optimization records of the algorithm model based on a unified development interface.

[0012] Furthermore, the fusion unit is used to acquire multi-source audiovisual raw data and perform preprocessing and feature extraction. When determining the target perception region based on multimodal feature data from different time phases, it includes: The fusion unit also includes a format standardization processing module, which is configured to perform format unification, decoding, and spatiotemporal alignment processing on raw audiovisual data from different sensors such as cameras and microphones.

[0013] The fusion unit also includes a feature extraction module, which is configured to extract feature vectors of each modality based on a preset audiovisual feature model library.

[0014] The fusion unit also includes a temporal correlation analysis module, which is configured to perform temporal correlation analysis on feature vectors of different time phases and determine key sensing areas based on feature changes and perception confidence.

[0015] Furthermore, the format standardization processing module performs format unification, decoding, and spatiotemporal alignment processing on raw audiovisual data from different sensors such as cameras and microphones, including: The format standardization processing module is configured to decode the video stream and generate a frame sequence with a preset frame rate and resolution.

[0016] The format standardization processing module is configured to decode the audio stream and generate audio waveform data with a preset sampling rate.

[0017] The format standardization processing module is configured to perform spatiotemporal alignment of video frame sequences and audio waveform data according to a unified timestamp, wherein the time alignment deviation meets the preset synchronization accuracy requirements.

[0018] Furthermore, when the feature extraction module extracts feature vectors for each modality based on a preset audiovisual feature model library, it includes: The feature extraction module is configured to load a pre-trained visual feature extraction model and extract the visual feature vectors corresponding to each video frame.

[0019] The feature extraction module is configured to load a pre-trained auditory feature vector model and extract the auditory feature vectors corresponding to each audio segment.

[0020] The feature extraction module is configured to fuse visual and auditory feature vectors based on feature fusion weights in the visual and auditory feature model library to generate multimodal fused feature vectors.

[0021] Furthermore, the temporal correlation analysis module performs temporal correlation analysis on feature vectors from different time phases. When determining key sensing areas based on feature changes and perception confidence, it includes: The time series correlation analysis module is configured to calculate the characteristic change ΔF between adjacent time phases, where the formula for calculating the characteristic change ΔF is: Where F_t represents the multimodal fusion feature vector of the t-th time phase, and T represents the time window length.

[0022] The time series correlation analysis module is configured to determine the feature change amount based on a preset change amount threshold. When ΔF is greater than the preset change amount threshold, the corresponding region is determined to be a region with significant change.

[0023] The temporal correlation analysis module is also configured to filter key sensing regions based on the perceived confidence C of regions with significant changes. The formula for calculating the perceived confidence C is as follows: Where α and β are preset fusion weight coefficients, ΔFmax is the preset maximum feature change, Svalid is the effective sensing area, and Sttotal is the total sensing area.

[0024] The temporal correlation analysis module is configured to identify regions with a perception confidence C greater than a preset confidence threshold as key perception regions and output the location coordinates and spatial range information of the key perception regions.

[0025] Furthermore, when the scheduling unit performs real-time scheduling of audiovisual perception tasks within a designated area, it includes: The scheduling unit also includes a task decomposition module, which is configured to decompose complex perception tasks into several sub-task units.

[0026] The scheduling unit also includes an adaptive load balancing module, which is configured to allocate resources based on the real-time computing power metrics of each computing node.

[0027] The scheduling unit also includes a task scheduling and distribution module, which is configured to select the optimal transmission path based on network conditions.

[0028] The scheduling unit also includes a dynamic task migration module, which is configured to migrate tasks based on their execution status to ensure real-time performance.

[0029] Furthermore, when the task decomposition module breaks down a complex perception task into several sub-task units, it includes: The task decomposition module is configured to divide tasks into hierarchical levels based on the type and complexity of the perceived tasks, and generate a task decomposition tree structure.

[0030] The task decomposition module is configured to determine the execution order based on the data dependencies between each subtask unit and generate a task scheduling topology sequence.

[0031] The task decomposition module is configured to calculate the computational cost W_i and the estimated execution time T_i for each subtask unit, where the computational cost W_i is determined based on the number of model parameters and the amount of input data.

[0032] Furthermore, when the adaptive load balancing module allocates resources based on the real-time computing power metrics of each computing node, it includes: The adaptive load balancing module is configured to collect the CPU utilization U_cpu, GPU utilization U_gpu, memory usage M, and network bandwidth B of each computing node in real time.

[0033] The adaptive load balancing module is configured to calculate a comprehensive load score L based on the real-time computing power metrics of each computing node. The formula for calculating the comprehensive load score L is as follows: Where w_1, w_2, w_3, and w_4 are preset weight coefficients, M_max is the maximum memory capacity, and B_total is the total network bandwidth.

[0034] The adaptive load balancing module is configured to determine the task allocation priority of each computing node based on the overall load score L, and assign subtasks to the node with the lowest overall load score for execution.

[0035] Furthermore, the dynamic task migration module performs task migration based on the execution status, ensuring real-time requirements, including: The dynamic task migration module is configured to monitor the task execution status of each computing node in real time, and trigger the task migration condition when the load of a node exceeds the preset load threshold.

[0036] The dynamic task migration module is configured to calculate the task migration cost C_mig, which includes data transfer cost and task restart cost.

[0037] The dynamic task migration module is configured to determine whether to perform task migration based on task migration overhead and real-time requirements. When the task execution delay exceeds a preset delay threshold, task migration is forced.

[0038] Furthermore, when the optimization unit performs compression quantization, operator adaptation, and performance testing on the adapted model, it includes: The optimization unit also includes a model pruning module, which is configured to remove redundant network parameters.

[0039] The optimization unit also includes a quantization module, which is configured to convert floating-point parameters into low-ratio fixed-point parameters.

[0040] The optimization unit also includes an operator adaptation module, which is configured to rewrite operators according to the target hardware platform.

[0041] The optimization unit also includes a performance testing module, which is configured to perform comprehensive testing on the optimized model.

[0042] Furthermore, when the model pruning module removes redundant network parameters, it includes: The model pruning module is configured to perform structured pruning on the model according to a preset compression ratio, removing preset pruning channels.

[0043] The model pruning module is configured to calculate the number of model parameters after pruning.

[0044] The model pruning module is configured to fine-tune the pruned model and restore its accuracy.

[0045] Furthermore, when the quantization module converts floating-point parameters to low-ratio specific-point parameters, it includes: The quantization processing module is configured to map 32-bit floating-point parameters to b-bit fixed-point parameters according to a preset quantization bit width b, wherein the quantization mapping function Q(·) satisfies: Where x is the original floating-point parameter, and x_max is the maximum value of the parameter.

[0046] The quantization module is configured to calculate the volume of the quantized model. ,in byte.

[0047] Furthermore, when the performance testing module performs comprehensive testing on the optimized model, it includes: The performance testing module is configured to measure the inference latency of the model on the target hardware platform. Inference delay This includes model loading time, data preprocessing time, and inference computation time.

[0048] The performance testing module is configured to measure the model's memory usage on the target hardware platform. .

[0049] The performance testing module is configured to measure the throughput (FPS) of the model on the target hardware platform, where .

[0050] The performance testing module is configured to output inference latency. Memory usage Performance test report on throughput (FPS).

[0051] Furthermore, the monitoring unit performs runtime monitoring, performance bottleneck location, and abnormal behavior identification based on the model's running status information. It generates a model tuning strategy based on the monitoring results and dynamically adjusts the algorithm model according to the tuning strategy, including: The monitoring unit also includes a runtime data acquisition module, which is configured to continuously collect model running status data from each computing node.

[0052] The monitoring unit also includes a time-series statistical analysis module, which is configured to perform time-series statistical analysis on the operating status data and identify abnormal fluctuations.

[0053] The monitoring unit also includes a deviation detection module, which is configured to detect deviations based on a preset performance baseline and trigger performance alarms.

[0054] The monitoring unit also includes an abnormal behavior identification module, which is configured to detect anomalies in the model input data and identify potential adversarial sample attacks.

[0055] The monitoring unit also includes a tuning strategy generation module, which is configured to analyze performance alarm information and generate model tuning strategies.

[0056] The monitoring unit also includes a dynamic model update module, which is configured to dynamically update the algorithm model according to the tuning strategy.

[0057] The monitoring unit also includes an effect evaluation module, which is configured to evaluate the effect of the model adjustment and verify whether the optimization indicators meet the requirements.

[0058] Furthermore, when the runtime data acquisition module continuously collects model runtime status data from each computing node, it includes: The runtime data acquisition module is configured to use a preset sampling frequency. Collect inference latency, memory usage, CPU usage, and GPU usage data for each node.

[0059] The runtime data acquisition module is configured to build runtime feature sequences. ,in For the first The running state feature vector at each sampling time.

[0060] The runtime data acquisition module is configured to preprocess runtime status data, including missing value imputation and outlier removal.

[0061] Furthermore, the time-series statistical analysis module performs time-series statistical analysis on the operational status data to identify abnormal fluctuations, including: The time series statistical analysis module is configured to perform moving average processing on the runtime feature sequence R, and calculate the moving average μ(t): Where w is the length of the moving window.

[0062] The time series statistical analysis module is configured to calculate the volatility coefficient σ of the runtime characteristic sequence, where the formula for calculating the volatility coefficient σ is: The time series statistical analysis module is configured to determine that there is abnormal fluctuation in the operating status when the fluctuation coefficient σ exceeds the preset fluctuation threshold.

[0063] Furthermore, the deviation detection module performs deviation detection based on a preset performance baseline, and when a performance alarm is triggered, it includes: The deviation detection module is configured to store performance baseline data for each compute node. Including baseline inference delay Baseline memory usage and baseline throughput .

[0064] The deviation detection module is configured to calculate the deviation rate δ between real-time performance and baseline performance, where the formula for calculating the deviation rate δ is: Treal represents the real-time inference latency.

[0065] The deviation detection module is configured to trigger a performance alarm when the deviation rate δ exceeds a preset deviation threshold.

[0066] Furthermore, when the optimization strategy generation module analyzes performance alarm information and generates model optimization strategies, it includes: The optimization strategy generation module is configured to determine the location of performance bottlenecks based on performance alarm information and analyze the type and parameter scale of bottleneck operators.

[0067] The optimization strategy generation module is configured to obtain the historical inference latency distribution characteristics of the bottleneck operator and calculate the latency fluctuation coefficient λ. in, For the first time in history Secondary reasoning delay, This represents the average inference delay.

[0068] The optimization strategy generation module is configured to generate strategies based on latency fluctuation coefficients. Determine the stability level of the operator and generate corresponding tuning strategies based on the stability level. The tuning strategies include parameter quantization bit width adjustment strategy, operator replacement strategy and model retraining strategy.

[0069] Furthermore, when the dynamic model update module dynamically adjusts the algorithm model according to the tuning strategy, it includes: The dynamic model update module is configured to generate optimized model parameters based on the tuning strategy. It is also configured to perform canary deployments on the target computing nodes, updating the model incrementally according to the canary release strategy. Furthermore, the module is configured to continuously monitor the model's running status during the canary deployment period to ensure the new model runs stably.

[0070] The performance evaluation module assesses the effectiveness of the model adjustments and verifies whether the optimization metrics meet the requirements. This module collects the inference latency of the optimized model. Calculate the ratio of optimized inference latency to baseline inference latency. Judge the optimization effect. When If the value is less than or equal to the preset optimization threshold, the model update will be pushed to all computing nodes; if... If the value exceeds the optimization threshold, the optimization effect is deemed insufficient, triggering the generation of a new round of optimization strategies.

[0071] in, For the first time in history Secondary reasoning delay, This represents the average inference delay.

[0072] The optimization strategy generation module is configured to generate strategies based on latency fluctuation coefficients. Determine the stability level of the operator and generate corresponding tuning strategies based on the stability level. The tuning strategies include parameter quantization bit width adjustment strategy, operator replacement strategy and model retraining strategy.

[0073] Furthermore, when the dynamic model update module dynamically adjusts the algorithm model according to the tuning strategy, it includes: the dynamic model update module being configured to generate optimized model parameters according to the tuning strategy; the dynamic model update module being configured to perform canary deployment on the target computing node, gradually updating the model according to the canary release strategy; and the dynamic model update module being configured to continuously monitor the model's running status during the canary deployment period to ensure the stable operation of the new model.

[0074] Furthermore, the performance evaluation module assesses the effectiveness of the model adjustment and verifies whether the optimization metrics meet the requirements, including: the performance evaluation module is configured to collect the inference latency of the optimized model. Calculate the ratio of optimized inference latency to baseline inference latency. The effect evaluation module is configured to when When the value is less than or equal to the preset optimization threshold, the optimization metric is deemed to meet the requirements, and the model update is pushed to all computing nodes. The performance evaluation module is configured to... If the value exceeds the optimization threshold, the optimization effect is deemed insufficient, triggering the generation of a new round of optimization strategies.

[0075] Furthermore, when the visualization configuration unit visualizes the development process, deployment configuration, running status, and optimization records of the algorithm model based on the unified development interface, it includes: The visualization configuration unit also includes a development process management module, which is configured to display the progress of each stage of model development in a preset visualization format.

[0076] The visualization configuration unit also includes a deployment topology display module, which is configured to display the deployment of the model on each computing node in the form of a topology graph.

[0077] The visualization configuration unit also includes a real-time performance dashboard module, which is configured to display the changing trends of various performance indicators in real time.

[0078] The visualization configuration unit also includes a tuning record timeline module, which is configured to record historical tuning operations in the form of a timeline.

[0079] Furthermore, when the development process management module displays the progress of each stage of model development in a preset visual format, it includes: The development process management module is configured to define the set of task nodes for model development. Each task node includes data preparation, model training, model evaluation, and model deployment.

[0080] The development process management module is configured to calculate the progress percentage of each task node. According to the progress percentage Update the display status of task nodes.

[0081] The development process management module is configured to display the time schedule and progress information of each task node in the form of a Gantt chart.

[0082] Furthermore, when deploying the topology display module to show the model's deployment on each computing node in the form of a topology graph, it includes: The deployment topology display module is configured to build a compute node topology graph. ,in For the set of computation nodes, It is a set of connections between nodes.

[0083] The deployment topology display module is configured to mark the deployment information of each node in the topology diagram, including the model version number, deployment time, and running status.

[0084] The deployment topology display module is configured to dynamically update the color labels of nodes in the topology graph based on the running status of each node.

[0085] Furthermore, when the real-time performance dashboard module displays the changing trends of various performance indicators in real time, it includes: The real-time performance dashboard module is configured to display the inference latency as a curve over time in the form of a line graph.

[0086] The real-time performance dashboard module is configured to display the resource usage distribution of each compute node in the form of a bar chart.

[0087] The real-time performance dashboard module is configured to display the overall system throughput in a dashboard format.

[0088] Furthermore, when the tuning record timeline module records historical tuning operations in timeline format, it includes: The tuning timeline module is configured to record the timestamps of each tuning operation. Optimization strategy and optimization effect .

[0089] The tuning record timeline module is configured to be based on timestamps. Mark the optimization operation nodes on the timeline.

[0090] The tuning record timeline module is configured to support the query and backtracking functions of tuning records.

[0091] Compared with existing technologies, the beneficial effects of this invention are as follows: Through deep collaboration among the fusion unit, scheduling unit, optimization unit, monitoring unit, and visualization configuration unit, combined with key technologies such as time-series correlation analysis algorithms, adaptive load balancing mechanisms, model quantization and compression techniques, and runtime monitoring and diagnostics, it achieves full-process coverage from multimodal data fusion processing to cross-platform model deployment and intelligent operation and maintenance monitoring. This system effectively solves the technical problems in existing technologies, such as limited multimodal data fusion processing capabilities, poor cross-hardware platform adaptability, low automation of model deployment, and insufficient runtime monitoring and fault diagnosis capabilities, providing strong technical support for improving the development efficiency and reducing the operation and maintenance costs of artificial intelligence application software. Attached Figure Description

[0092] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A functional block diagram of an artificial intelligence application software development system for computer audiovisual software provided in an embodiment of the present invention; Figure 2This is a flowchart illustrating an artificial intelligence application software development system for computer audiovisual software, provided as an embodiment of the present invention. Detailed Implementation

[0093] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0094] The Multi-Modal Data Fusion Unit (MMDFU) is a processing unit used to acquire multi-source audiovisual raw data and perform preprocessing and feature extraction. It is used to determine the target perception region based on multimodal feature data from different time phases and to provide perception task input for subsequent heterogeneous computing scheduling. This unit is the core hub of multimodal audiovisual data processing.

[0095] The Format Standardization Processing Module (FSPM) is a functional module that performs format unification, decoding, and spatiotemporal alignment processing on raw audiovisual data from different sensors such as cameras and microphones. It is used to eliminate data format differences between different sensors and provide standardized data input for subsequent feature extraction.

[0096] The Feature Extraction Module (FEM) is a processing module that extracts feature vectors of each modality based on a pre-defined audiovisual feature model library. It is used to extract high-dimensional feature representations from pre-processed audiovisual data, providing a feature foundation for multimodal fusion.

[0097] The Temporal Correlation Analysis Module (TCAM) is an analysis module that performs temporal correlation analysis on feature vectors of different time phases and determines key perception areas based on feature changes and perception confidence. It is used to identify dynamically changing areas and key targets of interest in audiovisual scenes.

[0098] The Heterogeneous Computing Scheduling Unit (HCSU) is a control unit that performs real-time scheduling of audiovisual perception tasks within a specified area. It is used to realize the dynamic allocation of computing resources and task scheduling. This unit is electrically connected to the fusion unit, receives the perception tasks output by the fusion unit, and performs scheduling and distribution.

[0099] The Task Decomposition Module (TDM) is a processing module that decomposes complex perception tasks into several sub-task units. It is used to break down a complete multimodal perception task into sub-tasks that can be executed in parallel.

[0100] The Adaptive Load Balancing Module (ALBM) is a scheduling module that allocates resources based on real-time computing power metrics of each computing node. It is used to dynamically adjust task allocation strategies based on metrics such as CPU utilization, GPU utilization, memory usage, and network bandwidth of computing nodes.

[0101] The TaskSchedulingDispatchModule (TSDM) is a scheduling module that selects the optimal transmission path based on network conditions to ensure that sensing task data can be efficiently transmitted to the target computing node.

[0102] The Dynamic Task Migration Module (DTMM) is a control module that migrates tasks based on their execution status to ensure real-time performance. It is used to migrate tasks to other nodes when the compute node is overloaded or malfunctions.

[0103] The Optimization Unit (MDOU) is a processing unit that performs compression and quantization of the algorithm model, operator adaptation, and performance testing of the adapted model. It is electrically connected to the fusion unit and the scheduling unit to realize the automated deployment optimization of the model.

[0104] The ModelPruningModule (MPM) is an optimization module that removes redundant network parameters. It is used to perform structured pruning on deep learning models according to a preset compression ratio, thereby reducing the number of model parameters and computational complexity.

[0105] The Quantization Processing Module (QPM) is an optimization module that converts floating-point parameters into low-ratio fixed-point parameters to reduce model storage space and accelerate inference computation.

[0106] The Operator Adaptation Module (OAM) is an adaptation module that rewrites operators according to the target hardware platform, enabling the algorithm model to run efficiently on different hardware platforms.

[0107] The Performance Testing Module (PTM) is an evaluation module that performs comprehensive testing on the optimized model. It is used to verify the model's performance metrics such as inference latency, memory usage, and throughput on the target hardware platform.

[0108] The Intelligent Operation Monitoring Unit (IOMU) is a monitoring unit that performs runtime monitoring, performance bottleneck location, and abnormal behavior identification based on model running status information. It is used to generate model tuning strategies based on monitoring results and dynamically adjust the algorithm model according to the tuning strategies. This unit is electrically connected to the optimization unit.

[0109] The Runtime Data Acquisition Module (RDAM) is a module that continuously collects model running status data from each computing node. It is used to collect runtime metrics such as inference latency, memory usage, CPU usage, and GPU usage at a preset sampling frequency.

[0110] The Temporal Statistical Analysis Module (TSAM) is an analysis module that performs time-series statistical analysis on operational status data to identify abnormal fluctuations. It is used to detect operational anomalies through moving averages and fluctuation coefficient calculations.

[0111] The Deviation Detection Module (DDM) is a monitoring module that detects deviations based on a preset performance baseline and triggers performance alarms. It is used to monitor the deviation between the model performance and the baseline performance in real time.

[0112] The Anomaly Behavior Identification Module (ABIM) is a security module that detects anomalies in the model's input data and identifies potential adversarial attacks to ensure the model's operational security.

[0113] The TuningStrategyGenerationModule (TSGM) is a strategy module that analyzes performance alarm information and generates model tuning strategies. It is used to generate targeted optimization solutions based on the characteristics of performance bottlenecks.

[0114] The Dynamic Model Update Module (DMUM) is a deployment module that dynamically updates the algorithm model according to the tuning strategy. It is used for canary deployment and model updates on the target computing node.

[0115] The Effect Evaluation Module (EEM) is a quality module that evaluates the effectiveness of model adjustments and verifies whether the optimization indicators meet the requirements. It is used to determine whether the model optimization has achieved the expected goals.

[0116] A Visualization Configuration Unit (VCU) is a configuration unit that uses a unified development interface to visualize the development process, deployment configuration, running status, and optimization records of an algorithm model. This unit is electrically connected to the monitoring unit.

[0117] The Development Process Management Module (DPMM) is a management module that displays the progress of each stage of model development in a preset visual format. It is used to display the time schedule and progress information of task nodes such as data preparation, model training, model evaluation and model deployment in the form of Gantt charts.

[0118] The DeploymentTopologyDisplayModule (DTDM) is a module that displays the deployment of the model on each computing node in the form of a topology diagram. It is used to build a computing node topology diagram and label the deployment information of each node.

[0119] The RealTimePerformanceDashboardModule (RTPDM) is a dashboard module that displays the changing trends of various performance indicators in real time. It is used to display system performance in the form of line charts, bar charts, and dashboards.

[0120] The TuningRecordTimelineModule (TRTLM) is a timeline module that records historical tuning operations in a timeline format. It is used to record the timestamps, tuning strategies, and tuning effects of each tuning operation.

[0121] It is important to note that the data processing units such as the fusion unit, scheduling unit, or optimization unit in some embodiments of the present invention can all be used to analyze, fuse, schedule, and optimize audiovisual data, thereby achieving stable control of the system's operating status. Secondly, the data processing units such as the fusion unit, scheduling unit, or optimization unit can all be industrial servers, cloud computing platforms, embedded computing boards, GPU computing clusters, edge computing gateways, or software processing modules running on the aforementioned hardware platforms.

[0122] like Figures 1-2 As shown, in some embodiments of the present invention, this embodiment provides an artificial intelligence application software development system for computer audiovisual software, including: a fusion unit, a scheduling unit, an optimization unit, a monitoring unit, and a visualization configuration unit.

[0123] Specifically, the fusion unit acquires multi-source audiovisual raw data and performs preprocessing and feature extraction. Based on multimodal feature data from different time phases, the fusion unit determines the target perception region. Specifically, the fusion unit includes a format normalization module, a feature extraction module, and a temporal correlation analysis module.

[0124] Understandably, the fusion unit uses a format standardization module to unify, decode, and spatiotemporally align the raw audiovisual data from different sensors such as cameras and microphones, eliminating data format differences between the sensors. The format standardization module decodes the video stream, generating frame sequences with preset frame rates and resolutions, such as common 30fps, 60fps, and 1080P / 4K resolution video frame sequences. It also decodes the audio stream, generating audio waveform data with preset sampling rates, such as common 16kHz and 44.1kHz sampling rates. Finally, the format standardization module spatiotemporally aligns the video frame sequences and audio waveform data according to a unified timestamp, where time alignment deviations are accounted for. To meet the preset synchronization accuracy requirements, typically The time limit is no more than 50 milliseconds to ensure the temporal consistency of audiovisual data.

[0125] For example, taking a behavior recognition task in a smart surveillance scenario, the format normalization processing module receives a 1080P, 30fps video stream from a high-definition camera and a 16kHz audio stream from an array microphone. The module performs H.264 decoding on the video stream to generate a 30fps video frame sequence and PCM decoding on the audio stream to generate 16kHz audio waveform data. The module performs spatiotemporal alignment based on the timestamps of each data stream, controlling the time deviation within 30 milliseconds to meet the preset synchronization accuracy requirements.

[0126] Specifically, the feature extraction module extracts feature vectors of each modality based on a preset audiovisual feature model library and generates a multimodal fusion feature vector.

[0127] Understandably, the feature extraction module loads a pre-trained visual feature extraction model, such as a ResNet or VGG convolutional neural network model, to extract visual feature vectors corresponding to each video frame. The dimension of these visual feature vectors is... The dimension is typically 512 or 1024. The feature extraction module loads a pre-trained auditory feature vector model, such as a CNN or RNN-based audio feature extraction model, and extracts the auditory feature vectors corresponding to each audio segment. The dimension of the auditory feature vectors is 512 or 1024. The dimensions are typically 128 or 256. The feature extraction module fuses visual and auditory feature vectors based on feature fusion weights from the audiovisual feature model library to generate a multimodal fused feature vector. The fusion weights can be learned or manually set according to the application scenario; visual features usually have higher weights because video contains more information.

[0128] For example, the visual feature extraction model uses the ResNet-50 network structure, outputting a 2048-dimensional visual feature vector. After dimensionality reduction through a fully connected layer, a 512-dimensional visual feature vector is obtained. The auditory feature extraction model uses a CRNN network structure to output a 256-dimensional auditory feature vector. The feature fusion weights were set to visual weight 0.7 and auditory weight 0.3, resulting in a 512-dimensional multimodal fusion feature vector.

[0129] Specifically, the temporal correlation analysis module performs temporal correlation analysis on feature vectors of different time phases and determines key sensing areas based on feature changes and perception confidence.

[0130] Understandably, the time series correlation analysis module calculates the characteristic changes between adjacent time phases. Among them, characteristic change amount The calculation formula is: ; in, Indicates the first Multimodal fusion feature vectors of time phases This indicates the length of the time-series window, typically 5 to 10 frames. The time-series correlation analysis module determines the feature changes based on a preset change threshold. When the change exceeds a preset threshold, the corresponding region is identified as a region of significant change. The time-series correlation analysis module then determines the region of significant change based on the perceived confidence level. Key sensing areas were screened, including sensing confidence levels. The calculation formula is: ; in, and To preset the fusion weight coefficients, typically Take 0.6, Take 0.4, To preset the maximum feature change, To effectively perceive the area, This represents the total sensing area. The temporal correlation analysis module will determine the sensing confidence level. Areas exceeding a preset confidence threshold are identified as key sensing areas, and the location coordinates and spatial range information of these key sensing areas are output to the scheduling unit.

[0131] For example, setting the timing window length The preset change threshold is 0.15, and the preset maximum feature change is... fusion weight coefficient , The preset confidence threshold is 0.5. The temporal correlation analysis module calculates the feature changes between five adjacent frames. ,when A value greater than 0.15 is considered a region of significant change. Calculate the perceived confidence score. ,when When the value is greater than 0.5, the area is identified as a key sensing area, and its location coordinates are output. The spatial range information is sent to the scheduling unit to trigger subsequent sensing tasks.

[0132] Specifically, the scheduling unit is electrically connected to the fusion unit, and performs real-time scheduling of audiovisual perception tasks within a designated area. Understandably, the scheduling unit decomposes complex perception tasks into several sub-task units through a task decomposition module. The task decomposition module hierarchically divides the tasks based on their type and complexity, generating a task decomposition tree structure. For example, it decomposes object detection tasks into sub-tasks such as feature extraction, region proposal, category classification, and boundary regression. The task decomposition module determines the execution order based on the data dependencies between sub-task units, generating a task scheduling topology sequence to ensure that data flows between sub-tasks in the correct order. The task decomposition module calculates the computational cost of each sub-task unit. and estimated execution time The computational load The number of model parameters and the amount of input data are typically determined. The unit is FLOPs (floating-point operations).

[0133] For example, the object detection task is broken down into four sub-tasks: Feature extraction subtask (computational cost) GFLOPs, estimated execution time ms); Region suggestion subtask (computational cost) GFLOPs, estimated execution time ms); Category classification subtask (computational complexity) GFLOPs, estimated execution time ms); Boundary regression subtask (computational cost) GFLOPs, estimated execution time ms).

[0134] The task scheduling topology sequence is: feature extraction → region proposal → category classification → boundary regression.

[0135] Specifically, the adaptive load balancing module allocates resources based on the real-time computing power metrics of each computing node, assigning subtasks to the optimal computing node for execution. This means the adaptive load balancing module collects the CPU utilization of each computing node in real time. GPU utilization Memory usage and network bandwidth The adaptive load balancing module calculates a comprehensive load score based on the real-time computing power metrics of each computing node. Among them, the comprehensive load score The calculation formula is: in, , , , To preset weighting coefficients, typically , , , , Maximum memory capacity This represents the total network bandwidth. The adaptive load balancing module uses a comprehensive load score... Determine the task allocation priority for each computing node, and assign subtasks to the node with the lowest overall load score for execution to achieve load balancing.

[0136] For example, compute node A has a CPU utilization of 0.5, a GPU utilization of 0.6, a memory usage of 4GB (maximum 8GB), and a network bandwidth usage of 200Mbps (total bandwidth 1Gbps). The overall load score is calculated. The various indicators of node B are calculated as follows: The adaptive load balancing module assigns the subtask to node A for execution.

[0137] Specifically, the dynamic task migration module migrates tasks based on their execution status to ensure real-time performance. This means the module monitors the task execution status of each computing node in real time, triggering a task migration when the load on a node exceeds a preset load threshold (e.g., 80%). The dynamic task migration module also calculates the task migration cost. ,in This includes data transfer overhead and task restart overhead. Data transfer overhead is proportional to the amount of intermediate data to be migrated, while task restart overhead is proportional to the number of model parameters. The dynamic task migration module determines whether to perform task migration based on task migration overhead and real-time requirements. When the task execution delay exceeds a preset delay threshold (e.g., 100ms), task migration is forcibly performed to ensure real-time performance.

[0138] For example, if a node's current load is 85%, exceeding the preset load threshold of 80%, the task migration condition is triggered. The amount of intermediate data to be migrated is 100MB, the model parameters are 50MB, the network transmission rate is 100Mbps, the data transmission overhead is approximately 8 seconds, the task restart overhead is approximately 2 seconds, and the total migration overhead is... Approximately 10 seconds. The current task execution delay is 120ms, exceeding the preset delay threshold of 100ms, forcing a task migration.

[0139] Specifically, the optimization unit is electrically connected to the fusion unit and the scheduling unit respectively. The optimization unit determines the target perception area as the designated task and sends it to the scheduling unit. The optimization unit performs compression quantization, operator adaptation, and performance testing on the adapted model.

[0140] Understandably, the optimization unit removes redundant network parameters through the model pruning module, based on a preset compression ratio. The model undergoes structured pruning, removing pre-defined pruning channels, such as channels in convolutional layers with a small number of kernels. The model pruning module calculates the number of model parameters after pruning. ,in , These are the original model parameters. The model pruning module fine-tunes the pruned model to restore its accuracy, typically involving 1 to 5 epochs of fine-tuning training.

[0141] For example, the number of parameters in the original model M, set the compression ratio. Parameters after pruning M. The pruned model was fine-tuned for 3 epochs to restore the model's accuracy to more than 95% of the original accuracy.

[0142] Specifically, the quantization module converts floating-point parameters into low-ratio fixed-point parameters, reducing model storage and accelerating inference. Understandably, the quantization module uses a preset quantization bit width... Map the 32-bit floating-point parameter to Bit-fixed-point parameters, where the quantization mapping function satisfy: ; in, These are the original floating-point parameters. The maximum value of the parameters is usually taken as the maximum absolute value of the parameters in each layer. The quantization module calculates the volume of the quantized model. ,in byte.

[0143] For example, setting the quantization bit width This maps 32-bit floating-point parameters to 8-bit fixed-point parameters. The original 32-bit model size was 400MB, the pruned model size was 50MB, and the quantized model size was... MB, the size has been reduced to about 12.5% ​​of the original.

[0144] Specifically, the operator adaptation module rewrites operators based on the target hardware platform, enabling the model to run efficiently on different hardware. In other words, the operator adaptation module replaces operators in the model with efficient implementations for different hardware platforms, such as NVIDIA GPUs, Intel CPUs, ARM embedded processors, and NPU neural network processors. The operator adaptation module uses platform-specific operator libraries, such as CUDA, OpenCL, TensorRT, and NNAPI, for operator rewriting and optimization. For GPU platforms, convolution operators are replaced with cuDNN-accelerated versions; for CPU platforms, MKL-DNN or OpenVINO are used for acceleration; and for embedded platforms, NCNN or TFLite are used for inference acceleration.

[0145] For example, if the target hardware platform is an NVIDIA Jetson edge computing device, the operator adaptation module replaces the standard convolution operator in the model with the TensorRT-optimized INT8 convolution operator, replaces the fully connected layer with the TensorRT FullyConnected layer, and performs layer fusion optimization to reduce memory access overhead.

[0146] Specifically, the performance testing module performs comprehensive testing on the optimized model and outputs a performance test report. This means the performance testing module measures the model's inference latency on the target hardware platform. Inference delay This includes model loading time, data preprocessing time, and inference computation time. The performance testing module measures the model's memory usage on the target hardware platform. The performance testing module measures the throughput (FPS) of the model on the target hardware platform, where... The performance test module output includes inference latency. Memory usage Performance test report on throughput (FPS).

[0147] For example, the inference latency was measured when the optimized object detection model was tested on an NVIDIA Jetson AGX device. ms (including 5ms for model loading, 3ms for data preprocessing, and 17ms for inference computation), memory usage MB, throughput This meets the real-time requirements.

[0148] Specifically, the monitoring unit is electrically connected to the optimization unit. The monitoring unit performs runtime monitoring, performance bottleneck location, and abnormal behavior identification based on the model's running status information. It generates a model tuning strategy based on the monitoring results and dynamically adjusts the algorithm model according to the tuning strategy.

[0149] Understandably, the monitoring unit continuously collects model runtime status data from each computing node through the runtime data acquisition module. The runtime data acquisition module uses a preset sampling frequency. Collect inference latency, memory usage, CPU usage, and GPU usage data for each node, typically at a sampling frequency of... Set to 1Hz to 10Hz. The runtime data acquisition module constructs runtime feature sequences. ,in For the first The runtime status feature vector at each sampling time includes metrics such as inference latency, memory usage, CPU usage, and GPU usage. The runtime data acquisition module preprocesses the runtime status data, including missing value imputation and outlier removal, to ensure data quality.

[0150] For example, setting the sampling frequency Hz, meaning sampling every 200ms. The runtime feature sequence constructed by the runtime data acquisition module. It includes four dimensions: inference latency (in milliseconds), memory usage (in MB), CPU usage (percentage), and GPU usage (percentage).

[0151] Specifically, the time-series statistical analysis module performs time-series statistical analysis on runtime status data to identify abnormal fluctuations. In other words, the time-series statistical analysis module analyzes runtime characteristic sequences. Perform moving average processing and calculate the moving average. : ; in, The moving window length is typically 5 to 20 sampling points. The time series statistical analysis module calculates the fluctuation coefficient of the characteristic sequence during runtime. Among them, the volatility coefficient The calculation formula is: ; Time series statistical analysis module when fluctuation coefficient When the fluctuation exceeds a preset fluctuation threshold (e.g., 0.2), it is determined that there is an abnormal fluctuation in the operating status, and an abnormal alarm is triggered.

[0152] For example, setting the length of the movable window. The preset fluctuation threshold is 0.2. The time series statistical analysis module calculates the moving average of the most recent 10 sampling points. Calculate the volatility coefficient .when If abnormal fluctuations are detected in the operating status, an alarm message will be output.

[0153] Specifically, the deviation detection module detects deviations based on a preset performance baseline and triggers performance alarms. It can be understood that the deviation detection module stores the performance baseline data for each compute node. Including baseline inference delay Baseline memory usage and baseline throughput The deviation detection module calculates the deviation rate between real-time performance and baseline performance. Among them, the deviation rate The calculation formula is: ; in, For real-time inference latency. The deviation detection module adjusts the deviation rate. When the deviation exceeds a preset threshold (e.g., 20%), a performance alarm is triggered.

[0154] For example, baseline inference delay ms, current real-time inference latency ms, calculate the deviation rate If the deviation exceeds the preset threshold of 20%, a performance alarm will be triggered.

[0155] Specifically, the optimization strategy generation module analyzes performance alarm information and generates model optimization strategies. In other words, this module identifies performance bottlenecks based on the alarm information and analyzes the type and parameter size of the bottleneck operators. The module also obtains the historical inference latency distribution characteristics of the bottleneck operators and calculates the latency fluctuation coefficient. : ; in, For the first time in history Secondary reasoning delay, The average inference latency is calculated. The optimization strategy generation module uses the latency fluctuation coefficient as the basis. Determine the stability level of the operator and generate corresponding tuning strategies based on the stability level. The tuning strategies include parameter quantization bit width adjustment strategy, operator replacement strategy and model retraining strategy.

[0156] For example, the delay ripple coefficient of the bottleneck operator is calculated. The operator stability level is determined to be medium. The optimization strategy generation module generates a quantization bit width adjustment strategy, adjusting the quantization bit width from 8 bits to 6 bits to further compress the model. At the same time, it generates an operator replacement strategy, replacing unstable operators with more stable implementations.

[0157] Specifically, the dynamic model update module dynamically adjusts the algorithm model according to the tuning strategy. This means that the dynamic model update module generates optimized model parameters based on the tuning strategy. The module performs canary deployment on the target computing nodes, updating the model gradually according to the canary release strategy. For example, it first updates on 10% of the nodes, and after verifying stability, it gradually expands to 50%, then 100%. During the canary deployment, the dynamic model update module continuously monitors the model's running status to ensure that the new model is running stably before full deployment.

[0158] For example, the optimization strategy involves adjusting the quantization bit width to 6 bits, and the dynamic model update module generates the quantized model parameters. Initially, a canary deployment is performed on two computing nodes, and the operational status is monitored for 24 hours. Once stability is confirmed, the deployment is expanded to all 50 computing nodes.

[0159] Specifically, the performance evaluation module assesses the effectiveness of the adjusted model and verifies whether the optimization metrics meet the requirements. Understandably, the performance evaluation module collects the inference latency of the optimized model. Calculate the ratio of optimized inference latency to baseline inference latency. The effect evaluation module is used when... When the value is less than or equal to a preset optimization threshold (e.g., 0.8, indicating a latency reduction of more than 20%), the optimization metric is deemed to meet the requirements, and the model update is pushed to all computing nodes. The performance evaluation module... If the value exceeds the optimization threshold, the optimization effect is deemed insufficient, triggering the generation of a new round of optimization strategies.

[0160] For example, baseline inference delay ms, optimized inference latency ms, calculation If the value is less than the optimization threshold of 0.8, the optimization index is deemed to meet the requirements, and the model update is pushed to all computing nodes.

[0161] Specifically, the visualization configuration unit is electrically connected to the monitoring unit. The visualization configuration unit uses a unified development interface to visualize the development process, deployment configuration, running status, and optimization records of the algorithm model.

[0162] Understandably, the development process management module defines the set of task nodes for model development. Each task node includes data preparation, model training, model evaluation, and model deployment. The development process management module calculates the progress percentage of each task node. According to the progress percentage Update the display status of task nodes. The development process management module displays the time schedule and progress information of each task node in the form of a Gantt chart, enabling developers to intuitively understand the project progress.

[0163] For example, task node set The current progress of each node is 100%, 65%, 30%, and 0%, respectively. The development process management module displays the start and end times and progress of each task in the form of a Gantt chart.

[0164] Specifically, the deployment topology display module shows the model's deployment on each compute node in the form of a topology graph. In other words, the deployment topology display module constructs a compute node topology graph. ,in For the set of computation nodes, This is a set of connections between nodes. The deployment topology display module marks the deployment information of each node in the topology graph, including the model version number, deployment time, and running status. The deployment topology display module dynamically updates the color labels of the nodes in the topology graph according to the running status of each node; for example, green indicates normal operation, yellow indicates high load, and red indicates failure.

[0165] For example, the topology graph contains 50 compute nodes, each labeled with model version number v2.1, deployment date 2024-01-15, and normal running status. Node A's current load is 85%, exceeding the threshold, and this node is displayed in yellow in the topology graph.

[0166] Specifically, the real-time performance dashboard module displays the changing trends of various performance metrics in real time. Understandably, it shows the inference latency over time as a line graph, facilitating the monitoring of latency fluctuations. It also displays the resource usage distribution of each computing node as a bar chart, comparing CPU usage, GPU usage, and memory usage. Finally, it displays the overall system throughput in a dashboard format, reflecting the system's processing capacity in real time.

[0167] For example, the real-time performance dashboard displays a line graph of inference latency over the past hour, with an average latency of 22ms, a minimum latency of 18ms, and a maximum latency of 35ms. A bar chart shows the resource usage distribution across 50 nodes, with CPU averaging 55%, GPU averaging 70%, and memory averaging 60%. The dashboard shows an overall system throughput of 45 FPS.

[0168] Specifically, the tuning record timeline module records historical tuning operations in a timeline format. This means that the tuning record timeline module records the timestamps of each tuning operation. Optimization strategy and optimization effect This creates a complete historical optimization archive. The optimization record timeline module is based on timestamps. The timeline marks optimization operation nodes, which users can click to view detailed optimization information. The optimization record timeline module supports querying and backtracking of optimization records, allowing users to filter and search based on time range, optimization type, and other criteria.

[0169] For example, the tuning timeline records three tuning operations: the first time stamped on 2024-01-10, the tuning strategy was model pruning (compression ratio 0.5), and the tuning effect was a 25% reduction in latency; the second time stamped on 2024-01-12, the tuning strategy was quantization (8 bits), and the tuning effect was a 15% reduction in latency and a 60% reduction in size; the third time stamped on 2024-01-15, the tuning strategy was operator adaptation (TensorRT), and the tuning effect was a 10% reduction in latency.

[0170] To enable those skilled in the art to fully understand and implement this invention, the specific implementation principles of this invention are further supplemented below with a specific application scenario.

[0171] Step 1: Multimodal Data Acquisition and Preprocessing Stage. The fusion unit acquires and preprocesses the audiovisual data of the monitored area. The format standardization processing module receives a 1080P, 30fps video stream from a high-definition camera and a 16kHz audio stream from a microphone array, and decodes them to generate video frame sequences and audio waveform data. The format standardization processing module performs spatiotemporal alignment of the video frames and audio data according to a unified timestamp, with the time alignment deviation controlled within 50 milliseconds.

[0172] Step 2, Feature Extraction and Fusion Stage: The feature extraction module loads the pre-trained visual feature extraction model ResNet-50 and the auditory feature extraction model CRNN, extracting 512-dimensional visual feature vectors and 256-dimensional auditory feature vectors respectively. The feature extraction module fuses the two feature vectors according to preset fusion weights (0.7 for visual and 0.3 for auditory) to generate a 512-dimensional multimodal fused feature vector.

[0173] Step 3: Temporal correlation analysis and target perception region determination stage. The temporal correlation analysis module calculates the feature changes of adjacent time phases using 5 frames as a window. ,when A value greater than 0.15 is considered a region of significant change. The time-series correlation analysis module calculates the perceived confidence score. ,when When the value is greater than 0.5, the area is identified as a key sensing area, and its location coordinates are output. The spatial range information is sent to the scheduling unit.

[0174] Step four, the heterogeneous computing scheduling and task allocation phase, involves the scheduling unit's task decomposition module breaking down the perception task into sub-tasks such as feature extraction, target detection, and target tracking. The adaptive load balancing module collects load metrics from each computing node in real time and calculates a comprehensive load score. The subtasks are assigned to the nodes with the lowest load for execution. When a node's load exceeds 80%, the dynamic task migration module triggers task migration to ensure real-time requirements.

[0175] Step 5, Model Deployment and Optimization Phase: The optimization unit performs compression optimization on the target detection model. The model pruning module performs structured pruning at a compression ratio of 0.5, reducing the number of parameters from 100M to 50M. The quantization module maps 32-bit floating-point parameters to 8-bit fixed-point parameters, reducing the model size from 400MB to 50MB. The operator adaptation module replaces the operators with the optimized version of TensorRT. The performance testing module tests the inference latency. ms, memory usage MB, throughput This meets the real-time requirements.

[0176] Step Six: Intelligent Operation and Maintenance Monitoring Phase. The monitoring unit continuously collects model runtime status data, constructing a runtime feature sequence at a 5Hz sampling frequency. The time-series statistical analysis module calculates the moving average and fluctuation coefficient; an anomaly alarm is triggered when the fluctuation coefficient exceeds 0.2. The deviation detection module calculates the deviation rate between real-time performance and baseline performance; a performance alarm is triggered when the deviation rate exceeds 20%. The optimization strategy generation module analyzes the alarm information and generates a quantization bit width adjustment strategy. The dynamic model update module performs canary deployment, and the effect evaluation module verifies the optimization effect, pushing the optimized model to all nodes.

[0177] Step 7: Visualization Phase. The visualization configuration unit displays the system status based on the unified development interface. The development process management module displays the progress of each development task using a Gantt chart. The deployment topology display module displays the deployment status of 50 computing nodes using a topology diagram, with node colors reflecting the running status in real time. The real-time performance dashboard module displays performance indicators in the form of line charts, bar charts, and dashboards. The tuning record timeline module records historical tuning operations, supporting querying and backtracking.

[0178] In practical intelligent monitoring applications, when the system detects abnormal behavior within the monitored area, the fusion unit determines the target sensing area through time-series correlation analysis, and the scheduling unit allocates the sensing task to the optimal computing node for execution. The optimization unit performs compression and quantization optimization on the target detection model, enabling the model to run in real time on edge devices. The monitoring unit continuously monitors the model's running status, automatically generating tuning strategies and updating the model when performance degradation is detected. The visualization configuration unit presents the system status to operations and maintenance personnel in an intuitive way, providing data support for operational and maintenance decisions.

[0179] In autonomous driving scenarios, the system needs to process audiovisual data collected by onboard cameras and microphones in real time. The fusion unit fuses camera video and microphone audio to identify key targets in the driving scene. The scheduling unit allocates tasks to the GPU or NPU for execution based on real-time computing power, ensuring that perception latency meets safety requirements. The optimization unit compresses and optimizes models for target detection, lane recognition, and obstacle detection, enabling them to run in real time on the onboard computing platform. The monitoring unit monitors the model's running status and automatically triggers model updates when abnormal behavior or performance degradation is detected, ensuring driving safety.

[0180] In the above embodiments, by setting up a fusion unit, a scheduling unit, an optimization unit, a monitoring unit, and a visualization configuration unit, a complete technical chain of "multimodal data fusion—heterogeneous computing scheduling—model deployment optimization—intelligent operation and maintenance monitoring" is formed, realizing full-process coverage from visual and auditory data acquisition and processing to cross-platform model deployment and intelligent operation and maintenance monitoring. First, the fusion unit achieves efficient fusion of visual and auditory information and identifies key perception areas through temporal correlation analysis, effectively solving the problem of limited multimodal data fusion processing capabilities in existing technologies. Second, the scheduling unit's adaptive load balancing and dynamic task migration mechanism achieves dynamic allocation of computing resources, ensuring the real-time requirements of perception tasks and effectively solving the problem of poor cross-hardware platform compatibility. Third, the optimization unit's automated compression and quantization, operator adaptation, and performance testing achieve efficient cross-platform model deployment, effectively solving the problem of low automation in model deployment. Finally, the monitoring unit's runtime monitoring, performance bottleneck location, and dynamic model updates achieve intelligent operation and maintenance of the system, effectively solving the problem of insufficient runtime monitoring and fault diagnosis capabilities.

[0181] In the preferred embodiment described above, the deep collaboration between the fusion unit and the scheduling unit forms a complete closed loop for processing the perception task. The target perception area information output by the fusion unit provides task input to the scheduling unit, which deploys the optimized model to the computing nodes to execute the perception task. The monitoring unit monitors the task execution status and generates optimization strategies, which are then fed back to the optimization unit for model optimization, forming a closed loop of continuous iterative optimization.

[0182] In another preferred embodiment, the combined application of the visualization configuration unit and the monitoring unit forms a comprehensive operation and maintenance monitoring system. The real-time performance dashboard module displays system performance indicators in real time, enabling operation and maintenance personnel to intuitively understand the system's operating status. The tuning record timeline module records historical tuning operations, providing data support for problem troubleshooting and experience summarization. When performance anomalies are detected, operation and maintenance personnel can quickly locate the problem node through the visualization configuration unit, view historical tuning records to analyze the cause of the problem, and formulate targeted optimization solutions.

[0183] In summary, the artificial intelligence application software development system for computer audiovisual software according to embodiments of the present invention, through deep collaboration of fusion units, scheduling units, optimization units, monitoring units, and visualization configuration units, combined with key technologies such as time-series correlation analysis algorithms, adaptive load balancing mechanisms, model quantization compression technology, and runtime monitoring and diagnosis, achieves full-process coverage from multimodal data fusion processing to cross-platform model deployment and intelligent operation and maintenance monitoring. This system effectively solves the technical problems in existing technologies, such as limited multimodal data fusion processing capabilities, poor cross-hardware platform adaptability, low automation of model deployment, and insufficient runtime monitoring and fault diagnosis capabilities, providing strong technical support for improving the development efficiency and reducing the operation and maintenance costs of artificial intelligence application software.

[0184] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program goods. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program goods according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0186] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0187] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. An artificial intelligence application software development system for computer audiovisual software, characterized in that, include: The data fusion unit is used to acquire raw audiovisual data from multiple sources, and to preprocess and extract features from the raw audiovisual data to generate multimodal feature data. The data fusion unit is also used to perform correlation analysis based on multimodal feature data corresponding to different time points to determine the target perception area; The scheduling unit is connected in communication with the data fusion unit. The scheduling unit is used to generate task scheduling information according to the audiovisual perception task corresponding to the target perception area, and to call the corresponding computing resources to execute the audiovisual perception task according to the task scheduling information. The optimization unit is connected to the data fusion unit and the scheduling unit respectively. The optimization unit is used to generate corresponding task configuration parameters according to the target perception area and send the task configuration parameters to the scheduling unit. The optimization unit is also used to perform model compression, model quantization and computation operator adaptation on the artificial intelligence algorithm model, and to perform performance testing on the processed artificial intelligence algorithm model. The monitoring unit is connected in communication with the optimization unit. The monitoring unit is used to collect the status information during the operation of the artificial intelligence algorithm model, and to perform operation status monitoring, performance anomaly analysis and abnormal behavior identification based on the status information. The monitoring unit is also used to generate model adjustment strategies based on monitoring results, and to adjust the parameters or deployment of artificial intelligence algorithm models based on the model adjustment strategies; The visualization configuration unit communicates with the monitoring unit and is used to display the development process information, deployment configuration information, running status information, and adjustment record information of the artificial intelligence algorithm model through the development configuration interface.

2. The artificial intelligence application software development system for computer audiovisual software as described in claim 1, characterized in that, The fusion unit includes a format standardization processing module, a feature extraction module, and a temporal correlation analysis module; The format standardization processing module is used to acquire raw data from different audiovisual sensing devices and perform format conversion, data decoding and spatiotemporal synchronization processing on the raw data to generate standardized audiovisual data. The feature extraction module is connected to the format standardization processing module. The feature extraction module is used to extract features from the standardized audiovisual data according to the preset audiovisual feature model library, and obtain feature vectors corresponding to different data modalities respectively. The temporal correlation analysis module communicates with the feature extraction module. The temporal correlation analysis module is used to perform correlation analysis on feature vectors corresponding to different time nodes, determine key perception areas based on the change in feature vectors and perception confidence, and send the key perception areas to the optimization unit.

3. The artificial intelligence application software development system for computer audiovisual software as described in claim 2, characterized in that, The temporal correlation analysis module is used to perform correlation analysis on multimodal fusion feature vectors corresponding to different time points, and to determine key perception areas based on feature changes and perception confidence. This includes: The temporal correlation analysis module is used to obtain the multimodal fusion feature vectors corresponding to each time node within a preset time window, and to calculate the feature changes between adjacent time nodes; The time series correlation analysis module is also used to compare the feature change amount with a preset change amount threshold. When the feature change amount is greater than the preset change amount threshold, the time series correlation analysis module will identify the corresponding region as a region with significant change. The temporal correlation analysis module is also used to filter key sensing regions based on the sensing confidence scores corresponding to regions of significant change, including: The temporal correlation analysis module is used to identify areas with a perception confidence level greater than a preset confidence threshold as key perception areas, and outputs the location coordinates and spatial range information of the key perception areas.

4. The artificial intelligence application software development system for computer audiovisual software as described in claim 1, characterized in that, The scheduling unit includes a task decomposition module, an adaptive load balancing module, a task scheduling and distribution module, and a dynamic task migration module; The task decomposition module is used to obtain the audiovisual perception task corresponding to the specified area, and divide the audiovisual perception task into multiple sub-task units according to the task processing flow and computing requirements. The adaptive load balancing module communicates with the task decomposition module. The adaptive load balancing module is used to obtain the real-time computing power index of each computing node, and determine the target computing node corresponding to each sub-task unit based on the real-time computing power index, and allocate the sub-task unit to the corresponding computing node for execution. The task scheduling and distribution module is connected to the adaptive load balancing module. The task scheduling and distribution module is used to obtain network status information between computing nodes, determine the task transmission path based on the network status information, and send the sub-task units to the corresponding computing nodes. The dynamic task migration module communicates with the task scheduling and distribution module. The dynamic task migration module is used to obtain the execution status information of the sub-task unit and determine whether to perform task migration based on the execution status information. When the migration conditions are met, the corresponding sub-task unit is migrated to other computing nodes for execution.

5. The artificial intelligence application software development system for computer audiovisual software as described in claim 4, characterized in that, The task decomposition module is used to hierarchically divide audiovisual perception tasks according to their task type and complexity. When generating the task decomposition tree structure, it includes: The task decomposition module is also used to obtain the task type information and complexity information corresponding to the audiovisual perception task, and divide the audiovisual perception task into multiple sub-task units according to the task type information and complexity information, establish the hierarchical relationship between the sub-task units, and generate a task decomposition tree structure. The task decomposition module is also used to analyze the data dependencies between sub-task units, determine the execution order based on the correlation between input and output data between sub-task units, and generate a task scheduling topology sequence based on the execution order. The task decomposition module is also used to calculate the computational load and estimated execution time of each sub-task unit, and send the computational load and estimated execution time to the adaptive load balancing module.

6. The artificial intelligence application software development system for computer audiovisual software as described in claim 5, characterized in that, The adaptive load balancing module is used to allocate resources based on the real-time computing power metrics of each computing node, including: The adaptive load balancing module is also used to collect the running status information of each computing node, including CPU utilization, GPU utilization, memory usage, and network bandwidth usage. The adaptive load balancing module is also used to calculate the comprehensive load score of each computing node based on the running status information. The comprehensive load score is calculated by weighting the CPU utilization, GPU utilization, memory usage ratio, and network bandwidth usage ratio. The memory usage ratio is the ratio of the current memory usage to the maximum memory capacity, and the network bandwidth usage ratio is the ratio of the used network bandwidth to the total network bandwidth. The weight coefficients in the weighted calculation are determined according to the preset configuration. The adaptive load balancing module is also used to determine the task allocation priority based on the comprehensive load score L corresponding to each computing node, and to allocate sub-task units to computing nodes whose comprehensive load scores meet preset conditions for execution.

7. The artificial intelligence application software development system for computer audiovisual software as described in claim 6, characterized in that, The dynamic task migration module is used to migrate tasks based on the task execution status of compute nodes, including: The dynamic task migration module is used to collect task execution status information corresponding to each computing node in real time and compare the task execution status information with the preset load threshold. When the load of the computing node is detected to exceed the preset load threshold, a task migration trigger command is generated. The dynamic task migration module is used to calculate the task migration cost based on the amount of data corresponding to the task to be migrated and the initialization information during the task resumption process. The task migration cost includes task data transmission cost and task restart cost. The dynamic task migration module is used to determine the task migration strategy based on the task migration cost and task execution delay. When the task execution delay exceeds the preset delay threshold, the corresponding task will be migrated to other computing nodes for execution.

8. The artificial intelligence application software development system for computer audiovisual software as described in claim 1, characterized in that, The optimization unit includes a model pruning module, a quantization processing module, an operator adaptation module, and a performance testing module; The model pruning module is used to perform structured pruning of the artificial intelligence algorithm model according to a preset compression ratio, remove redundant network parameters in the artificial intelligence algorithm model, and determine the number of model parameters after pruning based on the number of original model parameters and the preset compression ratio; the model pruning module is also used to fine-tune the training of the pruned artificial intelligence algorithm model; The quantization processing module communicates with the model pruning module. The quantization processing module is used to convert the pruned model parameters from floating-point parameters to fixed-point parameters according to the preset quantization bit width, and to determine the storage capacity of the quantized model according to the quantization bit width and the number of pruned model parameters. The operator adaptation module is connected to the quantization processing module. The operator adaptation module is used to adapt the computational operators in the artificial intelligence algorithm model according to the computing architecture of the target hardware platform, and to rewrite or replace the mismatched computational operators. The performance testing module communicates with the operator adaptation module. The performance testing module is used to perform performance testing on the adapted artificial intelligence algorithm model on the target hardware platform, obtain model inference latency, memory usage and throughput parameters, and generate a performance test report.

9. The artificial intelligence application software development system for computer audiovisual software as described in claim 1, characterized in that, The monitoring unit includes a runtime data acquisition module, a time-series statistical analysis module, a deviation detection module, and an abnormal behavior identification module, among which: The runtime data acquisition module is used to collect model running status data corresponding to each computing node according to a preset sampling frequency; the runtime data acquisition module is used to construct a running status feature sequence based on the running status data collected at each sampling time, and to preprocess the running status data, including missing data supplementation and abnormal data removal; The time-series statistical analysis module communicates with the runtime data acquisition module. The time-series statistical analysis module performs time-series statistical analysis on the runtime state characteristic sequence. It calculates a moving average of the runtime state characteristic data according to a preset time window to obtain the average runtime state value at the corresponding time node. It also calculates the runtime state fluctuation coefficient based on the deviation ratio between the runtime state characteristic data at each sampling time and the corresponding average runtime state value. When the fluctuation coefficient of the running status exceeds the preset fluctuation threshold, the time series statistical analysis module determines that there is abnormal fluctuation in the running status of the model; The abnormal behavior recognition module is used to detect anomalies in the model input data and identify abnormal input behaviors based on the detection results. The deviation detection module is used to store the performance baseline data corresponding to each computing node. The performance baseline data includes the baseline inference latency, baseline memory usage, and baseline throughput. The deviation detection module is also used to calculate the deviation rate between the real-time performance parameters and the corresponding performance baseline data. When the deviation rate of the real-time inference latency relative to the baseline inference latency exceeds the preset deviation threshold, a performance alarm is triggered.

10. A system for developing artificial intelligence application software for computer audiovisual software as claimed in claim 9, characterized in that, The monitoring unit also includes a tuning strategy generation module, a dynamic model update module, and an effect evaluation module, among which: The optimization strategy generation module is used to determine the location of performance bottlenecks during model operation based on performance alarm information, and to analyze the type and parameter scale of the corresponding bottleneck operators. The optimization strategy generation module is used to obtain historical inference latency data of the bottleneck operator, calculate the latency fluctuation coefficient λ based on the degree of deviation between multiple historical inference latency data and the average inference latency, and determine the stability level of the bottleneck operator based on the latency fluctuation coefficient λ. The optimization strategy generation module is used to generate corresponding model optimization strategies based on the stability level. The model optimization strategies include parameter quantization bit width adjustment strategies, operator replacement strategies, and model retraining strategies. The dynamic model update module is used to generate updated model parameters according to the model tuning strategy, and deploy the updated model to the target computing node for canary deployment. During the canary deployment process, the running status data of the updated model is continuously collected. The performance evaluation module is used to obtain the inference latency of the updated model and calculate the ratio between the inference latency of the updated model and the baseline inference latency. When the ratio is less than or equal to the preset optimization threshold, the updated model is pushed to all computing nodes. When the ratio is greater than the preset optimization threshold, the effect evaluation module generates information about insufficient optimization and triggers the optimization strategy generation module to regenerate the model optimization strategy.

Images