Full-stack autonomous controllable artificial intelligence inference system based on boarding GPU

Abstract

The invention belongs to the technical field of artificial intelligence reasoning, and particularly relates to a full-stack autonomous controllable artificial intelligence reasoning system based on a boarding GPU. According to the invention, the GPU initialization and video decoding module is combined with VPU hardware acceleration, multithreading and callback mechanisms, real-time decoding and processing of large-scale video streams are realized, and the TVM quantitative training and model import optimization module effectively reduces the model operation complexity, improves the reasoning speed, maintains high precision, and improves the efficiency. The quantization and operator fusion technology reduces the consumption of calculation and storage resources, is suitable for edge calculation and low-power-consumption equipment deployment, and the whole system can be applied to scenes such as digital power plants, intelligent inspection and security monitoring, and meets the requirements of high concurrency, real-time response and high security.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence reasoning technology, and specifically relates to a full-stack autonomous and controllable artificial intelligence reasoning system based on GPU. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, deep learning has been widely used in computer vision, natural language processing and other fields. Especially in the field of intelligent inspection, the use of deep learning models for target detection and state recognition has become an important means to improve inspection efficiency and reduce labor costs.

[0003] At present, AI reasoning systems are mainly composed of GPU, CPU, storage, network and other modules, among which GPU undertakes the main deep learning computing tasks. In order to ensure reasoning performance, mainstream GPUs usually use GPU products from foreign manufacturers such as NVIDIA, such as Tesla A100 and H100.

[0004] At present, the AI ​​inference systems on the market mainly rely on foreign GPU architectures. Most of the existing AI inference systems use foreign GPUs such as NVIDIA and AMD. Their underlying hardware architecture and driver systems are not fully open, resulting in hidden dangers in domestic adaptation and security. They are highly dependent on the GPU architecture of specific manufacturers and are difficult to directly migrate to domestic GPUs. The inference system has high power consumption, especially in large-scale inference tasks. It is difficult to balance performance and energy efficiency. The adaptability to domestic GPUs is weak and additional optimization is required to achieve the best inference performance. Most of them are general computing architectures and are not optimized for specific scenarios such as new energy power plants and smart manufacturing, making it difficult to meet industry needs. Summary of the invention

[0005] The purpose of this invention is to provide a full-stack autonomous and controllable artificial intelligence reasoning system based on Denglin GPU, which can be deeply optimized for Denglin GPU, provide efficient and low-power AI reasoning capabilities, and promote the development of domestic AI computing ecology.

[0006] The technical solution adopted by the present invention is as follows: A full-stack autonomous and controllable artificial intelligence reasoning system based on Denglin GPU, including: GPU initialization module, used to load SDK and graphics card driver, call cudaSetDevice to select at least one available GPU, and call dlvidGetDevice to obtain the VPU device connected to the GPU; The video decoding module is used to call dlvidCreateSession to create a decoding session according to the format of the video stream to be decoded, and to establish the video input and output buffer queues by calling dlvidCreateBuffer, and then use dlvidRegisterBuffer to register the buffer queues as the input and output of the decoding session; The multi-channel video stream processing module uses the RTSP protocol to obtain the video stream data to be decoded from the database, and adopts a multi-threaded processing mechanism and dlvidProcessSessionEvent to obtain the callback function, and obtains the decoded YUV frame data in real time when the output buffer queue is full; The model quantization training module is used to automatically insert pseudo-quantization nodes during the model training process using the TVM quantization tool, quantize and dequantize the input and output of the convolutional layer and the fully connected layer, and adjust the quantization parameters based on the dynamic statistical weights and the minimum and maximum values ​​of the activations of the training samples to simulate the rounding and saturation errors generated during the quantization process; The model import and optimization module is used to import the trained model file into TVM, convert it into TVM's intermediate representation, and reorganize and optimize the computational graph using operator fusion optimization technology; The model evaluation module is used to evaluate the accuracy of the quantized model.

[0007] In a preferred embodiment, the GPU initialization module includes an initialization unit, a driver loading unit, a device selection unit, and a VPU device acquisition unit: Initialization unit, used to initialize the GPU device; Driver loading unit, used to load the SDK and graphics card driver required by the GPU GPU device selection unit, used to call cudaSetDevice to specify the GPU device used by the current task; The VPU device acquisition unit is used to acquire the VPU device associated with the GPU by using dlvidGetDevice.

[0008] In a preferred embodiment, the video decoding module includes a decoding session creation unit, a cache queue management unit and a decoding registration unit: The decoding session creation unit obtains the format of the video stream to be decoded, wherein the formats include H.254 and H.265, calls the dlvidCreateSession function and creates a decoding session according to the format of the video stream to be decoded, so as to initialize the decoder and configure decoding parameters.

[0009] The buffer queue management unit creates a video input buffer queue and an output buffer queue by calling dlvidCreateBuffer, which are used to store compressed video stream data and store YUV frames generated after decoding respectively; The decoding registration unit is used to call dlvidRegisterBuffer to register the input buffer queue and the output buffer queue to the decoding session.

[0010] In a preferred embodiment, the multi-channel video stream processing module includes a video stream acquisition unit, a video decoding management unit, a multi-thread scheduling unit and an event callback processing unit: The video stream acquisition unit obtains the video stream data to be decoded from the database based on the RTSP protocol; A video decoding management unit, configuring a decoder according to the format of a decoded video stream; A multi-thread scheduling unit, used to create multiple threads, each of which processes a video stream to be decoded; The event callback processing unit is used to call the dlvidProcessSessionEvent registered callback function. When the output buffer queue is full, the callback function is triggered to obtain and process the decoded YUV frame data in real time.

[0011] In a preferred embodiment, the multi-channel video stream processing module further includes a decoded image preprocessing unit: The decoded image preprocessing unit is used to preprocess the YUY frame data into an input format that conforms to the model. The preprocessing includes converting the YUV format into the RGB format, image cropping, image scaling, and normalization.

[0012] In a preferred embodiment, the model quantization training module includes a pseudo quantization node insertion unit, a quantization parameter calculation unit, a quantization and dequantization unit, an error simulation unit, a dynamic adjustment unit and a training optimization unit: A pseudo-quantization node insertion unit is used to obtain key positions, where the key positions include convolutional layers and fully connected layers. Pseudo-quantization nodes are inserted into the convolutional layers and fully connected layers to simulate the quantization process from floating points to integers. The quantization process includes rounding errors and saturation errors. A quantization parameter calculation unit, used to count the minimum and maximum values ​​of weights and activation values, and calculate the quantization scale and zero point based on the statistical results; The quantization and dequantization unit is used to quantize floating-point data into low-precision data and dequantize low-precision data into floating-point data during forward propagation and pass it to the next layer; The error simulation unit is used to introduce rounding error and saturation error in the pseudo-quantization node, and adjust the model parameters according to the quantization error during the back-propagation process.

[0013] Dynamic adjustment unit, used to recalculate the minimum and maximum values ​​of weights and activation values ​​in each training cycle, and update the quantization parameters according to the new statistical results; The training optimization unit is used to update the model parameters using the gradient descent algorithm and adjust the model weights according to the quantization error.

[0014] In a preferred embodiment, the model import and optimization module includes a model file import unit, an IR conversion unit, a computational graph optimization unit, an operator fusion unit, and a hardware adaptation unit: The model file import unit is used to import the trained model file into the TVM quantization tool and read the structure and parameters of the model. The model file formats include .pth and .pt. IR conversion unit, used to parse the computational graph and parameters of the imported model file and convert the computational graph into the intermediate representation of TVM; The computational graph optimization unit is used to call the optimizer of the TVM quantization tool to optimize the IR; The operator fusion unit is used to identify operators that can be fused in the computation graph and call the optimizer of the TVM quantization tool to fuse the operators that can be fused; The hardware adaptation unit is used to call the TVM quantization tool to generate code for the target hardware and adjust the execution strategy of the computation graph according to the characteristics of the target hardware.

[0015] In a preferred embodiment, the model evaluation module includes a data loading and preprocessing unit, a model reasoning unit, an accuracy evaluation unit, a comparative analysis unit, and a visualization and report generation unit: The data loading and preprocessing unit is used to load the test data set and preprocess the test data set to obtain a normalized test data set that matches the model input format; The model inference unit is used to call the quantized model, input the normalized test data set for inference, and generate prediction results; An accuracy evaluation unit calculates evaluation data of the quantitative model according to the prediction results, wherein the evaluation data includes accuracy, recall, F1 value and average precision mean; A comparative analysis unit is used to compare the model performance before and after quantization based on the evaluation data. The model performance includes inference speed, computing resource consumption, and accuracy change. The visualization and report generation unit is used to generate evaluation reports based on model performance. The evaluation reports include curve graphs, histograms, and confusion matrices.

[0016] In a preferred embodiment, the evaluation data includes accuracy, recall, F1 value and average precision.

[0017] And, a full-stack autonomous and controllable artificial intelligence reasoning terminal based on Denglin GPU, including: one or more processors; a storage device having one or more programs stored thereon; When one or more programs are executed by one or more processors, one or more processors can realize a full-stack autonomous and controllable artificial intelligence reasoning system based on the Denglin GPU.

[0018] The technical effects achieved by the present invention are: The present invention realizes real-time decoding and processing of large-scale video streams through GPU initialization and video decoding module combined with VPU hardware acceleration, multi-threading and callback mechanism. The TVM quantization training and model import optimization module effectively reduce the complexity of model operation and improve the reasoning speed while maintaining high precision. The quantization and operator fusion technology reduces the consumption of computing and storage resources, which is suitable for edge computing and low-power device deployment. The whole system can be applied to digital power plants, intelligent inspections, security monitoring and other scenarios to meet the requirements of high concurrency, real-time response and high security. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of the method provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0021] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. The phrase "in a preferred embodiment" that appears in different places in this specification does not refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0023] Secondly, the present invention is described in detail in conjunction with schematic diagrams. When describing the embodiments of the present invention in detail, for the convenience of explanation, the schematic diagrams are only examples and should not limit the scope of protection of the present invention.

[0024] Please refer to the attached Figure 1 As shown, a full-stack autonomous and controllable artificial intelligence reasoning system based on Denglin GPU is provided, including: GPU initialization module, used to load SDK and graphics card driver, call cudaSetDevice to select at least one available GPU, and call dlvidGetDevice to obtain the VPU device connected to the GPU; The video decoding module is used to call dlvidCreateSession to create a decoding session according to the format of the video stream to be decoded, and to establish the video input and output buffer queues by calling dlvidCreateBuffer, and then use dlvidRegisterBuffer to register the buffer queues as the input and output of the decoding session; The multi-channel video stream processing module uses the RTSP protocol to obtain the video stream data to be decoded from the database, and adopts a multi-threaded processing mechanism and dlvidProcessSessionEvent to obtain the callback function, and obtains the decoded YUV frame data in real time when the output buffer queue is full; The model quantization training module is used to automatically insert pseudo-quantization nodes during the model training process using the TVM quantization tool, quantize and dequantize the input and output of the convolutional layer and the fully connected layer, and adjust the quantization parameters based on the dynamic statistical weights and the minimum and maximum values ​​of the activations of the training samples to simulate the rounding and saturation errors generated during the quantization process; The model import and optimization module is used to import the trained model file into TVM, convert it into TVM's intermediate representation, and reorganize and optimize the computational graph using operator fusion optimization technology; The model evaluation module is used to evaluate the accuracy of the quantized model.

[0025] As mentioned above, the GPU initialization module first loads the software development kit (SDK) and graphics card driver required to log in to the GPU during the system startup phase to ensure that the hardware and software environment are in an available state. In a multi-GPU environment, the system automatically selects at least one available GPU by calling the cudaSetDevice function, thereby allocating appropriate hardware resources for subsequent computing tasks. The dlvidGetDevice function is used to obtain the video processing unit (VPU) device connected to the selected GPU, so that the video decoding task can use the VPU for hardware acceleration processing. The video decoding module calls the dlvidCreateSession function to create a dedicated decoding session according to the format of the video stream to be decoded (such as H.254 or H.265), completes the initialization and parameter configuration of the decoder, and calls the dlvidCreateBuffer function to establish a video input cache queue (for storing compressed video stream data) and an output cache queue (for storing decoded YUV frame data) respectively. The established cache queue is registered in the decoding session through the dlvidRegisterBuffer function to ensure that the input data can enter the decoding smoothly. The decoding result can be output to the specified cache in time. The multi-channel video stream processing module uses the RTSP protocol to obtain the video stream data to be decoded from the database or network camera, and uses the multi-threading mechanism to process multiple video streams simultaneously to improve the concurrency of the overall data processing. The dlvidProcessSessionEvent function is called to register the callback function. When the output cache queue reaches the full state, the callback function is triggered immediately to obtain the decoded YUV frame data in real time for subsequent processing (such as image preprocessing or inference). The model quantization training module uses T The VM quantization tool automatically inserts pseudo-quantization nodes at the input and output of the convolutional layer and the fully connected layer to simulate the rounding and saturation errors when the data is converted from floating point to integer during the actual quantization process. The pseudo-quantization nodes quantize the activation values ​​and weights that have passed through, and then dequantize them back to high precision so that the model can perceive the quantization error during forward propagation. During the training process, the minimum and maximum values ​​of the weights and activation values ​​of each layer are counted in real time, and the quantization parameters (such as quantization scale and zero point) are dynamically adjusted according to the statistical results to ensure that the quantization process retains the original accuracy as much as possible. The model import and optimization module converts the trained model file (such as .pth or .pt) is imported into the TVM framework and converted into the intermediate representation (IR) of TVM. The TVM optimizer is used to reorganize the computational graph in the IR. The operator fusion technology is used to merge continuous operations (such as convolution and activation functions) into an efficient operation to reduce memory transfer and redundant calculations. The model evaluation module uses the test data set to compare and evaluate the models before and after quantization, and calculates indicators such as accuracy, recall, F1 value, and mean average precision (mAP). The GPU initialization and video decoding module combines VPU hardware acceleration, multi-threading and callback mechanism to achieve real-time decoding and processing of large-scale video streams. The TVM quantization training and model import optimization module effectively reduces the model operation complexity and improves the inference speed while maintaining high accuracy. Quantization and operator fusion technology reduce the consumption of computing and storage resources, which is suitable for edge computing and low-power device deployment. The entire system can be applied to digital power plants, intelligent inspections, security monitoring and other scenarios to meet high concurrency, real-time response and high security requirements. .

[0026] It is worth mentioning that cudaSetDevice is used to specify the GPU device used when the current program is running. In a multi-GPU environment, by passing in the device ID, subsequent computing tasks are ensured to be executed on the correct GPU; dlvidGetDevice is used to obtain the VPU (video processing unit) device associated with the GPU, which helps to confirm the hardware resources available for video processing in the system; dlvidCreateSession is used to create a video decoding session according to the video stream format (such as H.254 or H.265), which is responsible for managing the initialization, parameter configuration and execution of the decoding process; dlvidCreateBuffer is used to create the input and output buffer queues required for video decoding. The input buffer is used to store compressed video stream data, and the output buffer is used to store the YUV frame data obtained after decoding; dlvidRegisterBuffer is used to register the created buffer queue to the decoding session to ensure that the decoding module can correctly read the input data and output the processing results to the corresponding buffer; dlvidProcessSessionEvent is used to register the callback function to handle events in the decoding session, such as triggering a callback when the output buffer queue is full to obtain the processed YUV frame data. This function is usually used for event processing and data acquisition in a multi-threaded environment.

[0027] In a preferred embodiment, the GPU initialization module includes an initialization unit, a driver loading unit, a device selection unit, and a VPU device acquisition unit: Initialization unit, used to initialize the GPU device; Driver loading unit, used to load the SDK and graphics card driver required by the GPU GPU device selection unit, used to call cudaSetDevice to specify the GPU device used by the current task; The VPU device acquisition unit is used to acquire the VPU device associated with the GPU by using dlvidGetDevice.

[0028] As mentioned above, when the system starts or the task starts, the initialization unit is responsible for the overall initialization of the GPU device, including the detection and configuration of hardware resources and the initialization of necessary internal data structures to ensure that the GPU device is in an operational state. During the initialization process, the driver loading unit is responsible for loading the SDK and graphics card driver required for the GPU to ensure the interface compatibility and normal communication between the software and the GPU hardware. In a multi-GPU environment, the GPU device selection unit calls the cudaSetDevice function through device status detection to automatically select and specify the GPU device used by the current task. The VPU device acquisition unit calls dlvidGetDevic e function, obtains the video processing unit (VPU) device connected to the currently selected GPU. The VPU is dedicated to video data processing and acceleration, and provides dedicated hardware support for tasks such as video decoding and transcoding. Each unit works together, from device initialization to driver loading, device selection, and then to VPU acquisition, to form an efficient and stable GPU hardware foundation. Through automated and intelligent hardware resource management, it ensures that the collaborative acceleration capabilities of the GPU and VPU are fully utilized, providing a solid foundation for subsequent video decoding and deep learning inference tasks. The standardized initialization and driver loading process reduces the risk of system failures caused by configuration errors or device abnormalities, and improves the reliability and operating efficiency of the overall system.

[0029] In a preferred embodiment, the video decoding module includes a decoding session creation unit, a cache queue management unit and a decoding registration unit: The decoding session creation unit obtains the format of the video stream to be decoded, wherein the formats include H.254 and H.265, calls the dlvidCreateSession function and creates a decoding session according to the format of the video stream to be decoded, so as to initialize the decoder and configure decoding parameters.

[0030] The buffer queue management unit creates a video input buffer queue and an output buffer queue by calling dlvidCreateBuffer, which are used to store compressed video stream data and store YUV frames generated after decoding respectively; The decoding registration unit is used to call dlvidRegisterBuffer to register the input buffer queue and the output buffer queue to the decoding session.

[0031] In the above, the decoding session creation unit obtains the format of the video stream to be decoded (such as H.254 and H.265), determines the decoding strategy, calls the dlvidCreateSession function, creates corresponding decoding sessions according to different video formats, completes the initialization of the decoder and the configuration of the decoding parameters, and the cache queue management unit calls the dlvidCreateBuffer function to create a video input cache queue and an output cache queue respectively. The input cache queue is used to store compressed video stream data, and the output cache queue is used to store the YUV frames generated after decoding. The decoding registration unit calls the dlvidRegisterBuffer function to register the input and output cache queues created above to the decoding session. After registration, the decoding session can directly access the data in the cache, realize the reception of input compressed stream data and the storage of output decoded YUV frames. Accurate session creation and parameter configuration enable the decoder to quickly adapt to different video formats. Reasonable cache queue management ensures continuous and stable data flow to meet real-time processing requirements. The registration mechanism ensures efficient interaction between the decoder and the cache, reduces latency, and improves system reliability.

[0032] In a preferred embodiment, the multi-channel video stream processing module includes a video stream acquisition unit, a video decoding management unit, a multi-thread scheduling unit and an event callback processing unit: The video stream acquisition unit obtains the video stream data to be decoded from the database based on the RTSP protocol; A video decoding management unit, configuring a decoder according to the format of a decoded video stream; A multi-thread scheduling unit, used to create multiple threads, each of which processes a video stream to be decoded; The event callback processing unit is used to call the dlvidProcessSessionEvent registered callback function. When the output buffer queue is full, the callback function is triggered to obtain and process the decoded YUV frame data in real time.

[0033] It should be noted that the multi-channel video stream processing module also includes a decoded image preprocessing unit: The decoded image preprocessing unit is used to preprocess the YUY frame data into an input format that conforms to the model. The preprocessing includes converting the YUV format into the RGB format, image cropping, image scaling, and normalization.

[0034] The above-mentioned video stream acquisition unit, based on the RTSP protocol, continuously obtains the video stream data to be decoded from the database to ensure that the data source is updated in real time. The video decoding management unit configures and initializes the decoder according to the obtained video stream format (such as H.254, H.265) to ensure that the appropriate decoding strategy and parameters are called. The multi-thread scheduling unit creates multiple threads, each thread independently processes a video stream to be decoded, and realizes parallel decoding and processing. The event callback processing unit registers the callback function using the dlvidProcessSessionEvent function, and automatically triggers the callback when the output cache queue is full, so as to obtain and process the decoded YUV frame data in time. The decoded image preprocessing unit processes the decoded YUV frame The data is preprocessed and converted into a format that meets the model input requirements. The preprocessing steps include: YUV to RGB, converting the video frame from YUV color space to RGB color space; image cropping, cropping according to the target area; image scaling, adjusting the image size to meet the model input size; normalization, normalizing the pixel values, unifying the data distribution, and realizing real-time and efficient processing of large-scale video data through multi-channel video stream acquisition, multi-threaded scheduling and event callback mechanism. The image after the preprocessing module meets the model requirements and provides reliable and high-quality data support for subsequent deep learning reasoning. The units work together to not only adapt to a variety of video formats and traffic fluctuations, but also can be flexibly expanded to meet larger-scale application scenarios.

[0035] In a preferred embodiment, the model quantization training module includes a pseudo quantization node insertion unit, a quantization parameter calculation unit, a quantization and dequantization unit, an error simulation unit, a dynamic adjustment unit and a training optimization unit: A pseudo-quantization node insertion unit is used to obtain key positions, where the key positions include convolutional layers and fully connected layers. Pseudo-quantization nodes are inserted into the convolutional layers and fully connected layers to simulate the quantization process from floating points to integers. The quantization process includes rounding errors and saturation errors. A quantization parameter calculation unit, used to count the minimum and maximum values ​​of weights and activation values, and calculate the quantization scale and zero point based on the statistical results; The quantization and dequantization unit is used to quantize floating-point data into low-precision data and dequantize low-precision data into floating-point data during forward propagation and pass it to the next layer; The error simulation unit is used to introduce rounding error and saturation error in the pseudo-quantization node, and adjust the model parameters according to the quantization error during the back-propagation process.

[0036] Dynamic adjustment unit, used to recalculate the minimum and maximum values ​​of weights and activation values ​​in each training cycle, and update the quantization parameters according to the new statistical results; The training optimization unit is used to update the model parameters using the gradient descent algorithm and adjust the model weights according to the quantization error.

[0037] The pseudo-quantization node insertion unit mentioned above automatically identifies the key positions in the model (mainly convolutional layers and fully connected layers). These layers have a great influence on the calculation accuracy. Pseudo-quantization nodes are inserted in the above key layers to simulate the conversion process from floating-point numbers to integer representations in the actual quantization process. Rounding errors and saturation errors are introduced in the inserted pseudo-quantization nodes to truly reflect the numerical deviations that may occur in the quantization process. The quantization parameter calculation unit dynamically counts the minimum and maximum values ​​of the weights and activation values ​​of each key layer during training. According to the statistical results, the scale and zero point suitable for quantization are calculated. These parameters are used to map floating-point data to the integer range. The quantization and dequantization unit converts high-precision floating-point data into low-precision integer data according to the preset quantization parameters during the forward propagation of the model, and converts the quantized low-precision data into floating-point data again, and passes it to the next layer to ensure that the numerical calculation process of the entire network can be carried out continuously. The error simulation unit consciously introduces rounding errors and saturation errors in the pseudo-quantization nodes to simulate the possible numerical deviations during real hardware quantization. In the process of back propagation, the gradient is adjusted according to the introduced quantization error to make the model parameters have a certain robustness to the quantization error. The dynamic adjustment unit re-counts the minimum and maximum values ​​of the weights and activation values ​​in each training cycle to reflect the dynamic changes of the model parameters during the training process. According to the latest statistical results, the quantization parameters such as quantization scale and zero point are updated in time to enable the quantization strategy to adapt to the model state. The training optimization unit uses the gradient descent algorithm to update the model parameters during the entire training process. According to the quantization errors generated in forward and backward propagation, the model weights are dynamically adjusted to make it more suitable for low-precision computing environments. The quantization training module uses pseudo-quantization nodes, error simulation and dynamic parameter adjustment to keep the model highly accurate in low-precision environments while reducing computing resources and energy consumption. It simulates and compensates for quantization errors in advance during the training process to lay a solid foundation for the model when it is deployed on actual hardware. The dynamic adjustment and training optimization units ensure that the model can adapt to changes and improve the system's resistance to quantization errors, ultimately achieving stable and efficient reasoning performance.

[0038] When using TVM's quantization tool for quantized training, automatically inserting pseudo-quantization nodes is a key operation. Pseudo-quantization nodes simulate the quantization process from floating point to integer, including the simulation of rounding error and saturation error. For example, for the weight of a convolutional layer, during the forward propagation process, the pseudo-quantization node will map the floating-point weight value to the corresponding integer interval according to the set quantization bit width (such as INT8), and record the error information generated in the process. This allows the model to perceive the impact of quantization during training, thereby prompting the model to adjust its own parameters to adapt to the quantized representation.

[0039] In each forward propagation of the training process, the pseudo-quantization node will quantize and dequantize the data according to the preset quantization parameters (such as quantization scale and zero point). Taking a simple neural network as an example, when the input data is calculated by the first layer of neurons to obtain the activation value, the pseudo-quantization node will intervene. It will first quantize the activation value from high precision (F32) to low precision (INT8), which simulates the data conversion during actual quantization. Then the low-precision data is dequantized back to high precision and passed to the next layer of neurons for subsequent calculations. In this way, during the forward propagation process, each layer of the model can be affected by quantization. In the back propagation stage, due to the existence of the pseudo-quantization node, the gradient calculation will take into account the errors caused by the quantization and dequantization operations. Based on the gradient descent algorithm, the parameter update of the model will be adjusted according to the quantized error. The normal convolution calculation formula is as follows: ; Among them, x out is the output value, x in is the input value, w is the convolution kernel, and b is the bias.

[0040] When inserting a pseudo-quantization node, insert pseudo-quantization at the input of the convolution layer. Suppose the low-precision data after pseudo-quantization is input as (x in ) q , first quantize according to the above quantization formula: ; Where round means rounding operation, s in is the quantitative scale, z in is zero point, x in is the input value.

[0041] Then, after the convolution calculation, pseudo quantization is inserted into the output of the convolution layer. Suppose the low-precision data after pseudo quantization is (x out ) q , and quantize first: ; Where round represents the rounding operation, x out is the output value, zout is zero point, s out For quantitative scale.

[0042] Then, dequantization is performed and the dequantized output data is passed to the next layer. , used to replace the original x in Participate in convolution calculation; for the output dequantized data , passed to the next layer.

[0043] In terms of dynamically adjusting quantization parameters, TVM performs statistical analysis on the weights and activation values ​​in the model in each training batch or a certain training cycle. For the weights, calculate their minimum value (f min ) and maximum value (f max ), which can be achieved by traversing all elements in the weight tensor. For example, in the weight matrix of a fully connected layer, the value of each element is compared in turn to determine the minimum and maximum values ​​in the entire matrix. For the activation value, the minimum and maximum values ​​of its output are also recorded during each forward propagation.

[0044] When the actual quantization is performed based on the parameters obtained from these statistics, a formula is used to convert the floating point value into a quantized integer representation. Taking INT8 quantization as an example, for the weight ω, the quantized integer representation Q ω Calculated by the following formula: ; Where round represents the rounding operation, ω represents the weight, Q ω Expressed as a quantized integer, f min Expressed as the minimum value and f max In the calculation process, it is necessary to ensure that the quantized integers can accurately reflect the relative size relationship of the original floating-point values, and that the model performance will not be seriously degraded due to quantization errors in subsequent calculations.

[0045] In a preferred embodiment, the model import and optimization module includes a model file import unit, an IR conversion unit, a calculation graph optimization unit, an operator fusion unit and a hardware adaptation unit: The model file import unit is used to import the trained model file into the TVM quantization tool and read the structure and parameters of the model. The model file formats include .pth and .pt. IR conversion unit, used to parse the computational graph and parameters of the imported model file and convert the computational graph into the intermediate representation of TVM; The computational graph optimization unit is used to call the optimizer of the TVM quantization tool to optimize the IR; The operator fusion unit is used to identify operators that can be fused in the computation graph and call the optimizer of the TVM quantization tool to fuse the operators that can be fused; The hardware adaptation unit is used to call the TVM quantization tool to generate code for the target hardware and adjust the execution strategy of the computation graph according to the characteristics of the target hardware.

[0046] The above-mentioned model file import unit is responsible for importing the trained model file (the format supports .pth and .pt) into the TVM quantization tool, reading and parsing the model's structure, parameters and weight information. The IR conversion unit parses the calculation graph and parameters in the imported model file and converts them into TVM's intermediate representation (IR). IR is a unified and abstract calculation graph representation method, which is helpful for subsequent cross-platform optimization and code generation. The calculation graph optimization unit uses the built-in optimizer of the TVM quantization tool to optimize the calculation graph in the IR as a whole. The optimization process includes deleting redundant calculations, reordering operations, adjusting data layout, etc., so as to improve computing efficiency. The operator fusion unit identifies continuous operators that can be fused in the calculation graph (such as convolution and activation functions, convolution and BN layers, etc.), calls the TVM optimizer to fuse operators, and combines multiple continuous operators. The hardware adapter unit merges the subsequent operations into a more efficient operation, reducing intermediate data transmission and computing redundancy. The hardware adapter unit calls the TVM quantization tool to generate code for the target hardware. According to the characteristics of the target hardware (such as memory structure, computing unit, parallelism, etc.), the execution strategy of the computational graph is adjusted and optimized. From model file import to IR conversion, to overall optimization of the computational graph and operator fusion, each step improves the running efficiency of the model and reduces resource consumption. The hardware adapter unit generates efficient code for a specific hardware platform, so that the same model can be efficiently deployed on different hardware. The optimized model has lower latency and higher throughput, which is suitable for real-time reasoning scenarios. At the same time, it also reduces energy consumption. The TVM quantization tool is used to achieve full process automation and standardization from model import to hardware code generation, providing reliable technical support for large-scale deployment.

[0047] TVM's optimizer performs a series of optimization operations on the IR. These operations are performed according to pre-defined optimization strategies and the characteristics of the target hardware. The YOLOv5 model is optimized through operator fusion, whose main purpose is to merge multiple consecutive operations into a more efficient operation. For a typical convolutional neural network such as YOLOv5, each convolution operation is followed by an activation function (such as ReLU). In the unoptimized case, data needs to be transferred between the convolution layer and the activation function layer, which will incur a certain amount of time and resource overhead. Through operator fusion, these two operations are merged into a new operation, which can reduce the number of data transfers between the memory and the computing unit, because the fused operation can complete the data processing that originally required two operations in one calculation process, thereby significantly improving performance.

[0048] In a preferred embodiment, the model evaluation module includes a data loading and preprocessing unit, a model reasoning unit, an accuracy evaluation unit, a comparative analysis unit, and a visualization and report generation unit: The data loading and preprocessing unit is used to load the test data set and preprocess the test data set to obtain a normalized test data set that matches the model input format; The model inference unit is used to call the quantized model, input the normalized test data set for inference, and generate prediction results; An accuracy evaluation unit calculates evaluation data of the quantitative model according to the prediction results, wherein the evaluation data includes accuracy, recall, F1 value and average precision mean; A comparative analysis unit is used to compare the model performance before and after quantization based on the evaluation data. The model performance includes inference speed, computing resource consumption, and accuracy change. The visualization and report generation unit is used to generate evaluation reports based on model performance. The evaluation reports include curve graphs, histograms, and confusion matrices.

[0049] It is worth mentioning that the evaluation data includes accuracy, recall, F1 value and average precision.

[0050] As mentioned above, the data loading and preprocessing unit loads the original data from the test data set and preprocesses the data. The preprocessing steps include data format conversion, normalization, and necessary image enhancement to make the data consistent with the model input requirements. The model inference unit calls the quantized and optimized model to perform inference calculations on the preprocessed test data, generate prediction results, process the test data in real time or in batches, and record the model output. The accuracy evaluation unit calculates evaluation indicators such as accuracy, recall, F1 value, and mean average precision (mAP) based on the model's prediction results and true labels. These indicators comprehensively reflect the performance of the model in classification or detection tasks. The comparative analysis unit compares the model evaluation data before and after quantization, covering inference speed, computing resource consumption, and accuracy. The visualization and report generation unit generates graphical reports including curve graphs, histograms, confusion matrices, etc. according to the evaluation results, organizes and displays various indicators and comparative analysis results to form a complete evaluation report. The data preprocessing module ensures that the input data matches the model requirements to avoid data bias affecting the evaluation results. The multi-dimensional evaluation model is evaluated from inference speed, resource consumption to accuracy, recall rate, F1 value, mAP, etc. to ensure that the quantized model is both efficient and accurate. The comparative analysis unit can provide timely feedback on the performance changes of the model before and after quantification, support further adjustment and optimization of the quantization strategy, and the generated evaluation report intuitively displays key indicators and comparison results, helping developers and managers to quickly make technical decisions and deployment plans.

[0051] In the image classification scenario, the test data set is used to predict the model before and after quantization, and the matching degree of the prediction results with the real labels is compared. For example, in the smart power plant inspection, the classification algorithm for indicator lights is deployed and trained. The accuracy on the test set after quantization should remain at a similar level as before quantization. If the accuracy drops significantly, further analysis is needed, which may be due to the loss of some key information during the quantization process or improper adjustment of the quantization parameters.

[0052] Focusing on the intelligent inspection algorithm reasoning system serving wind power, photovoltaic new energy, hydropower and traditional thermal power digital power plants, the system uses visible light measurement technology to identify the position of substation switches, switch status and other states, as well as the status and readings of transformer switches, oil levels, etc., and remotely detects the status of various equipment in the substation through cameras.

[0053] In order to solve the problems encountered in power generation scenarios, such as camera offset and difficulty in locating equipment due to a wide variety of equipment, the optimized target detection model YOLOv5 after the above operations is used as the initial processing module to automatically detect the locations of the instruments and equipment to be identified.

[0054] Secondly, the framework will process the images of the detected devices. Specifically, it will select from a variety of processing methods such as cropping, denoising, and grayscale according to the specific needs of different devices. Through such processing, the image can more clearly and accurately reflect the characteristics of the device, providing better input judgment for subsequent status judgment.

[0055] In the state judgment stage, the optimized classification network or detection network is used to judge the state of the image obtained by the detection model. After analysis and processing, the state result of the equipment is finally obtained and output, providing accurate and timely equipment state information to relevant personnel so that they can take corresponding measures for maintenance and management. And, a full-stack autonomous and controllable artificial intelligence reasoning terminal based on Denglin GPU, including: one or more processors; a storage device having one or more programs stored thereon; When one or more programs are executed by one or more processors, one or more processors can realize a full-stack autonomous and controllable artificial intelligence reasoning system based on the Denglin GPU.

[0056] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. A full-stack autonomous and controllable artificial intelligence reasoning system based on Denglin GPU, characterized in that: include: GPU initialization module, used to load SDK and graphics card driver, call cudaSetDevice to select at least one available GPU, and call dlvidGetDevice to obtain the VPU device connected to the GPU; The video decoding module is used to call dlvidCreateSession to create a decoding session according to the format of the video stream to be decoded, and to establish the video input and output buffer queues by calling dlvidCreateBuffer, and then use dlvidRegisterBuffer to register the buffer queues as the input and output of the decoding session; The multi-channel video stream processing module uses the RTSP protocol to obtain the video stream data to be decoded from the database, and adopts a multi-threaded processing mechanism and dlvidProcessSessionEvent to obtain the callback function, and obtains the decoded YUV frame data in real time when the output buffer queue is full; The model quantization training module is used to automatically insert pseudo-quantization nodes during the model training process using the TVM quantization tool, quantize and dequantize the input and output of the convolutional layer and the fully connected layer, and adjust the quantization parameters based on the dynamic statistical weights and the minimum and maximum values ​​of the activations of the training samples to simulate the rounding and saturation errors generated during the quantization process; The model import and optimization module is used to import the trained model file into TVM, convert it into TVM's intermediate representation, and reorganize and optimize the computational graph using operator fusion optimization technology; The model evaluation module is used to evaluate the accuracy of the quantized model.

2. The full-stack autonomous and controllable artificial intelligence reasoning system based on Denglin GPU according to claim 1 is characterized in that: The GPU initialization module includes an initialization unit, a driver loading unit, a device selection unit, and a VPU device acquisition unit: Initialization unit, used to initialize the GPU device; Driver loading unit, used to load the SDK and graphics card driver required by the GPU GPU device selection unit, used to call cudaSetDevice to specify the GPU device used by the current task; The VPU device acquisition unit is used to obtain the VPU device associated with the GPU by using dlvidGetDevice.

3. The full-stack autonomous and controllable artificial intelligence reasoning system based on Denglin GPU according to claim 1 is characterized in that: The video decoding module includes a decoding session creation unit, a cache queue management unit, and a decoding registration unit: The decoding session creation unit obtains the format of the video stream to be decoded, wherein the formats include H.254 and H.265, calls the dlvidCreateSession function and creates a decoding session according to the format of the video stream to be decoded, so as to initialize the decoder and configure decoding parameters; The buffer queue management unit creates a video input buffer queue and an output buffer queue by calling dlvidCreateBuffer, which are used to store compressed video stream data and store YUV frames generated after decoding respectively; The decoding registration unit is used to call dlvidRegisterBuffer to register the input buffer queue and the output buffer queue to the decoding session.

4. The full-stack autonomous and controllable artificial intelligence reasoning system based on Denglin GPU according to claim 1 is characterized in that: The multi-channel video stream processing module includes a video stream acquisition unit, a video decoding management unit, a multi-thread scheduling unit and an event callback processing unit: The video stream acquisition unit obtains the video stream data to be decoded from the database based on the RTSP protocol; A video decoding management unit, configuring a decoder according to the format of a decoded video stream; A multi-thread scheduling unit, used to create multiple threads, each of which processes a video stream to be decoded; The event callback processing unit is used to call the dlvidProcessSessionEvent registered callback function. When the output buffer queue is full, the callback function is triggered to obtain and process the decoded YUV frame data in real time.

5. The full-stack autonomous and controllable artificial intelligence reasoning system based on Denglin GPU according to claim 4 is characterized in that: The multi-channel video stream processing module also includes a decoded image preprocessing unit: The decoded image preprocessing unit is used to preprocess the YUY frame data into an input format that conforms to the model. The preprocessing includes converting the YUV format into the RGB format, image cropping, image scaling, and normalization.

6. The full-stack autonomous and controllable artificial intelligence reasoning system based on Denglin GPU according to claim 1 is characterized in that: The model quantization training module includes a pseudo-quantization node insertion unit, a quantization parameter calculation unit, a quantization and dequantization unit, an error simulation unit, a dynamic adjustment unit, and a training optimization unit: A pseudo-quantization node insertion unit is used to obtain key positions, where the key positions include convolutional layers and fully connected layers. Pseudo-quantization nodes are inserted into the convolutional layers and fully connected layers to simulate the quantization process from floating points to integers. The quantization process includes rounding errors and saturation errors. A quantization parameter calculation unit, used to count the minimum and maximum values ​​of weights and activation values, and calculate the quantization scale and zero point based on the statistical results; The quantization and dequantization unit is used to quantize floating-point data into low-precision data and dequantize low-precision data into floating-point data during forward propagation and pass it to the next layer; The error simulation unit is used to introduce rounding error and saturation error in the pseudo-quantization node, and adjust the model parameters according to the quantization error during the back-propagation process; Dynamic adjustment unit, used to recalculate the minimum and maximum values ​​of weights and activation values ​​in each training cycle, and update the quantization parameters according to the new statistical results; The training optimization unit is used to update the model parameters using the gradient descent algorithm and adjust the model weights according to the quantization error.

7. The full-stack autonomous and controllable artificial intelligence reasoning system based on Denglin GPU according to claim 1 is characterized in that: The model import and optimization module includes a model file import unit, an IR conversion unit, a computational graph optimization unit, an operator fusion unit, and a hardware adaptation unit: The model file import unit is used to import the trained model file into the TVM quantization tool and read the structure and parameters of the model. The model file formats include .pth and .pt. IR conversion unit, used to parse the computational graph and parameters of the imported model file and convert the computational graph into the intermediate representation of TVM; The computational graph optimization unit is used to call the optimizer of the TVM quantization tool to optimize the IR; The operator fusion unit is used to identify operators that can be fused in the computation graph and call the optimizer of the TVM quantization tool to fuse the operators that can be fused; The hardware adaptation unit is used to call the TVM quantization tool to generate code for the target hardware and adjust the execution strategy of the computation graph according to the characteristics of the target hardware.

8. The full-stack autonomous and controllable artificial intelligence reasoning system based on Denglin GPU according to claim 1 is characterized in that: The model evaluation module includes data loading and preprocessing unit, model reasoning unit, accuracy evaluation unit, comparative analysis unit and visualization and report generation unit: The data loading and preprocessing unit is used to load the test data set and preprocess the test data set to obtain a normalized test data set that matches the model input format; The model inference unit is used to call the quantized model, input the normalized test data set for inference, and generate prediction results; An accuracy evaluation unit calculates evaluation data of the quantitative model according to the prediction results, wherein the evaluation data includes accuracy, recall, F1 value and average precision mean; A comparative analysis unit is used to compare the model performance before and after quantization based on the evaluation data. The model performance includes inference speed, computing resource consumption, and accuracy change. The visualization and report generation unit is used to generate evaluation reports based on model performance. The evaluation reports include curve graphs, histograms, and confusion matrices.

9. The full-stack autonomous and controllable artificial intelligence reasoning system based on Denglin GPU according to claim 8 is characterized in that: The evaluation data include accuracy, recall, F1 value and average precision.

10. A full-stack autonomous and controllable artificial intelligence reasoning terminal based on Denglin GPU, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When one or more programs are executed by one or more processors, one or more processors implement the full-stack autonomous and controllable artificial intelligence reasoning system based on the GPU as described in any one of claims 1 to 9.

Images