AI vision-based video vlm model inference acceleration method

Abstract

The application discloses an AI vision-based video VLM model inference acceleration method, comprising the following steps: S1, synchronously collecting a video stream and hardware state parameters; S2, dynamically sparsifying sampling to generate a sparse frame sequence; S3, allocating a continuous video memory physical address pool; S4, using an improved VideoMAE model and constructing a video memory direct-reading forward propagation layer to obtain a base address pointer to skip a standard interface, performing streaming calculation to reconstruct features in an SM, and outputting a compressed Token sequence; S5, using remaining space to store key-value caches to perform large model calculation to output semantic features; S6, using an Orca algorithm to insert micro-batch calculation during SM idle periods to cover up delayed output inference texts; and S7, releasing the caches and dynamically adjusting a frame rate and a compression ratio to perform a cycle. The application eliminates redundant copying and significantly improves inference throughput.

Description

Technical Field

[0001] This invention relates to the intersection of artificial intelligence and computer architecture, and in particular to a method for accelerating inference of video VLM models based on AI vision. Background Technology

[0002] Video visual large language models, due to their cross-modal understanding capabilities that deeply integrate complex visual content with natural language, have been widely applied in recent years in fields such as intelligent security, autonomous driving, and embodied intelligence, becoming an important development direction for realizing refined machine cognition and interaction. However, in practical applications, real-time inference scenarios involving long video streams face numerous challenges, including limited GPU memory capacity, memory access bandwidth bottlenecks, and idle computing units, and the deployment effectiveness of video visual large language models remains constrained by various factors.

[0003] Currently, most video inference methods rely on the default memory management mechanism of deep learning frameworks for tensor scheduling, which makes it difficult to fully utilize the continuity of the underlying physical memory of the GPU. Frequent memory allocation and release lead to severe memory fragmentation, which exposes the massive key-value cache generated during the decoding stage of large language models to overflow risks, severely restricting the stability of long video processing. Some systems only use a fixed frame rate and full resolution strategy to input video data, ignoring the dynamic fluctuations of the underlying bus bandwidth utilization. They fail to combine the background redundancy in the temporal dimension of the video with the invalid perceptual regions in the spatial dimension for adaptive downsampling, resulting in a large amount of computing power and bandwidth being consumed in the forward computation of low-value pixels, limiting the system's throughput capacity to handle high-concurrency video streams.

[0004] Furthermore, existing video inference architectures lack fine-grained pipeline scheduling mechanisms when performing sequential tasks of visual encoding and large language model decoding. During the decoding stage, when the computing unit (SM) idles for a long time while waiting for memory data to load, it cannot proactively switch to the pre-computation task of the next video segment. This causes the masking logic to fail to accurately align the computation time with the memory access wait time, resulting in a large amount of idle and wasted hardware computing resources. This makes it difficult to adapt to the ever-changing and evolving scenarios of high real-time and low-latency video stream analysis, seriously affecting the practical value and inference efficiency of large models in real-world edge or cloud scenarios.

[0005] Therefore, how to provide a method for accelerating video VLM model inference based on AI vision is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose an AI vision-based method for accelerating video VLM model inference. This invention fully integrates key steps such as underlying resource state awareness, dynamic spatiotemporal sparsity sampling, continuous memory physical address pool allocation, an improved VideoMAE model and memory direct-read forward propagation layer construction, and Orca algorithm dual-stream scheduling. It constructs a video inference acceleration process with real-time hardware state quantization, memory physical address locking, adaptive invalid data removal, and overlapping computation and memory access execution, achieving efficient processing of video stream inference in resource-constrained environments. This invention possesses advantages such as zero-copy memory direct reading, high-ratio compression of spatiotemporal features, accurate masking during idle periods, and dynamic closed-loop parameter configuration. It can significantly improve memory utilization, computational unit parallelism, and end-to-end throughput in complex video scenarios, thereby effectively solving problems such as fragmented memory allocation, redundant invalid computation, and high memory access latency in existing methods.

[0007] The video VLM model inference acceleration method based on AI vision according to embodiments of the present invention includes the following steps: S1. Synchronously collect the video stream data to be inferred and the underlying resource status parameters of the current inference hardware; S2. Based on the underlying resource status parameters, the video stream data is dynamically spatiotemporally sparsified to remove redundant background frames in the time dimension and high-frequency invalid perception areas in the spatial dimension, generating a sparse video frame sequence. S3. Based on the current GPU memory free threshold, pre-define and allocate a contiguous physical address pool in the memory. S4. Using the improved VideoMAE model, random masking operation is performed on non-critical spatiotemporal pixel blocks in sparse video frame sequences. A direct read forward propagation layer is constructed to obtain the base address pointer. Streaming matrix multiplication is performed on the visible unmasked pixel blocks in the computing unit SM to perform feature aggregation and reconstruction in the temporal and spatial dimensions, and output a compressed visual token sequence. S5. In the contiguous video memory physical address pool, the compressed visual token sequence is input into the large language model to perform forward computation, and the remaining physical address space is used to store the key-value cache generated during the decoding process, and the visual semantic features are output. S6. The Orca algorithm is used to process visual semantic features. In the forward calculation of the token in the current video segment decoding, if the computing unit SM is detected to be in the idle period of waiting for video memory, the current decoding state is paused and saved. The sparse video frame sequence of the next video segment is inserted into the idle SM as a micro-batch for execution. After completion, the decoding state is restored, and the video inference text result is output by masking the waiting delay. S7. After the current video segment inference is completed, release the key-value cache in the contiguous video memory physical address pool, and dynamically adjust the sampling frame rate and token compression ratio of the next video segment according to the bus bandwidth utilization. Repeat S1 to S6 until the video stream ends.

[0008] Optionally, S1 specifically includes: S11. Synchronously trigger video stream reading operation through system-level hardware polling interface to obtain video stream data to be inferred; S12. Call the GPU underlying monitoring driver to read the current available video memory capacity of the current inference hardware's video memory management unit. Read the operating system underlying video memory usage value, the deep learning framework static basic overhead value, and the minimum key-value cache volume value generated by a single decoding of a large language model from the pre-written configuration file. Add the above three read values ​​together to obtain the preset minimum safe video memory capacity. Calculate the difference between the current available video memory capacity and the preset minimum safe video memory capacity, and set it as the current GPU video memory idle threshold. S13. Call the GPU underlying monitoring driver to read the instruction issue count and completion count of all computing units (SMs) in the current clock cycle, calculate the ratio of instruction issue count to completion count, and record the ratio as the computing unit (SM) utilization rate. S14. Call the GPU underlying monitoring driver to read the number of bytes of data transfer between the memory controller and the computing unit SM within the current time window, read the peak physical bandwidth of the bus as specified by the GPU hardware factory, calculate the percentage of data transfer bytes to the peak physical bandwidth of the bus, and mark the percentage as the bus bandwidth utilization rate. S15. Pack and align the current GPU memory free threshold, computing unit SM occupancy rate and bus bandwidth utilization rate according to the sampling timestamp, and output the underlying resource status parameters.

[0009] Optionally, S2 specifically includes: S21. Read the SM occupancy rate of the underlying resource status parameters, divide the video stream data into video segments according to the time window, calculate the sum of the absolute values ​​of the differences between corresponding pixels between two adjacent video images, and when the sum of the absolute values ​​of the differences is lower than the preset static threshold, mark the next video image as a redundant background frame in the time dimension and remove it. S22. Calculate the average frame rate of the remaining video frames in the current video segment, read the bus bandwidth utilization rate in the underlying resource status parameters, and if the bus bandwidth utilization rate exceeds the preset bandwidth threshold, calculate the ratio of the average frame rate to the preset bandwidth threshold and round the ratio down as the spatial sampling step size. S23. If the bus bandwidth utilization does not exceed the preset bandwidth threshold, the space sampling step size is set to 1. S24. Divide the single-frame video image into grids according to the spatial sampling step size, calculate the variance of all pixel gray values ​​in each grid, and mark grids with variances lower than the preset texture threshold as high-frequency invalid perception regions in the spatial dimension and remove them. S25. After removing redundant background frames and high-frequency invalid sensing areas, the remaining video image frames are reassembled in chronological order to output a sparse video frame sequence.

[0010] Optionally, S3 specifically includes: S31. Read the current GPU memory free threshold from the underlying resource status parameters and obtain the maximum contiguous physical block size supported by the current inference hardware in a single memory allocation. S32. Compare the current GPU memory free threshold with the maximum contiguous physical block size for a single memory allocation. If the current GPU memory free threshold is greater than the maximum contiguous physical block size for a single memory allocation, then set the maximum contiguous physical block size for a single memory allocation as the size of the allocation pool. S33. If the current GPU memory free threshold is less than or equal to the maximum contiguous physical block size in a single memory allocation, then set the current GPU memory free threshold to the size of the pool to be allocated. S34. Call the GPU low-level driver interface, lock the physical address mapping table of video memory, find a free area in video memory with a volume greater than or equal to the volume of the pool to be allocated, and prevent the operating system from performing page swapping operations on the free area. S35. Obtain the physical starting address of the free area as the base address pointer, bind the base address pointer to the volume of the pool to be allocated, mark it as a continuous video memory physical address pool and output it.

[0011] Optionally, S4 specifically includes: S41. Divide each video frame image in the sparse video frame sequence into non-overlapping pixel blocks. According to the encoder input size requirements of the improved VideoMAE model, calculate the RGB three-channel mean of all pixels in the pixel block and concatenate them into an initial feature vector. S42. Construct a random mask generator to generate a random number matrix with values ​​ranging from 0 to 1. Set the positions in the random number matrix with values ​​greater than the preset mask ratio to 1, and set the positions with values ​​less than or equal to the preset mask ratio to 0 to form a binary mask image. Mark the pixel blocks corresponding to the positions set to 0 in the binary mask image as non-critical spatiotemporal pixel blocks, and mark the pixel blocks corresponding to the positions set to 1 as visible unmasked pixel blocks. S43. Construct a forward propagation layer for direct reading of video memory, read the base address pointer of the continuous video memory physical address pool, disable the memory allocation and data copy functions in the standard tensor loading interface, and directly write the initial feature vectors corresponding to the visible unmasked pixel blocks in a continuous arrangement of video memory physical addresses. S44. Within the computing unit SM, allocate a shared memory region, load the initial feature vectors of visible unmasked pixel blocks stored in the continuous video memory physical address pool into the shared memory region, read the weight parameters of the improved VideoMAE model, perform streaming matrix multiplication calculation of the initial feature vectors and weight parameters within the shared memory region, and output the visible feature vectors. S45. Based on the image resolution and pixel block size before the sparse video frame sequence is divided, calculate the number of pixel blocks in the width and height directions of the image, multiply them to obtain the total number of pixel blocks, and construct a zero matrix with the dimension of the total number of pixel blocks multiplied by the number of feature channels as the full-size feature matrix. S46. Iterate through each position in the binary mask image one by one. When the value of the current position is 1, read the corresponding visible feature vector and copy it to the data row at the same index position in the full-size feature matrix. Otherwise, read the preset learnable mask token vector, copy it to the data row at the same index position in the full-size feature matrix, and generate the filled feature matrix. S47. Add the filled feature matrix to the preset position encoding vector at the corresponding positions to obtain the feature matrix with embedded position information. Calculate the dot product of each mask token vector with all other vectors in the entire image. Divide the dot product result by the preset scaling factor and input it into the Softmax function to calculate the normalized attention weight. Then, perform a weighted sum with the values ​​of all other vectors in the entire image to obtain the updated mask feature vector. Add the updated mask token vector to complete the residual connection and feature reconstruction. S48. Along the time axis, concatenate the reconstructed feature vector of the current frame with the feature vectors of the same position in the preceding and following frames, perform a one-dimensional convolution operation, calculate the weighted sum of features in the time dimension, perform layer normalization on the weighted sum result, and output a compressed visual token sequence.

[0012] Optionally, S5 specifically includes: S51. Read the compressed visual token sequence and use it as an input tensor. Map it to the starting storage area of ​​the continuous video memory physical address pool. Call the embedding layer weights of the large language model. Perform matrix multiplication on the compressed visual token sequence in the computing unit SM. Map the visual features to the word vector space of the large language model to generate a visual embedding vector sequence. S52. Read the remaining physical address space in the contiguous video memory physical address pool after removing the space occupied by the visual embedding vector sequence, and mark it as a key-value cache dedicated storage area. During the decoding forward calculation of each layer of the large language model, calculate the query vector, key vector and value vector of the current token, and append the calculated key vector and value vector to the key-value cache dedicated storage area in the order of calculation. S53. Read the stored historical key vector and value vector from the key-value cache dedicated storage area, perform a dot product operation between the historical key vector and the query vector of the current token, calculate the attention score, and perform a weighted summation of the attention score and the value vector to obtain the output feature vector of the current layer. S54. Repeat the forward computation and key-value cache read / write operations layer by layer until the last layer of the large language model outputs. Normalize the output feature vector to generate visual semantic features and output them.

[0013] Optionally, S6 specifically includes: S61. Construct a dual-stream scheduling state machine for the Orca algorithm, read the instruction issue queue register inside the computing unit SM in real time, calculate the number of instructions to be executed in the current clock cycle, divide the number of instructions to be executed by the maximum instruction issue width of the computing unit SM to obtain the instruction queue fill rate, and when the instruction queue fill rate is lower than the preset fill rate threshold, determine that the computing unit SM is in the idle period of waiting for video memory and generate a scheduling switching signal. S62. Respond to the scheduling switching signal, freeze the decoding process of the current video segment, read the hidden layer state matrix of each layer in the Transformer architecture of the large language model layer by layer, calculate the data volume of the hidden layer state matrix of each layer, apply for a snapshot storage area of ​​the same volume in the continuous video memory physical address pool, copy the hidden layer state matrix byte by byte to the snapshot storage area, read the token index position generated by the current decoding, and combine the token index position with the first address pointer of the snapshot storage area to form the decoding scene snapshot data packet. S63. Read the sparse video frame sequence of the next video segment as a micro-batch, align the data start address of the micro-batch to the base address of the continuous video memory physical address pool, call the computing unit SM in the idle period to perform the forward calculation of the improved VideoMAE model, extract the visual semantic features of the micro-batch, write the extracted visual semantic features into the free area in the continuous video memory physical address pool, and generate the feature pointer to be inferred. S64. After the visual semantic features of the micro-batch are written, an encoding completion signal is generated. Based on the first address pointer in the decoded snapshot data packet, the hidden layer state matrix in the snapshot storage area is copied back to the on-chip memory of the computing unit SM. Based on the Token index position, the context environment for generating the decoded text is restored, and the Token prediction calculation for the current video segment is continued. S65. Calculate the masking duration by the time difference between the start and end times of the micro-batch forward calculation in step S63. Read the busy / idle status parameters of the video memory controller to obtain the video memory loading wait time. Calculate the difference between the masking duration and the video memory loading wait time. When the difference is greater than zero, the masking is determined to be successful. Overlap and align the execution time period of the micro-batch calculation process with the time period of video memory data loading on the time axis, and output the video inference text result of the current video segment.

[0014] Optionally, S7 specifically includes: S71. Detect the signal that the video inference text result of the current video segment has been output, call the video memory release instruction, clear the data of the key-value cache special storage area stored in the continuous video memory physical address pool, and release the storage area from the lock. S72. Read the bus bandwidth utilization rate from the underlying resource status parameters, calculate the absolute value of the difference between the bus bandwidth utilization rate and the preset bandwidth threshold, and divide the absolute value of the difference by the bus physical peak bandwidth to obtain the bandwidth load fluctuation rate. S73. Multiply the preset baseline frame rate by 1 and the difference between the bandwidth load fluctuation rate to calculate the sampling frame rate of the next video segment. Multiply the preset baseline compression ratio by 1 and the sum of the bandwidth load fluctuation rate to calculate the token compression ratio of the next video segment. S74. Update the parameter configuration in S2 and S4 based on the calculated sampling frame rate and token compression ratio, jump to execute S1, read the video stream data and underlying resource status parameters of the next moment, and execute S1 to S6 in a loop until the video stream data reading is finished.

[0015] The beneficial effects of this invention are: This invention addresses the issues of fragmented memory allocation, high data transfer overhead, and high memory access latency in video VLM model inference by constructing a low-level resource state awareness mechanism and deploying a contiguous physical memory address pool. It employs a hardware polling interface to read GPU state parameters, combined with dynamic spatiotemporal sparsity sampling to remove redundant background frames and invalid perception regions, generating a sparse video frame sequence. By locking the physical memory address mapping table, a contiguous physical memory address pool is pre-defined. The sparse video frame sequence is then input into an improved VideoMAE model, and through an innovative construction of a direct-read forward propagation layer for memory, the system directly obtains the desired data. By fetching the base address pointer and disabling memory allocation and data copy functions in the standard tensor loading interface, visible pixel blocks are directly written to the shared memory of the computation unit (SM) in a contiguous manner according to their physical memory addresses to perform streaming matrix multiplication, completely eliminating data transfer overhead at the frame layer and outputting a compressed visual token sequence. During the decoding stage, an Orca algorithm is used to construct a dual-stream scheduling state machine, accurately identifying the idle period of the computation unit and inserting micro-batch calculations of the next video segment. Micro-batch calculations are used to mask duration overlap and align memory loading wait times. Furthermore, the sampling frame rate and compression ratio are dynamically adjusted based on bandwidth fluctuation. Ultimately, a closed-loop acceleration is achieved through hardware and software collaboration, effectively improving memory bandwidth utilization, underlying computational resource parallelism, and model inference throughput. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 The flowchart shows the video VLM model inference acceleration method based on AI vision proposed in this invention. Figure 2 This is a flowchart of the feature extraction and compressed visual token sequence generation process of the improved VideoMAE model based on the direct-read forward propagation layer of video memory proposed in this invention. Figure 3 This is a flowchart illustrating the Orca algorithm based on a dual-stream scheduling state machine proposed in this invention for calculating memory access masking and outputting text results for video inference. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0018] refer to Figures 1-3 An AI vision-based method for accelerating video VLM model inference includes the following steps: S1. Synchronously collect the video stream data to be inferred and the underlying resource status parameters of the current inference hardware. The underlying resource status parameters include the current GPU memory idle threshold, the computing unit SM occupancy rate, and the bus bandwidth utilization rate. S2. Based on the underlying resource status parameters, the video stream data is dynamically spatiotemporally sparsified to remove redundant background frames in the time dimension and high-frequency invalid perception areas in the spatial dimension, generating a sparse video frame sequence. S3. Based on the current GPU memory free threshold, pre-define and allocate a contiguous physical address pool in the memory. S4. Using the improved VideoMAE model, random masking operation is performed on non-critical spatiotemporal pixel blocks in sparse video frame sequences. A direct-read forward propagation layer for video memory is constructed to obtain the base address pointer of the continuous video memory physical address pool. The standard tensor loading interface is skipped, and streaming matrix multiplication is performed on the visible unmasked pixel blocks in the computing unit SM. The extracted visible features are mapped back to the masked positions and feature aggregation and reconstruction are performed in the temporal and spatial dimensions. The compressed visual token sequence is output. S5. In the contiguous video memory physical address pool, the compressed visual token sequence is input into the large language model to perform forward computation, and the remaining physical address space is used to store the key-value cache generated during the decoding process, and the visual semantic features are output. S6. The Orca algorithm is used to process visual semantic features. In the forward calculation of the token in the current video segment decoding, if the computing unit SM is detected to be in the idle period of waiting for video memory, the current decoding state is paused and saved. The sparse video frame sequence of the next video segment is inserted into the idle SM as a micro-batch for execution. After completion, the decoding state is restored, and the video inference text result is output by masking the waiting delay. S7. After the current video segment inference is completed, release the key-value cache in the contiguous video memory physical address pool, and dynamically adjust the sampling frame rate and token compression ratio of the next video segment according to the bus bandwidth utilization. Repeat S1 to S6 until the video stream ends.

[0019] This invention significantly improves the inference throughput and real-time response capability of large video models in complex scenarios. By synchronously collecting underlying resource state parameters, dynamic spatiotemporal sparse sampling of the video stream is achieved, eliminating redundant data at the source and greatly reducing ineffective computational overhead. In the feature extraction stage, a direct-read forward propagation layer for video memory is constructed, skipping the standard tensor loading interface and directly obtaining the base address pointer for streaming matrix calculation. This completely eliminates the frequent data copying and memory allocation bottlenecks in traditional frameworks, pushing the utilization of video memory physical bandwidth to the extreme. In the large model decoding stage, an innovative Orca algorithm dual-stream scheduling mechanism is introduced to accurately capture the tiny idle periods of computation units. Micro-batch insertion is used to mask video memory waiting delays, achieving perfect overlap and alignment of computation and memory access on the microscopic time axis. Combined with dynamic partitioning of the video memory pool and efficient reuse of key-value caching, a precipitous reduction in end-to-end inference latency is achieved while ensuring extremely high semantic restoration accuracy. By dynamically adjusting the sampling frame rate and compression ratio through closed-loop feedback, the system demonstrates excellent adaptive scheduling robustness when dealing with massive high-concurrency video streams, fundamentally breaking through the hardware computing power barrier and providing revolutionary technical support for the low-cost and high-energy-efficiency deployment of ultra-large-scale visual models.

[0020] In this embodiment, S1 specifically includes: S11. Synchronously trigger video stream reading operation through system-level hardware polling interface to obtain video stream data to be inferred; S12. Call the GPU underlying monitoring driver to read the current available video memory capacity of the current inference hardware's video memory management unit. Read the operating system underlying video memory usage value, the deep learning framework static basic overhead value, and the minimum key-value cache volume value generated by a single decoding of a large language model from the pre-written configuration file. Add the above three read values ​​together to obtain the preset minimum safe video memory capacity. Calculate the difference between the current available video memory capacity and the preset minimum safe video memory capacity, and set it as the current GPU video memory idle threshold. S13. Call the GPU underlying monitoring driver to read the instruction issue count and completion count of all computing units (SMs) in the current clock cycle, calculate the ratio of instruction issue count to completion count, and record the ratio as the computing unit (SM) utilization rate. S14. Call the GPU underlying monitoring driver to read the number of bytes of data transfer between the memory controller and the computing unit SM within the current time window, read the peak physical bandwidth of the bus as specified by the GPU hardware factory, calculate the percentage of data transfer bytes to the peak physical bandwidth of the bus, and mark the percentage as the bus bandwidth utilization rate. S15. Pack and align the current GPU memory idle threshold, SM utilization rate and bus bandwidth utilization rate according to the sampling timestamp, and output the underlying resource status parameters with timestamp tags.

[0021] In this embodiment, S2 specifically includes: S21. Read the SM occupancy rate of the underlying resource status parameters, divide the video stream data into video segments according to the time window, calculate the sum of the absolute values ​​of the differences between corresponding pixels in two adjacent video images, and when the sum of the absolute values ​​of the differences is lower than the preset static threshold, mark the next video image as a redundant background frame in the time dimension and remove it. The preset static threshold is the product of 5% of the total number of pixels in a single frame video image and the maximum pixel value in a single frame. S22. Calculate the average frame rate of the remaining video frames in the current video segment, read the bus bandwidth utilization rate in the underlying resource status parameters, and if the bus bandwidth utilization rate exceeds the preset bandwidth threshold, calculate the ratio of the average frame rate to the preset bandwidth threshold, and round the ratio down as the spatial sampling step size. The preset bandwidth threshold is 80% of the bus physical peak bandwidth. S23. If the bus bandwidth utilization does not exceed the preset bandwidth threshold, the space sampling step size is set to 1. S24. Divide the single-frame video image into grids according to the spatial sampling step size, calculate the variance of all pixel grayscale values ​​in each grid, and mark grids with variances lower than a preset texture threshold as high-frequency invalid perception regions in the spatial dimension and remove them. The preset texture threshold is 10% of the maximum range of grayscale value variation. S25. After removing redundant background frames and high-frequency invalid sensing areas, the remaining video image frames are reassembled in chronological order to output a sparse video frame sequence.

[0022] In this embodiment, S3 specifically includes: S31. Read the current GPU memory free threshold from the underlying resource status parameters and obtain the maximum contiguous physical block size supported by the current inference hardware in a single memory allocation. S32. Compare the current GPU memory free threshold with the maximum contiguous physical block size for a single memory allocation. If the current GPU memory free threshold is greater than the maximum contiguous physical block size for a single memory allocation, then set the maximum contiguous physical block size for a single memory allocation as the size of the allocation pool. S33. If the current GPU memory free threshold is less than or equal to the maximum contiguous physical block size in a single memory allocation, then set the current GPU memory free threshold to the size of the pool to be allocated. S34. Call the GPU low-level driver interface, lock the physical address mapping table of video memory, find a free area in video memory with a volume greater than or equal to the volume of the pool to be allocated, and prevent the operating system from performing page swapping operations on the free area. S35. Obtain the physical starting address of the free area as the base address pointer, bind the base address pointer to the volume of the pool to be allocated, mark it as a continuous video memory physical address pool and output it.

[0023] In this embodiment, S4 specifically includes: S41. Divide each video frame image in the sparse video frame sequence into non-overlapping 16×16 pixel blocks. According to the encoder input size requirements of the improved VideoMAE model, calculate the RGB three-channel mean of all pixels in the pixel block and concatenate them into an initial feature vector. S42. Construct a random mask generator to generate a random number matrix with values ​​ranging from 0 to 1. Set the positions in the random number matrix with values ​​greater than a preset mask ratio to 1, and set the positions with values ​​less than or equal to the preset mask ratio to 0, forming a binary mask image. Mark the pixel blocks corresponding to the positions set to 0 in the binary mask image as non-critical spatiotemporal pixel blocks, and mark the pixel blocks corresponding to the positions set to 1 as visible unmasked pixel blocks. The preset mask ratio is set to 0.75. S43. Construct a forward propagation layer for direct reading of video memory, read the base address pointer of the continuous video memory physical address pool, disable the memory allocation and data copy functions in the standard tensor loading interface, and directly write the initial feature vectors corresponding to the visible unmasked pixel blocks in a continuous arrangement of video memory physical addresses. S44. Within the computing unit SM, allocate a shared memory region, load the initial feature vectors of visible unmasked pixel blocks stored in the continuous video memory physical address pool into the shared memory region, read the weight parameters of the improved VideoMAE model, perform streaming matrix multiplication calculation of the initial feature vectors and weight parameters within the shared memory region, and output the visible feature vectors. S45. Based on the image resolution and pixel block size before the sparse video frame sequence is divided, calculate the number of pixel blocks in the width and height directions of the image, multiply them to obtain the total number of pixel blocks, and construct a zero matrix with the dimension of the total number of pixel blocks multiplied by the number of feature channels as the full-size feature matrix. S46. Iterate through each position in the binary mask image one by one. When the value of the current position is 1, read the corresponding visible feature vector and copy it to the data row at the same index position in the full-size feature matrix. Otherwise, read the preset learnable mask token vector and copy it to the data row at the same index position in the full-size feature matrix to generate the filled feature matrix. The preset learnable mask token vector is a shared parameter vector used to characterize the semantics of the masked region, which is obtained by gradient descent optimization during the training phase of the improved VideoMAE model. S47. Add the filled feature matrix to the preset position encoding vector at corresponding positions to obtain the feature matrix with embedded position information. Calculate the dot product of each mask token vector with all other vectors in the entire image. Divide the dot product result by the preset scaling factor and input it into the Softmax function to calculate the normalized attention weight. Then, perform a weighted sum with the values ​​of all other vectors in the entire image to obtain the updated mask feature vector. Add this to the mask token vector before the update to complete the residual connection and feature reconstruction. The preset position encoding vector is a parameter vector obtained by gradient descent optimization during the training phase of the improved VideoMAE model, used to distinguish pixel blocks at different spatial locations, or a fixed constant vector calculated based on the waveform formulas of the sine and cosine functions. The preset scaling factor is the square root of the dimension of the visible feature vector. S48. Along the time axis, concatenate the reconstructed feature vector of the current frame with the feature vectors of the same position in the preceding and following frames, perform a one-dimensional convolution operation, calculate the weighted sum of features in the time dimension, perform layer normalization on the weighted sum result, and output a compressed visual token sequence.

[0024] In this implementation, an improved VideoMAE model combined with a direct-to-memory (DTM) read mechanism achieves efficient extraction and extreme compression of video features. Video frames are divided into pixel blocks, and initial feature vectors are generated. A random mask image is used to accurately remove 75% of non-critical spatiotemporal pixel blocks, significantly reducing redundant computation. A DTM read forward propagation layer is constructed, directly obtaining the base address pointer and skipping the standard tensor interface. Visible pixel blocks are written to DTM, and streaming matrix multiplication is performed within the shared memory of the computation unit (SM), completely avoiding memory copy latency. Through full-size feature matrix reconstruction and attention residual connections, visible features are accurately mapped back to the masked region, and temporal features are aggregated using one-dimensional convolution. This implementation ensures complete visual semantic restoration while achieving significant compression of token sequences, fundamentally breaking through the DTM bandwidth bottleneck and laying a solid underlying computational foundation for the rapid inference of subsequent large language models.

[0025] The improved VideoMAE model of this invention is similar to the original VideoMAE model in that both retain the core architecture of the mask autoencoder, which divides the video frame into non-overlapping pixel blocks and transforms them into initial feature vectors. Both use a random masking strategy to cover some pixel blocks, and both use a self-attention mechanism to calculate the association between visible pixel blocks and mask tokens. They use residual connections to reconstruct features and aggregate temporal features through convolution operations on the time axis.

[0026] The difference lies in that this invention breaks away from the computational paradigm of the original VideoMAE model, which relies on standard deep learning frameworks for conventional tensor loading and memory copying. Instead, it introduces a joint scheduling mechanism of GPU memory passthrough and shared memory. Building upon the original model's direct data loading via interface calls, this invention constructs a GPU memory direct-read forward propagation layer in step S43, disabling memory allocation and copy functions in the standard tensor loading interface, and directly obtaining the base address pointer of the contiguous GPU memory physical address pool for physical-level writing. Then, in step S44, abandoning the traditional global GPU memory computation method, a dedicated high-speed shared memory region is allocated within the computation unit SM. After loading data into this region, streaming matrix multiplication computation is directly performed within the shared memory.

[0027] Based on the aforementioned improvements, the beneficial effects of this invention are that, through direct memory writing and shared memory acceleration, the improved VideoMAE model can completely avoid the heavy data migration overhead and host-side memory allocation latency in the traditional framework, achieving zero copy and maximum bandwidth utilization in the visible feature extraction stage. This design significantly shortens the time consumed in a single forward computation, precisely allocating the saved computing resources to the feature reconstruction process after a high proportion of masking, significantly enhancing the model's ability to restore complex spatiotemporal semantics. While ensuring an extremely low token compression ratio and high semantic accuracy, it effectively suppresses the idle rate of computing units, providing strong underlying acceleration support for the high-concurrency decoding of subsequent large language models.

[0028] In this embodiment, S5 specifically includes: S51. Read the compressed visual token sequence and use it as an input tensor. Map it to the starting storage area of ​​the continuous video memory physical address pool. Call the embedding layer weights of the large language model. Perform matrix multiplication on the compressed visual token sequence in the computing unit SM. Map the visual features to the word vector space of the large language model to generate a visual embedding vector sequence. S52. Read the remaining physical address space in the contiguous video memory physical address pool after removing the space occupied by the visual embedding vector sequence, and mark it as a key-value cache dedicated storage area. During the decoding forward calculation of each layer of the large language model, calculate the query vector, key vector and value vector of the current token, and append the calculated key vector and value vector to the key-value cache dedicated storage area in the order of calculation. S53. Read the stored historical key vector and value vector from the key-value cache dedicated storage area, perform a dot product operation between the historical key vector and the query vector of the current token, calculate the attention score, and perform a weighted summation of the attention score and the value vector to obtain the output feature vector of the current layer. S54. Repeat the forward computation and key-value cache read / write operations layer by layer until the last layer of the large language model outputs. Normalize the output feature vector to generate visual semantic features and output them.

[0029] In this embodiment, S6 specifically includes: S61. Construct a dual-stream scheduling state machine for the Orca algorithm, read the instruction issue queue register inside the computing unit SM in real time, calculate the number of instructions to be executed in the current clock cycle, divide the number of instructions to be executed by the maximum instruction issue width of the computing unit SM to obtain the instruction queue fill rate, and when the instruction queue fill rate is lower than the preset fill rate threshold, determine that the computing unit SM is in the idle period of waiting for video memory, and generate a scheduling switching signal. The preset fill rate threshold is set to 0.25. S62. Respond to the scheduling switching signal, freeze the decoding process of the current video segment, read the hidden layer state matrix of each layer in the Transformer architecture of the large language model layer by layer, calculate the data volume of the hidden layer state matrix of each layer, apply for a snapshot storage area of ​​the same volume in the continuous video memory physical address pool, copy the hidden layer state matrix byte by byte to the snapshot storage area, read the token index position generated by the current decoding, and combine the token index position with the first address pointer of the snapshot storage area to form the decoding scene snapshot data packet. S63. Read the sparse video frame sequence of the next video segment as a micro-batch, align the data start address of the micro-batch to the base address of the continuous video memory physical address pool, call the computing unit SM in the idle period to perform the forward calculation of the improved VideoMAE model, extract the visual semantic features of the micro-batch, write the extracted visual semantic features into the free area in the continuous video memory physical address pool, and generate the feature pointer to be inferred. S64. After the visual semantic features of the micro-batch are written, an encoding completion signal is generated. Based on the first address pointer in the decoded snapshot data packet, the hidden layer state matrix in the snapshot storage area is copied back to the on-chip memory of the computing unit SM. Based on the Token index position, the context environment for generating the decoded text is restored, and the Token prediction calculation for the current video segment is continued. S65. Calculate the masking duration by the time difference between the start and end times of the micro-batch forward calculation in step S63. Read the busy / idle status parameters of the video memory controller to obtain the video memory loading wait time. Calculate the difference between the masking duration and the video memory loading wait time. When the difference is greater than zero, the masking is determined to be successful. Overlap and align the execution time period of the micro-batch calculation process with the time period of video memory data loading on the time axis, and output the video inference text result of the current video segment.

[0030] This invention utilizes an improved Orca algorithm to construct a dual-stream scheduling state machine, achieving precise capture of the idling period in large language model decoding and efficient masking of computational resources. By real-time reading of the instruction issuance queue and calculating the fill rate, the invention accurately determines the memory waiting idling period of the computing unit (SM) and triggers scheduling switching. Upon responding to the switching signal, it extracts and packages the hidden layer state matrix and token index layer by layer to generate a decoding snapshot, ensuring lossless suspension of the context environment. It then inserts micro-batch execution of the next video segment during the SM idling interval to perform forward computation and extract features. After completion, it instantly resumes the decoding process based on the snapshot. By comparing the computation time with the memory waiting time, it achieves perfect overlap and alignment of micro-batch execution and memory loading on the timeline. This invention can thoroughly utilize the underlying computing power in complex scenarios involving high-concurrency video streams and large model decoding, eliminating the waiting dead zone in traditional serial mechanisms. After micro-batch insertion masking, it significantly improves the utilization rate of hardware computing units, achieving a multiple increase in end-to-end inference throughput without increasing additional power consumption.

[0031] In this embodiment, S7 specifically includes: S71. Detect the signal that the video inference text result of the current video segment has been output, call the video memory release instruction, clear the data of the key-value cache special storage area stored in the continuous video memory physical address pool, and release the storage area from the lock. S72. Read the bus bandwidth utilization rate from the underlying resource status parameters, calculate the absolute value of the difference between the bus bandwidth utilization rate and the preset bandwidth threshold, and divide the absolute value of the difference by the bus physical peak bandwidth to obtain the bandwidth load fluctuation rate. S73. Multiply the preset baseline frame rate by 1 and the difference between the bandwidth load fluctuation rate to calculate the sampling frame rate of the next video segment. Multiply the preset baseline compression ratio by 1 and the sum of the bandwidth load fluctuation rate to calculate the token compression ratio of the next video segment. The preset baseline frame rate is set to 24fps. S74. Update the parameter configuration in S2 and S4 based on the calculated sampling frame rate and token compression ratio, jump to execute S1, read the video stream data and underlying resource status parameters of the next moment, and execute S1 to S6 in a loop until the video stream data reading is finished.

[0032] Example 1: To verify the feasibility of this invention in practice, it was applied to a visual large-scale model inference and monitoring platform for unmanned tractor-trailers and remotely controlled quay cranes at a fully automated container terminal in a coastal smart hub city. This terminal, a national-level core logistics node, has five 100,000-ton deep-water berths and over twenty remotely automated quay cranes, handling over 30,000 TEUs of containers daily. The terminal has deployed over 1,500 high-definition cameras and edge computing nodes, comprehensively covering the quay crane hoisting area, unmanned tractor-trailer transfer channels, and container yards. It needs to process massive video streams in real time to complete complex tasks such as vehicle collision avoidance warnings, abnormal container number identification and detection, and analysis of unauthorized personnel intrusion. Traditional visual large-scale model inference systems expose extremely serious underlying hardware bottlenecks when faced with such a large and high-frequency video stream input. Due to frequent memory allocation and data copying operations during data transfer using the standard tensor loading interface, the video memory allocation is highly fragmented, and the effective bandwidth utilization rate has long remained at a low level. Especially during peak hours when container trucks are frequently entering and leaving the port at night, concurrent inference of multiple high-resolution video streams can easily cause memory congestion. Computing units are often forced into a long period of idling while waiting for memory, resulting in high memory access latency. End-to-end inference throughput is severely suppressed, and critical warning screens are often delayed by several seconds or even dropped frames, which completely fails to meet the stringent requirements of automated terminals for millisecond-level response.

[0033] In actual deployment, this invention completely reconstructs the underlying inference scheduling logic of the large video model. When multiple dock monitoring video streams are input to the inference cluster in the form of sparse frame sequences, the system first takes over the feature extraction task through the improved VideoMAE model. At this stage, the model abandons the cumbersome data transfer mechanism in the traditional framework and directly activates the innovatively constructed direct-read forward propagation layer. This layer, by locking the base address pointer of the contiguous physical address pool of video memory, forcibly bypasses the memory allocation and data copy functions in the standard tensor loading interface, and seamlessly pours the feature vectors of visible pixel blocks into the shared memory of the computing unit in a streaming direct-write manner. This mechanism completely eliminates redundant copying of intermediate states in the extraction stage of the massive video feature matrix, maximizes the physical bandwidth of video memory, and fundamentally improves the parallelism of the underlying computing resources. Subsequently, the system seamlessly switches to the dual-stream scheduling state machine built by the Orca algorithm in the decoding stage. The state machine is like an extremely precise micro traffic commander, constantly monitoring the fill rate of the instruction issuance queue inside the computing unit. Once a very short idle period with a fill rate below a preset threshold is detected, the state machine freezes the decoding process of the current video segment within microseconds, packages the hidden layer state matrix into a snapshot, and then immediately hijacks idle computing units, forcibly inserting the micro-batch forward computation of the next video segment in the queue. By accurately calculating the computation time of the micro-batch and comparing it with the waiting time for memory loading, the system can perfectly overlap and align the originally mutually exclusive computation and memory access processes on the timeline, turning the originally wasted computing power window into actual effective output.

[0034] After three consecutive months of operation in a real-world online environment, this invention demonstrated a revolutionary performance leap in the complex and demanding operational scenarios of the port. To accurately quantify this improvement, the platform's operations team extracted real-world load data for two consecutive weeks during peak nighttime truck arrival and departure times in three core scenarios: the quay crane lifting area, the unmanned tractor transfer channel, and the container yard. This data was then compared in detail with the previous generation of traditional visual inference system before the deployment of this invention. Table 1. Comparative Analysis of Underlying Performance and Business Metrics of the Smart Terminal Multi-Scenario Video Large Model Inference System

[0035] As can be seen from the comparison data shown in Table 1, the large model low-level inference acceleration method proposed in this invention shows an overwhelming performance advantage over traditional systems in multi-channel concurrent video processing scenarios at smart terminals, especially in key indicators such as inference time, memory utilization, idle time suppression, and anomaly detection, achieving a leapfrog improvement.

[0036] In terms of inference response time, this invention achieves several-fold speedup in three types of high-load scenarios. For example, in a container yard with the highest concurrency pressure, the traditional system's single-path inference time is as high as 568.2ms, while this invention reduces it to 135.6ms through a direct memory read mechanism, completely reversing the predicament of computing power exhaustion under high concurrency. At the same time, the average latency of the warning screen is significantly reduced, from 2150ms to 485ms in the unmanned vehicle flow channel, gaining a decisive time window for collision avoidance intervention.

[0037] In terms of underlying hardware resource scheduling, traditional systems are constrained by cumbersome data copying, resulting in a video memory bandwidth utilization rate of less than 35% and a computing unit idle rate exceeding 40%. This invention utilizes a dual-stream scheduling state machine to precisely fill the idle period, stabilizing the bandwidth utilization rate to over 72% and suppressing the idle rate to around 5%, directly driving the end-to-end throughput to achieve a breakthrough from 345fps to 1028fps in the shore-bridge area.

[0038] At the operational assurance level, with the release of underlying performance, the failure rate of anomaly detection has been reduced from a maximum of 14.5% in traditional systems to below 2%. Overall, this invention, through hardware passthrough and micro-temporal overlap, fundamentally cures the persistent problem of large-scale visual models in ultra-large-scale video stream processing, and has extremely high engineering application value.

[0039] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for accelerating inference of video VLM models based on AI vision, characterized in that, Includes the following steps: S1. Synchronously collect the video stream data to be inferred and the underlying resource status parameters of the current inference hardware; S2. Based on the underlying resource status parameters, the video stream data is dynamically spatiotemporally sparsified to remove redundant background frames in the time dimension and high-frequency invalid perception areas in the spatial dimension, generating a sparse video frame sequence. S3. Based on the current GPU memory free threshold, pre-define and allocate a contiguous physical address pool in the memory. S4. Using the improved VideoMAE model, random masking operation is performed on non-critical spatiotemporal pixel blocks in sparse video frame sequences. A direct read forward propagation layer is constructed to obtain the base address pointer. Streaming matrix multiplication is performed on the visible unmasked pixel blocks in the computing unit SM to perform feature aggregation and reconstruction in the temporal and spatial dimensions, and output a compressed visual token sequence. S5. In the contiguous video memory physical address pool, the compressed visual token sequence is input into the large language model to perform forward computation, and the remaining physical address space is used to store the key-value cache generated during the decoding process, and the visual semantic features are output. S6. The Orca algorithm is used to process visual semantic features. In the forward calculation of the token in the current video segment decoding, if the computing unit SM is detected to be in the idle period of waiting for video memory, the current decoding state is paused and saved. The sparse video frame sequence of the next video segment is inserted into the idle SM as a micro-batch for execution. After completion, the decoding state is restored, and the video inference text result is output by masking the waiting delay. S7. After the current video segment inference is completed, release the key-value cache in the contiguous video memory physical address pool, and dynamically adjust the sampling frame rate and token compression ratio of the next video segment according to the bus bandwidth utilization. Repeat S1 to S6 until the video stream ends.

2. The method for accelerating video VLM model inference based on AI vision according to claim 1, characterized in that, S1 specifically includes: S11. Synchronously trigger video stream reading operation through system-level hardware polling interface to obtain video stream data to be inferred; S12. Call the GPU underlying monitoring driver to read the current available video memory capacity of the current inference hardware's video memory management unit. Read the operating system underlying video memory usage value, the deep learning framework static basic overhead value, and the minimum key-value cache volume value generated by a single decoding of a large language model from the pre-written configuration file. Add the above three read values ​​together to obtain the preset minimum safe video memory capacity. Calculate the difference between the current available video memory capacity and the preset minimum safe video memory capacity, and set it as the current GPU video memory idle threshold. S13. Call the GPU underlying monitoring driver to read the instruction issue count and completion count of all computing units (SMs) in the current clock cycle, calculate the ratio of instruction issue count to completion count, and record the ratio as the computing unit (SM) utilization rate. S14. Call the GPU underlying monitoring driver to read the number of bytes of data transfer between the memory controller and the computing unit SM within the current time window, read the peak physical bandwidth of the bus as specified by the GPU hardware factory, calculate the percentage of data transfer bytes to the peak physical bandwidth of the bus, and mark the percentage as the bus bandwidth utilization rate. S15. Pack and align the current GPU memory free threshold, computing unit SM occupancy rate and bus bandwidth utilization rate according to the sampling timestamp, and output the underlying resource status parameters.

3. The method for accelerating video VLM model inference based on AI vision according to claim 1, characterized in that, S2 specifically includes: S21. Read the SM occupancy rate of the underlying resource status parameters, divide the video stream data into video segments according to the time window, calculate the sum of the absolute values ​​of the differences between corresponding pixels between two adjacent video images, and when the sum of the absolute values ​​of the differences is lower than the preset static threshold, mark the next video image as a redundant background frame in the time dimension and remove it. S22. Calculate the average frame rate of the remaining video frames in the current video segment, read the bus bandwidth utilization rate in the underlying resource status parameters, and if the bus bandwidth utilization rate exceeds the preset bandwidth threshold, calculate the ratio of the average frame rate to the preset bandwidth threshold and round the ratio down as the spatial sampling step size. S23. If the bus bandwidth utilization does not exceed the preset bandwidth threshold, the space sampling step size is set to 1. S24. Divide the single-frame video image into grids according to the spatial sampling step size, calculate the variance of all pixel gray values ​​in each grid, and mark grids with variances lower than the preset texture threshold as high-frequency invalid perception regions in the spatial dimension and remove them. S25. After removing redundant background frames and high-frequency invalid sensing areas, the remaining video image frames are reassembled in chronological order to output a sparse video frame sequence.

4. The method for accelerating video VLM model inference based on AI vision according to claim 1, characterized in that, S3 specifically includes: S31. Read the current GPU memory free threshold from the underlying resource status parameters and obtain the maximum contiguous physical block size supported by the current inference hardware in a single memory allocation. S32. Compare the current GPU memory free threshold with the maximum contiguous physical block size for a single memory allocation. If the current GPU memory free threshold is greater than the maximum contiguous physical block size for a single memory allocation, then set the maximum contiguous physical block size for a single memory allocation as the size of the allocation pool. S33. If the current GPU memory free threshold is less than or equal to the maximum contiguous physical block size in a single memory allocation, then set the current GPU memory free threshold to the size of the pool to be allocated. S34. Call the GPU low-level driver interface, lock the physical address mapping table of video memory, find a free area in video memory with a volume greater than or equal to the volume of the pool to be allocated, and prevent the operating system from performing page swapping operations on the free area. S35. Obtain the physical starting address of the free area as the base address pointer, bind the base address pointer to the volume of the pool to be allocated, mark it as a continuous video memory physical address pool and output it.

5. The method for accelerating video VLM model inference based on AI vision according to claim 1, characterized in that, S4 specifically includes: S41. Divide each video frame image in the sparse video frame sequence into non-overlapping pixel blocks. According to the encoder input size requirements of the improved VideoMAE model, calculate the RGB three-channel mean of all pixels in the pixel block and concatenate them into an initial feature vector. S42. Construct a random mask generator to generate a random number matrix with values ​​ranging from 0 to 1. Set the positions in the random number matrix with values ​​greater than the preset mask ratio to 1, and set the positions with values ​​less than or equal to the preset mask ratio to 0 to form a binary mask image. Mark the pixel blocks corresponding to the positions set to 0 in the binary mask image as non-critical spatiotemporal pixel blocks, and mark the pixel blocks corresponding to the positions set to 1 as visible unmasked pixel blocks. S43. Construct a forward propagation layer for direct reading of video memory, read the base address pointer of the continuous video memory physical address pool, disable the memory allocation and data copy functions in the standard tensor loading interface, and directly write the initial feature vectors corresponding to the visible unmasked pixel blocks in a continuous arrangement of video memory physical addresses. S44. Within the computing unit SM, allocate a shared memory region, load the initial feature vectors of visible unmasked pixel blocks stored in the continuous video memory physical address pool into the shared memory region, read the weight parameters of the improved VideoMAE model, perform streaming matrix multiplication calculation of the initial feature vectors and weight parameters within the shared memory region, and output the visible feature vectors. S45. Based on the image resolution and pixel block size before the sparse video frame sequence is divided, calculate the number of pixel blocks in the width and height directions of the image, multiply them to obtain the total number of pixel blocks, and construct a zero matrix with the dimension of the total number of pixel blocks multiplied by the number of feature channels as the full-size feature matrix. S46. Iterate through each position in the binary mask image one by one. When the value of the current position is 1, read the corresponding visible feature vector and copy it to the data row at the same index position in the full-size feature matrix. Otherwise, read the preset learnable mask token vector, copy it to the data row at the same index position in the full-size feature matrix, and generate the filled feature matrix. S47. Add the filled feature matrix to the preset position encoding vector at the corresponding positions to obtain the feature matrix with embedded position information. Calculate the dot product of each mask token vector with all other vectors in the entire image. Divide the dot product result by the preset scaling factor and input it into the Softmax function to calculate the normalized attention weight. Then, perform a weighted sum with the values ​​of all other vectors in the entire image to obtain the updated mask feature vector. Add the updated mask token vector to complete the residual connection and feature reconstruction. S48. Along the time axis, concatenate the reconstructed feature vector of the current frame with the feature vectors of the same position in the preceding and following frames, perform a one-dimensional convolution operation, calculate the weighted sum of features in the time dimension, perform layer normalization on the weighted sum result, and output a compressed visual token sequence.

6. The method for accelerating video VLM model inference based on AI vision according to claim 1, characterized in that, S5 specifically includes: S51. Read the compressed visual token sequence and use it as an input tensor. Map it to the starting storage area of ​​the continuous video memory physical address pool. Call the embedding layer weights of the large language model. Perform matrix multiplication on the compressed visual token sequence in the computing unit SM. Map the visual features to the word vector space of the large language model to generate a visual embedding vector sequence. S52. Read the remaining physical address space in the contiguous video memory physical address pool after removing the space occupied by the visual embedding vector sequence, and mark it as a key-value cache dedicated storage area. During the decoding forward calculation of each layer of the large language model, calculate the query vector, key vector and value vector of the current token, and append the calculated key vector and value vector to the key-value cache dedicated storage area in the order of calculation. S53. Read the stored historical key vector and value vector from the key-value cache dedicated storage area, perform a dot product operation between the historical key vector and the query vector of the current token, calculate the attention score, and perform a weighted summation of the attention score and the value vector to obtain the output feature vector of the current layer. S54. Repeat the forward computation and key-value cache read / write operations layer by layer until the last layer of the large language model outputs. Normalize the output feature vector to generate visual semantic features and output them.

7. The method for accelerating video VLM model inference based on AI vision according to claim 1, characterized in that, S6 specifically includes: S61. Construct a dual-stream scheduling state machine for the Orca algorithm, read the instruction issue queue register inside the computing unit SM in real time, calculate the number of instructions to be executed in the current clock cycle, divide the number of instructions to be executed by the maximum instruction issue width of the computing unit SM to obtain the instruction queue fill rate, and when the instruction queue fill rate is lower than the preset fill rate threshold, determine that the computing unit SM is in the idle period of waiting for video memory and generate a scheduling switching signal. S62. Respond to the scheduling switching signal, freeze the decoding process of the current video segment, read the hidden layer state matrix of each layer in the Transformer architecture of the large language model layer by layer, calculate the data volume of the hidden layer state matrix of each layer, apply for a snapshot storage area of ​​the same volume in the continuous video memory physical address pool, copy the hidden layer state matrix byte by byte to the snapshot storage area, read the token index position generated by the current decoding, and combine the token index position with the first address pointer of the snapshot storage area to form the decoding scene snapshot data packet. S63. Read the sparse video frame sequence of the next video segment as a micro-batch, align the data start address of the micro-batch to the base address of the continuous video memory physical address pool, call the computing unit SM in the idle period to perform the forward calculation of the improved VideoMAE model, extract the visual semantic features of the micro-batch, write the extracted visual semantic features into the free area in the continuous video memory physical address pool, and generate the feature pointer to be inferred. S64. After the visual semantic features of the micro-batch are written, an encoding completion signal is generated. Based on the first address pointer in the decoded snapshot data packet, the hidden layer state matrix in the snapshot storage area is copied back to the on-chip memory of the computing unit SM. Based on the Token index position, the context environment for generating the decoded text is restored, and the Token prediction calculation for the current video segment is continued. S65. Calculate the masking duration by the time difference between the start and end times of the micro-batch forward calculation in step S63. Read the busy / idle status parameters of the video memory controller to obtain the video memory loading wait time. Calculate the difference between the masking duration and the video memory loading wait time. When the difference is greater than zero, the masking is determined to be successful. Overlap and align the execution time period of the micro-batch calculation process with the time period of video memory data loading on the time axis, and output the video inference text result of the current video segment.

8. The method for accelerating video VLM model inference based on AI vision according to claim 1, characterized in that, Specifically, S7 includes: S71. Detect the signal that the video inference text result of the current video segment has been output, call the video memory release instruction, clear the data of the key-value cache special storage area stored in the continuous video memory physical address pool, and release the storage area from the lock. S72. Read the bus bandwidth utilization rate from the underlying resource status parameters, calculate the absolute value of the difference between the bus bandwidth utilization rate and the preset bandwidth threshold, and divide the absolute value of the difference by the bus physical peak bandwidth to obtain the bandwidth load fluctuation rate. S73. Multiply the preset baseline frame rate by 1 and the difference between the bandwidth load fluctuation rate to calculate the sampling frame rate of the next video segment. Multiply the preset baseline compression ratio by 1 and the sum of the bandwidth load fluctuation rate to calculate the token compression ratio of the next video segment. S74. Update the parameter configuration in S2 and S4 based on the calculated sampling frame rate and token compression ratio, jump to execute S1, read the video stream data and underlying resource status parameters of the next moment, and execute S1 to S6 in a loop until the video stream data reading is finished.

Images