Deep learning based video content semantic understanding system

Abstract

The application belongs to the technical field of image data processing, and relates to a video content semantic understanding system based on deep learning, which comprises an initial matrix generation module, an environment state evaluation module, a control channel establishment module, a general feature acquisition module, a dynamic decoding control module and a knowledge fusion reasoning module. The initial matrix generation module processes video frames to generate an initial video pixel block matrix. The environment state evaluation module generates environment state evaluation parameters. The control channel establishment module constructs a conventional transition control channel by combining a parameter smoothing matrix and network parameters. The general feature acquisition module analyzes the channel, reduces the dimension of data, and generates general visual initial features. The dynamic decoding control module extracts spatiotemporal features, calculates a decay rate to generate a spatiotemporal feature pulse, and adjusts an effective area of an attention mask matrix. The knowledge fusion reasoning module fuses a knowledge graph risk prior tensor to reshape a self-attention gate query matrix, and combines initial features to generate a structured video semantic event. The application solves the problem that the analysis efficiency and accuracy are limited due to the fixed allocation of computing resources and the lack of common sense verification in existing video analysis models.

Description

Technical Field

[0001] This invention belongs to the technical field of image data processing and relates to a video content semantic understanding system based on deep learning. Background Technology

[0002] In applications such as smart cities, public safety monitoring, and industrial automation, video surveillance systems are crucial for acquiring environmental information and ensuring safe production. Intelligent semantic understanding and anomaly identification of massive, continuous video data streams can effectively improve management efficiency and ensure the stable operation of critical facilities. However, existing video understanding technologies often face challenges in computing power allocation and scheduling when processing massive amounts of data, and the generalization and common-sense reasoning capabilities of deep learning models need further improvement.

[0003] For video data processing, existing technologies mainly employ two approaches. The first is a unified processing model, which uses a fixed deep learning analysis architecture. It performs forward inference computation of equal intensity on all video time segments through a combination of convolutional and recurrent neural networks or a video transformer model. The second is a two-stage filtering approach. This approach first deploys low-computational-cost motion detection algorithms such as optical flow or background subtraction for initial screening. When significant changes are detected, a subsequent computationally intensive semantic analysis model is activated for in-depth analysis.

[0004] The aforementioned technical solutions all have certain limitations in complex and ever-changing scenarios. The unified processing model uses equal computing power to process a large amount of static or redundant background information in surveillance videos, increasing unnecessary hardware and energy costs and reducing the system's concurrent processing capacity and the economic benefits of large-scale deployment. The effectiveness of the two-stage filtering scheme relies heavily on the front-end change detection algorithm. Such simple algorithms are sensitive to non-target dynamic interference such as changes in lighting or shadows, easily generating false triggers and frequently waking up the heavily used back-end model, making it difficult to effectively optimize computing power allocation. Furthermore, this mechanism, which relies on changes in motion features, is insufficient in responding to subtle but semantically important hidden behaviors, posing a risk of missed detections. Moreover, the analysis logic of the above solutions is mainly driven by pure data, with little integration of common-sense knowledge. When encountering scene combinations outside the training data and with illogical reasoning, the model is prone to semantic misjudgments based solely on local features. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a video content semantic understanding system based on deep learning.

[0006] A deep learning-based video content semantic understanding system includes: The initial matrix generation module is used to acquire the original continuous frame stream of video, perform decoding and size transformation, and generate the initial video pixel block matrix; The environmental status assessment module is used to obtain the system processing cycle and integrate it with the spatial arrangement attributes of the initial video pixel block matrix to generate environmental status assessment parameters. The control channel establishment module is used to smooth the initial video pixel block matrix based on environmental state assessment parameters to obtain video feature data, and to construct a conventional transition control channel in combination with network layer parameters. The general feature acquisition module is used to parse the conventional transition control channel, input the video feature data into the deep vision analysis architecture for feature dimensionality reduction, and generate general visual initial features; The dynamic decoding control module is used to extract the spatiotemporal feature vectors of the initial video pixel block matrix to form a data matrix, calculate the attenuation rate of the eigenvalue distribution of the covariance matrix to generate spatiotemporal feature pulses, and dynamically adjust the effective area of ​​the attention mask matrix of the deep vision analysis architecture accordingly. The knowledge fusion reasoning module is used to obtain the risk prior tensor output by the knowledge graph, and fuse it with the original visual query matrix to reshape the self-attention gated query matrix. Based on the reshaped query matrix and general visual initial features, dot product attention calculation is performed in combination with common sense constraint threshold to generate structured video semantic events.

[0007] In a further embodiment of the present invention, the initial matrix generation module is specifically used for: Decode the original continuous video frame stream, extract the video stream state parameters, and assign a timestamp to each decoded frame. Compare the pixel size of the decoded frame with the resolution in the video stream status parameters to generate a validity flag for the initial running parameters; In response to a preset number of consecutive initial running parameter validity flags being true, the corresponding consecutive multiple frames of original pixel images are treated as a processing batch. Perform spatial size transformation and pixel value normalization on each frame in the processing batch; The pixel data, after size transformation and normalization, are stacked in chronological order to construct the initial video pixel block matrix.

[0008] In a further embodiment of the present invention, the environmental status assessment module is specifically used for: Query the current hardware status indicators of the computing platform and allocate contiguous memory space for the initial video pixel block matrix; Extract the dimensional information of the initial video pixel block matrix, and combine it with the allocated memory space address pointer to generate the basic running boundary conditions; The basic operating boundary conditions and system processing cycle are encapsulated to generate environmental status assessment parameters.

[0009] In a further embodiment of the present invention, the control channel establishment module is specifically used for: Analyze environmental status assessment parameters, read the system default resource pool status, and establish intermediate layer signal transmission rules; Load the general network layer parameters according to the intermediate layer signal transmission rules; Based on the preset filtering window radius, a sliding window filtering operation is applied to the time dimension of the initial video pixel block matrix to generate video feature data; Video feature data and general network layer parameters are written into pre-allocated video memory space to construct an intermediate buffer. The memory address pointing to the intermediate buffer, the handle of the general network layer parameters, and the system processing cycle are encapsulated to generate a regular transition control channel.

[0010] In a further embodiment of the present invention, the general feature acquisition module is specifically used for: Parse the data entry node of the conventional transition control channel and load the initial gating threshold of the deep vision analysis architecture; Input video feature data into the feature dimensionality reduction operator within the deep visual analysis architecture; The pixel spatial information is mapped into a sequence of feature vectors of the target dimension by using feature dimensionality reduction operators; The feature vector sequence is temporarily stored in a pre-defined circular buffer to construct a feature extraction cache. Encapsulate the memory address pointer to the feature extraction cache and the feature sequence dimension information to generate general visual initial features.

[0011] In a further embodiment of the present invention, when the dynamic decoding control module extracts the spatiotemporal feature vectors of the initial video pixel block matrix to form a data matrix and calculates the attenuation rate of the eigenvalue distribution of the covariance matrix to generate spatiotemporal feature pulses, it is specifically used for: Features are extracted independently from each frame in the initial video pixel block matrix, the spatial dimension information of the feature map is preserved, and a spatiotemporal feature map tensor is generated. Based on a preset sliding window mechanism, the feature map tensor moves along the time dimension, flattens it, and extracts the feature vectors within the window to form a data matrix. Calculate the covariance matrix of the data matrix and perform eigenvalue decomposition to obtain the eigenvalues ​​in descending order; After normalizing the eigenvalues, calculate the eigenvalue distribution decay rate; Spatiotemporal characteristic pulses are generated based on the attenuation rate of eigenvalue distribution.

[0012] In a further aspect of this invention, the dynamic decoding control module, when dynamically adjusting the effective area of ​​the attention mask matrix in the deep visual analysis architecture, is specifically used for: Monitor the amplitude at each time point in the spatiotemporal characteristic pulse; In response to the amplitude not reaching the system's set threshold, the off-diagonal elements in the attention mask matrix at the corresponding time point within the deep visual analysis architecture are cleared to zero. In response to an amplitude greater than or equal to a system-set threshold, the effective area of ​​the attention mask matrix within the corresponding time point and preset time window is unlocked, and a complete self-attention calculation is performed.

[0013] In a further aspect of this invention, when the knowledge fusion reasoning module obtains the risk prior tensor output by the knowledge graph, it is specifically used for: Based on the general visual initial features, basic entities are identified, and an initial entity list containing category labels is generated; Input the category labels as entry nodes into the external knowledge graph array; Random walks are performed on the node network of the external knowledge graph array to generate path sequences; Convert the path sequence into a path embedding vector; The path embedding vector is projected into a risk prior tensor that matches the dimension of the visual feature space by using a multilayer perceptron.

[0014] In a further aspect of this invention, the knowledge fusion reasoning module, when generating structured video semantic events, is specifically used for: Retain the feature extraction channels within the deep visual analysis architecture used to generate the native visual query matrix, and generate the native visual query matrix. Extract the risk prior tensor and reshape the self-attention gated query matrix by performing a gated addition operation with the native visual query matrix; A linear transformation is performed on the general visual initial features to generate a visual key matrix and a visual value matrix. Dot product multiplication attention calculation is then performed based on the reshaped self-attention gating query matrix, visual key matrix, and visual value matrix. In the dot product multiplication attention calculation process, a common sense constraint threshold is set to generate a sparse constraint mask matrix, and dot product terms smaller than the threshold are mapped to negative infinity. A feature blocking mechanism is introduced: if all dot product terms corresponding to a query vector are smaller than the threshold, the attention output feature vector corresponding to the query vector is forced to zero, and finally the structured video semantic event is output.

[0015] In a further embodiment of the present invention, after performing normalization processing on the feature values, the dynamic decoding control module calculates the feature value distribution attenuation rate. When generating spatiotemporal feature pulses based on the feature value distribution attenuation rate, it is specifically used for: Extract the eigenvalues ​​obtained after eigenvalue decomposition of the covariance matrix; Normalize each feature value and calculate its proportion. Before calculating the natural logarithm, add a constant to the proportion of each normalized feature value. Select the top K dominant feature values ​​whose cumulative proportion reaches the preset energy threshold and construct a two-dimensional data point set composed of the dominant feature value index and the corresponding natural logarithm value. Perform least squares linear regression on a two-dimensional data point set to obtain the slope of the regression line; Extract the absolute value of the slope, introduce a smooth scaling adjustment coefficient, and calculate the power function value with the natural constant as the base and the negative of the product of the absolute value of the slope and the coefficient as the exponent. The power function value is determined as the amplitude of the spatiotemporal characteristic pulse corresponding to that time point.

[0016] In summary, the present invention has the following beneficial technical effects: 1. By extracting the spatiotemporal features of the initial video pixel block matrix and calculating the attenuation rate of its covariance matrix eigenvalue distribution, a spatiotemporal feature pulse is generated as a computational state control signal. Based on the comparison between this pulse amplitude and a preset threshold, the system dynamically adjusts the effective region of the attention mask matrix: when the pulse amplitude is below the preset threshold, the weights of the off-diagonal elements of the attention mask matrix are reset to zero, guiding the data flow through a low-power bypass; when the pulse amplitude reaches or exceeds the preset threshold, the attention mask matrix in a specific temporal domain is unlocked to perform deep temporal feature fusion. This non-uniform temporal decoding mechanism establishes a dynamic coupling relationship between computational resource allocation and video stream information volume, reducing unnecessary matrix operation overhead when processing highly redundant video segments, thereby optimizing the overall system's energy efficiency and data throughput.

[0017] 2. By acquiring the risk prior tensor output from an external knowledge graph array and fusing it with the native visual query matrix within a pre-built deep visual analysis architecture, a self-attention gating query matrix is ​​reshaped. This mechanism introduces structured common-sense logic into the underlying feature computation flow of the neural network, ensuring that the model, when performing dot-product attention calculations, not only relies on feature-driven visual data but is also subject to logical constraints from cross-modal knowledge. This helps the model suppress visually similar but anomalous responses that violate common-sense rules in the feature matching space, thereby reducing false positives caused by complex scene interference or model generalization limitations, and improving the accuracy of outputting structured video semantic events.

[0018] 3. A two-stage analysis link connecting feature detection and logical reasoning is constructed by combining a dynamic computing power scheduling mechanism based on spatiotemporal feature pulses with a common-sense logic verification mechanism based on risk prior tensors. In the first stage, the spatiotemporal feature pulse mechanism is used to quickly filter out suspicious time windows with significant changes in information entropy at a low computational cost, avoiding indiscriminate heavy reasoning on global continuous data. In the second stage, the knowledge fusion analysis module is triggered only for the selected suspicious time windows to perform refined semantic judgment. This overall architecture controls the computing power consumption during the analysis process while compensating for the shortcomings of a single visual feature model by introducing prior common sense, thus balancing the processing efficiency and recognition accuracy of the video semantic understanding system. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. The drawings are used to provide a further understanding of the present invention.

[0020] Figure 1 This is a schematic diagram of the framework in the embodiments of this application.

[0021] Figure 2 This is a flowchart illustrating an embodiment of this application.

[0022] Figure 3 This is a comparison chart showing the decay trend of the eigenvalues ​​of the covariance matrix of the dynamic decoding control module in the embodiments of this application under static and rapidly changing scenarios.

[0023] Figure 4 This is a schematic diagram of the threshold triggering mechanism for the change of spatiotemporal characteristic pulse amplitude over time in an embodiment of this application.

[0024] Figure 5 This is a schematic diagram of the structural form of the attention mask matrix under different pulse amplitudes in the embodiments of this application.

[0025] Figure 6 This is a Pareto front curve of the effect of the smooth scaling adjustment coefficient on the pulse trigger recall rate and computing power consumption ratio in the embodiments of this application. Detailed Implementation

[0026] The following is in conjunction with the appendix Figures 1-6 A preferred description of the present invention is provided below.

[0027] See attached document Figures 1-6 This invention proposes a video content semantic understanding system based on deep learning, comprising the following modules: The initial matrix generation module is used to acquire the original continuous frame stream of video, perform decoding and size transformation, and generate the initial video pixel block matrix; The environmental status assessment module is used to obtain the system processing cycle and integrate it with the spatial arrangement attributes of the initial video pixel block matrix to generate environmental status assessment parameters. The control channel establishment module is used to smooth the initial video pixel block matrix based on environmental state assessment parameters to obtain video feature data, and to construct a conventional transition control channel in combination with network layer parameters. The general feature acquisition module is used to parse the conventional transition control channel, input the video feature data into the deep vision analysis architecture for feature dimensionality reduction, and generate general visual initial features; The dynamic decoding control module is used to extract the spatiotemporal feature vectors of the initial video pixel block matrix to form a data matrix, calculate the attenuation rate of the eigenvalue distribution of the covariance matrix to generate spatiotemporal feature pulses, and dynamically adjust the effective area of ​​the attention mask matrix of the deep vision analysis architecture accordingly. The knowledge fusion reasoning module is used to obtain the risk prior tensor output by the knowledge graph, and fuse it with the original visual query matrix to reshape the self-attention gated query matrix. Based on the reshaped query matrix and general visual initial features, dot product attention calculation is performed in combination with common sense constraint threshold to generate structured video semantic events.

[0028] In one embodiment of the present invention, the initial matrix generation module is configured to perform the following steps: decode the original continuous video frame stream, extract video stream state parameters, and assign a timestamp to each decoded frame; compare the pixel size of the decoded frame with the resolution in the video stream state parameters to generate an initial running parameter validity flag; in response to a preset number of consecutive initial running parameter validity flags being true, treat the corresponding consecutive multiple frames of original pixel images as a processing batch; perform spatial size transformation and pixel value normalization processing on each frame image within the processing batch; and stack the pixel data after size transformation and normalization processing in chronological order to construct an initial video pixel block matrix.

[0029] A central server equipped with a graphics processing unit receives raw, continuous video frame streams in H.264 or H.256 encoded format from front-end surveillance cameras via a real-time transmission protocol. These raw, continuous video frame streams refer to the raw video signal data continuously and uninterruptedly collected and uploaded by industrial-grade network cameras deployed in specific monitoring scenarios, such as road intersections, bank lobbies, or substations.

[0030] First, it calls an internally integrated multimedia processing framework such as FFmpeg to decode the original continuous video frame stream. During the decoding process, it parses the metadata embedded in the original continuous video frame stream and establishes a metadata structure including the frame rate. Resolution And the video stream status parameters encoded in color space.

[0031] Specifically, frame rate This is a key basis for guiding the sampling period of subsequent timing analysis modules. Based on different application scenarios and their varying real-time requirements, its typical value is set to 25fps or 30fps to ensure the smoothness of the video footage matches the normal human visual perception. Resolution The spatial resolution of a single frame is defined, and usually 1920px×1080px or higher is selected based on the requirements of the monitoring range and target recognition accuracy.

[0032] Assign a monotonically increasing timestamp to each decoded frame and perform basic integrity checks, such as verifying that the pixel size of the decoded frame matches the resolution in the video stream state parameters. If the check is consistent, a boolean initial running parameter validity flag is generated.

[0033] Confirmed Continuity After the initial runtime parameters of the frame are set to true, the original pixel images are grouped together as a batch. Then, a uniform size transformation is performed using bilinear or bicubic interpolation algorithms to adapt it to the standard input resolution. At the same time, pixel value normalization is performed, linearly mapping the pixel values ​​of each color channel from the original integer range of 0 to 255 to the floating-point range of 0.0 to 1.0.

[0034] Ultimately, this The pixel data of the frames, after being resized and normalized, are stacked in chronological order to construct a [frame structure]. A four-dimensional floating-point tensor, which is the initial video pixel block matrix. This serves as the standard input data structure for subsequent processing.

[0035] It should be noted that the initial video pixel block matrix It is a standard data container constructed to adapt to deep neural network computing frameworks. Among them, the time dimension... To handle the number of frames contained in a batch, the value ranges from 16 to 64. This setting balances the need to capture sequential actions with the need to control memory overhead; sequences that are too short are difficult to form a coherent context, while sequences that are too long cause a surge in computation; spatial dimension This is to adapt to the input size of the pre-trained visual model, usually set to 224px × 224px or 384px × 384px; color channel dimensions It is usually 3, corresponding to the three color channels: red, green, and blue.

[0036] In one example of this invention, in a scenario of monitoring urban traffic intersections, the front-end camera generates a continuous stream of raw video frames in H.264 encoding format. After receiving this data stream, the central server calls the FFmpeg decoding library for processing. First, it parses the video stream status parameters to confirm its frame rate. At 25fps, resolution 1920px × 1080px. Set batch size. Set the value to 16, and set the standard input resolution for subsequent models. The initial resolution is 224px × 224px. At the start of processing, 16 frames of video are decoded consecutively, and the size of each frame is checked to ensure it is 1920px × 1080px. Simultaneously, the timestamp interval between adjacent frames is confirmed to be stable at approximately 40ms, thus validating the initial operating parameters. Next, an interpolation algorithm is used to downsample the resolution of each frame from 1920 × 1080 to 224 × 224. Then, the values ​​of each pixel channel are linearly normalized to the range of 0.0 to 1.0. Finally, these 16 processed two-dimensional pixel planes are stacked in chronological order to generate a four-dimensional floating-point tensor of shape 16 × 224 × 224 × 3, which is the initial video pixel block matrix. .

[0037] In one embodiment of the present invention, the environmental status assessment module is used to perform the following operations: query the hardware status indicators of the current computing platform and allocate contiguous memory space for the initial video pixel block matrix; extract the dimensional information of the initial video pixel block matrix and combine it with the allocated memory space address pointer to generate basic operating boundary conditions; and encapsulate the basic operating boundary conditions and the system processing cycle to generate environmental status assessment parameters.

[0038] When the central server receives the initial video pixel block matrix Then, the system's main control process immediately initiates the routine system resource assessment mechanism. This mechanism is a preprocessing procedure to ensure stable operation of computing tasks. Its core is to proactively query the hardware status to avoid computational interruptions caused by insufficient resources, such as memory overflow or core overheating. This mechanism calls underlying hardware management interfaces, such as the NVIDIA management library, to query the real-time status of the graphics processing unit, obtaining key indicators including remaining memory capacity, core load rate, and core temperature.

[0039] Based on the query results, the underlying runtime stack of the computing platform is dynamically configured. For example, the graphics processing unit device number used for this calculation is specified by setting CUDA environment variables, and an independent computing flow is created for it.

[0040] The system controller initializes the video pixel block matrix. The spatial arrangement attributes are analyzed, that is, its four-dimensional information is read. The required video memory size for processing the matrix and its subsequent products is calculated based on the dimension information. Then, a video memory allocation function is called to reserve a contiguous block of memory on the graphics processing unit. Successful execution of this process generates a set of basic operational boundary conditions, which encapsulate a device pointer to the allocated video memory region and the initial video pixel block matrix. Dimensional information.

[0041] It should be noted that the basic operational boundary conditions are data structures, designed to clearly convey the location and logical structure of data in physical video memory to downstream processing modules, ensuring the correctness and efficiency of data access. These structures must at least include a matrix for storing the initial video pixel blocks. The device memory starting address pointer, and records its four dimensions. Metadata for specific numerical values.

[0042] Extract the frame rate from the established video stream state parameters. And calculate the system processing cycle corresponding to a single frame. In this embodiment, the system processing cycle is... It is based on the frame rate of the original video continuous frame stream. The calculated theoretical processing time interval provides a unified time reference for subsequent operations requiring timing synchronization. This is achieved by relating the basic operational boundary conditions to the system processing cycle. The data is integrated and encapsulated into a structured data object with unified temporal characteristics. This object is the environmental status assessment parameter, which will serve as the benchmark for subsequent modules to perform resource scheduling and synchronization control.

[0043] Specifically, the environmental status assessment parameters integrate the data location, data format, and time base of this processing and use them as task descriptors for various modules within the system to call in order to complete the initial configuration of the execution environment.

[0044] In one example of the present invention, based on the generated initial video pixel block matrix with a shape of 16×224×224×3... And the frame rate is known. The frame rate is 25fps. The initial video pixel block matrix has been received. Next, the hardware management interface was used to check that the currently specified graphics processing unit had 6.5GB of remaining video memory and a core load of 15%, indicating sufficient resources. Then, the initial video pixel block matrix was analyzed. The dimension, to obtain , , , The required storage size for this matrix, with a shape of 16×224×224×3 and data type of single-precision floating-point numbers, is calculated to be approximately 9.2MB. Immediately, a contiguous block of 9.2MB of memory is allocated on the graphics processing unit, and a device memory address pointer, such as 0x1A00B200, is obtained. At this point, the basic operational boundary conditions are generated, consisting of a data structure containing the pointer 0x1A00B200 and the dimension array [16, 224, 224, 3]. Simultaneously, the system processing cycle is calculated. Subsequently, the basic operational boundary conditions and system processing cycle are integrated into environmental state assessment parameters. These structured parameters can be represented as JSON objects and directly output to downstream modules for their use.

[0045] In one embodiment of the present invention, the control channel establishment module is used to perform the following operations: parse the environmental state assessment parameters, read the system default resource pool state, and establish intermediate layer signal transmission rules; load general network layer parameters according to the intermediate layer signal transmission rules; apply sliding window filtering operation to the time dimension of the initial video pixel block matrix based on the preset filtering window radius to generate video feature data; write the video feature data and general network layer parameters into the pre-allocated video memory space to construct an intermediate buffer; and encapsulate the memory address pointing to the intermediate buffer, the handle of the general network layer parameters, and the system processing cycle to generate a conventional transition control channel.

[0046] The system's main control module first parses the generated environmental state assessment parameters and extracts the initial video pixel block matrix from them. Device memory address pointers, dimension information, and system processing cycles Based on the internal system scheduling configuration file, such as a preset XML or YAML format configuration file, the system reads the default resource pool state corresponding to the current "normal processing" mode. This state defines the storage path of the processing function identifier to be called and the required network model parameters. This process establishes the intermediate layer signaling rules used to guide the data flow.

[0047] The intermediate layer signal transmission rules are configured as a static routing table, which binds different task stages to the corresponding algorithm modules or model weight files to standardize the data processing flow called by the network model.

[0048] The system follows the intermediate layer signal transmission rules, loading the general network layer parameters pointed to by the path from the hard disk into the graphics processing unit's video memory. These general network layer parameters specifically refer to the weights of a non-task-specific neural network model, such as a simple network containing only two to three convolutional layers. Their purpose is not to perform complex semantic recognition, but rather to act as a learnable filter to perform preliminary feature stabilization of the signal; and to process the initial video pixel block matrix stored at the device memory address pointer. Perform a standard pre-smoothing process on the signal. This process aims to reduce high-frequency noise in the original video signal, specifically by smoothing the initial video pixel block matrix. A sliding window filter operation is applied to the time dimension.

[0049] The smoothed data after processing, along with the loaded general network layer parameters, are written into pre-allocated video memory, forming an intermediate buffer. It should be noted that the intermediate buffer is a logical concept, corresponding to one or more regions within the graphics processing unit's video memory. It stores the pre-processed data and the model parameters to be called, forming a set of resources awaiting use.

[0050] Extract the new video memory address pointing to the intermediate cache, the general network parameter handle, and the processing cycle, and encapsulate them uniformly. This generates a regular transition control channel adapted to subsequent deep processing cycles.

[0051] In this embodiment, the conventional transition control channel is a standardized data structure that serves as an interface between software modules. It encapsulates all the entry information required to execute the next operation, including data pointers, model handles, and timing control parameters, thereby achieving modular decoupling of the processing flow.

[0052] The standard signal pre-smoothing process can be achieved using a time-domain moving average filtering algorithm, the calculation formula of which is as follows: ; In the formula, The timestamp in the initial video pixel block matrix is A single-frame pixel block matrix. This is a new single-frame pixel block matrix generated at the same timestamp after smoothing. is a positive integer representing the radius of the filtering window, which defines the number of adjacent frames involved in the averaging operation; For local frame indexes within the sliding window, For timestamps The corresponding adjacent single-frame pixel block matrix.

[0053] Filter window radius The value is usually set to 1 or 2, which is a trade-off between effectively suppressing random noise and avoiding excessive smoothing that leads to legitimate motion blur.

[0054] In one example of this invention, the environmental state evaluation parameter {"memory_pointer":"0x1A00B200","dimensions":[16,224,224,3],"processing_interval_ms":40} is received. The system main control module parses this parameter and queries its internal schedule_config.yaml file, finding the rule defined under the default_transition entry: {"function":"temporal_smooth","weights":" / models / lite_filter.pth"}, thus establishing the intermediate layer signal transmission rule. The system then loads the general network layer parameters from the path / models / lite_filter.pth into the video memory. Next, the initial video pixel block matrix with dimensions of 16×224×224×3 stored at address 0x1A00B200 is processed. Perform standard signal pre-smoothing processing. Set the filter window radius. Then for time index from arrive Each frame, its smoothed new pixel matrix Through calculation The result is obtained. For frames 0 and 15 at the boundary, edge pixel copying is used for padding calculation. After processing, a new smoothed video pixel block matrix is ​​generated and stored at a new video memory address, for example, 0x2C00D400. This new data, together with the loaded network parameters, constitutes the intermediate buffer. Finally, a regular transition control channel is generated, the contents of which can be represented as: {"data_pointer":"0x2C00D400","model_handle":<handle_to_lite_filter> ,"metadata":{"processing_interval_ms":40}}.

[0055] In one embodiment of the present invention, a general feature acquisition module is configured to perform the following steps: parse the data entry node of the conventional transition control channel and load the initial gating threshold of the deep visual analysis architecture; input video feature data into the feature dimensionality reduction operator inside the deep visual analysis architecture; map pixel spatial information into a feature vector sequence of the target dimension through the feature dimensionality reduction operator; temporarily store the feature vector sequence in a preset circular buffer to construct a feature extraction cache; encapsulate the memory address pointer pointing to the feature extraction cache and the feature sequence dimension information to generate general visual initial features.

[0056] Upon receiving the standard transition control channel, the data structure is immediately parsed to obtain a pointer to the device memory address of the smoothed video data, along with related metadata. Subsequently, the data entry node defined by the control channel is parsed. Based on the processing configuration file, the pre-trained video Transformer model weight file is loaded into the graphics processing unit's memory, and model instantiation is completed.

[0057] During this process, the system loads the initial gating threshold of a pre-defined deep visual analysis architecture, for example, initializing the bias term of the multilayer perceptron module inside the model to zero. The pre-defined deep visual analysis architecture is a large-scale deep neural network model with powerful spatiotemporal feature learning capabilities. In this embodiment, it specifically refers to a video Swing Transformer model, which captures multi-scale dynamic information in videos through hierarchical self-attention computation.

[0058] It should be noted that the initial gating threshold is a set of parameters set during model initialization to ensure that the model is in a stable and controllable initial state during the early stages of training or the start of inference. For example, the temperature coefficient used to scale the dot product in the attention module is set to a specific constant, or the initial weights of the residual connections are set to values ​​close to zero to promote gradient flow.

[0059] The system follows standard neural network forward inference rules, taking the smooth video pixel block matrix pointed to by the conventional transition control channel as input and feeding it into the instantiated pre-built deep visual analysis architecture. The data stream first passes through the feature dimensionality reduction operator within the pre-built deep visual analysis architecture, which maps high-dimensional pixel space information to the target-dimensional feature vector space.

[0060] Specifically, the feature dimensionality reduction operator is the input layer of the pre-built deep visual analysis architecture, which corresponds to the 3D tile embedding layer in the video Swin Transformer model. It is used to segment video frames into non-overlapping 3D spatiotemporal tiles and linearly transform each tile into a one-dimensional feature vector, i.e., feature words.

[0061] The intermediate feature sequence generated after processing by this operator is temporarily stored in a specific circular buffer located in the graphics processing unit's video memory. This buffer constitutes the feature extraction buffer. The feature extraction buffer is a first-in, first-out (FIFO) video memory queue specifically used to store the feature vector sequence generated by the feature reduction operator. Its capacity is usually set to be sufficient to hold the feature data corresponding to several seconds of video, ensuring that subsequent processing modules can always obtain context information with a sufficient duration.

[0062] The memory address pointer to the feature extraction buffer and metadata containing feature sequence dimension information are encapsulated to generate and output general initial visual features. In this embodiment, the general initial visual features are data structures that mark the location and shape of visual features in memory, providing a data entry point for subsequent higher-level semantic analysis. Afterward, the system enters a normal operating state, repeatedly performing the above feature extraction and temporary storage operations on subsequent continuously incoming video data blocks, thereby continuously refreshing the content in the feature extraction buffer and providing a real-time underlying visual feature stream for subsequent dynamic analysis.

[0063] In one example of this invention, after receiving and parsing the conventional transition control channel, the pre-trained video Swin Transformer model weights are loaded into GPU memory according to the configuration file, and instantiation is completed. The system initializes the bias terms of all multilayer perceptron modules of the model to zero as the initial gating threshold setting. Subsequently, a smoothed video pixel block matrix with dimensions of 16×224×224×3 stored at address 0x2C00D400 is input into the video Swin Transformer model. The data first passes through the model's feature dimensionality reduction operator, namely the 3D tile embedding layer. This layer sets the block size to 2×4×4px and the embedding dimension to 96. Therefore, the input video blocks are transformed into feature vectors, and the output feature sequence has a dimension of 8×56×56×96. To adapt to the local attention computation mechanism of this pre-built deep vision analysis architecture, the four-dimensional feature tensor is divided into multiple non-overlapping three-dimensional spatiotemporal local windows, for example, setting the local window size to 8×7×7, thereby reshaping the original data into multiple independent local feature sequences. The sequence length within each local window is 392, and the feature dimension is 96. These local feature sequences are stored in a new video memory buffer, for example, at address 0x4F00A800, which is the feature extraction buffer. Finally, a general initial visual feature is output, the content of which can be represented as: {"pool_pointer":"0x4F00A800","feature_shape":[392,96],"window_num":64}, and this process is continuously executed on subsequent video data.

[0064] In one embodiment of the present invention, the dynamic decoding control module is configured to perform the following operations: independently extract features from each frame in the initial video pixel block matrix, retain the spatial dimension information of the feature map, and generate a spatiotemporal feature map tensor; move the spatiotemporal feature map tensor along the time dimension based on a preset sliding window mechanism, flatten it, and extract feature vectors within the window to form a data matrix; calculate the covariance matrix of the data matrix and perform eigenvalue decomposition to obtain eigenvalues ​​arranged in descending order; perform normalization processing on the eigenvalues ​​and calculate the eigenvalue distribution decay rate; generate a spatiotemporal feature pulse based on the eigenvalue distribution decay rate; monitor the amplitude of each time point in the spatiotemporal feature pulse; in response to the amplitude not reaching the system-set threshold, clear the off-diagonal elements in the attention mask matrix at the corresponding time point within the deep visual analysis architecture; in response to the amplitude being greater than or equal to the system-set threshold, unlock the effective area of ​​the attention mask matrix at the corresponding time point and within the preset time window, and perform complete self-attention calculation; After normalizing the eigenvalues, the eigenvalue distribution decay rate is calculated. When generating spatiotemporal characteristic pulses based on this decay rate, the process is specifically as follows: Each eigenvalue is extracted from the covariance matrix after eigenvalue decomposition; each eigenvalue is normalized to calculate its proportion, and before calculating the natural logarithm, a minimal constant is added to the proportion of each normalized eigenvalue; the top K dominant eigenvalues ​​whose cumulative proportions reach a preset energy threshold are selected to construct a two-dimensional data point set composed of the dominant eigenvalue index and the corresponding natural logarithm value; least squares linear regression is performed on the two-dimensional data point set to obtain the slope of the regression line; the absolute value of the slope is extracted, a smoothing scaling adjustment coefficient is introduced, and a power function value is calculated with the natural constant as the base and the negative of the product of the absolute value of the slope and the coefficient as the exponent; the power function value is confirmed as the amplitude of the spatiotemporal characteristic pulse corresponding to that time point.

[0065] First, a lightweight 2D convolutional neural network is used as the feature extractor, such as the first few layers of a pre-trained MobileNetV2, to extract the initial video pixel block matrix. Features are extracted independently from each frame. This process preserves the spatial grid dimension of the feature map, converting it to a dimension of... The initial video pixel block matrix is ​​transformed into a dimension of The spatiotemporal feature map tensor, where For frame number, This represents the spatial resolution of the feature map. For feature dimensions.

[0066] A sliding window mechanism is used, with a step size of 1 and a window size of [value missing]. In this way, it moves along the time dimension of the spatiotemporal feature map tensor. At the center of each time window... , in the window Frame feature map flattening, extraction The data matrix is ​​composed of eigenvectors, and the matrix is ​​calculated along the feature dimension. 3D characteristic tensor covariance matrix By analyzing the characteristic tensor covariance matrix Perform eigenvalue decomposition to obtain a set of eigenvalues ​​arranged in descending order.

[0067] First, the set of eigenvalues ​​is normalized, and the eigenvalue distribution decay rate, which characterizes the energy concentration, is calculated. This decay rate is then used as a quantitative indicator of the rate of change in information disorder at that moment, thus generating a sequence of length [length missing]. A one-dimensional continuous signal, this is the spatiotemporal characteristic pulse. .

[0068] It should be noted that spatiotemporal characteristic pulses It is a characteristic signal that quantifies the degree of drastic change in the amount of information in video content. When the video frame tends to be still or the change is gradual, the inter-frame features are highly correlated, and the eigenvalues ​​of the covariance matrix decay rapidly, resulting in a slope. The absolute value is very large, which in turn causes the pulse amplitude to be large. Close to 0, see Figure 3 .

[0069] When new objects appear, there is rapid movement, or scene changes in the video, the independence of inter-frame features increases, and the feature value decay becomes smoother, resulting in a steeper slope. The absolute value decreases, and the pulse amplitude decreases. It will then rise sharply, forming a peak.

[0070] Real-time monitoring of spatiotemporal characteristic pulses Each time point amplitude Combining Figure 5 As shown, if If the system's set threshold is not reached, it indicates that the video content has temporal redundancy. The system forcibly clears the off-diagonal elements of the self-attention mask matrix at the corresponding time point to zero. The attention mask matrix is ​​a core component in the Transformer architecture that controls the flow of information. It is a matrix with the same dimension as the square of the length of the input sequence. It uses element-level multiplication operations to mask or allow attention calculations between different positions. After clearing to zero, the inter-frame computation data stream is disconnected, allowing the data to be processed through the residual connection structure.

[0071] Conversely, if the system detects a certain moment pulse amplitude Exceeded the system's set threshold This forms an activation pulse peak; here, the system sets a threshold. It is a key discrimination threshold, and its setting is based on statistical analysis of large-scale labeled video datasets.

[0072] Specifically, the system plots receiver operating characteristic (ROC) curves on the spatiotemporal pulse data of the two types of video clips mentioned above, and selects the pulse amplitude corresponding to the inflection point of the highest recall rate under low false alarm conditions as a threshold, typically ranging from 0.75 to 0.9; the system then unlocks this breakthrough moment. The effective computation area of ​​the attention mask matrix within its preset time window is allowed to be restored to the standard lower triangular matrix or fully connected matrix form, triggering the preset deep visual analysis architecture to perform intensive matrix intra-matrix computational fusion in this specific time domain, that is, to perform complete self-attention computation.

[0073] Furthermore, the non-uniform hard temporal decoding mentioned in this embodiment refers to the fact that the processing of the entire video sequence does not adopt a uniform computing strategy, but rather uses a hard-switching method based on spatiotemporal feature pulses to forcibly switch computing modes in different time regions. In areas with flat information, a bypass mode with extremely low computing power is used, while in areas with dense information, a full attention mode with high computing power is used.

[0074] Spatiotemporal characteristic pulse The calculation formula is: ; In the formula, It is the characteristic tensor covariance matrix The eigenvalues ​​are obtained in descending order after eigenvalue decomposition. To eliminate the influence of dimensions, the proportion of each eigenvalue is first calculated. Normalization is performed. By selecting the cumulative percentage that reaches the preset energy threshold One dominant eigenvalue for a two-dimensional data point set The slope of the effective decaying regression line obtained after performing least squares linear regression, where To prevent the occurrence of a minimal constant in mathematical calculations that causes the natural logarithm to collapse, it is usually taken as... . Represents the absolute value of the slope. The scaling adjustment factor is used to map the exponent to a reasonable range. For the natural constant An exponential function with base 0. Set a threshold for the system to determine the peak of the pulse wave.

[0075] In one example of this invention, it is converted into a 16×(7×7)×128 dimensional spatiotemporal feature map tensor using MobileNetV2. The sliding window size is then set. When the processing window covers frames 2 through 6, the system extracts 245 feature vectors to form a data matrix, ensuring that the subsequently generated 128×128-dimensional covariance matrix is ​​full rank. During this time period, the scene is static, and the calculated covariance matrix... The eigenvalues ​​decay rapidly, for example [15.2, 2.1, 0.4, ...]. The system first normalizes them to a percentage, and then extracts the preceding values. Linear regression of the natural logarithmic values ​​of the dominant eigenvalues ​​yields the slope. Set to -35.0. This sets the smooth scaling adjustment factor. Therefore, the calculated pulse amplitude System set threshold It is 0.8. Because... The attention mask matrix corresponding to the 4th time step is updated to a diagonal matrix. For example, for a 4×4 local mask [[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]], inter-frame attention calculation is disabled. Figure 4 The spatiotemporal characteristic pulse threshold triggering diagram is shown below. Subsequently, the processing window slides to frames 8 through 12. At this point, a car suddenly enters the frame, causing a dramatic scene change. Covariance matrix. The eigenvalues ​​decay gradually, for example [12.1, 11.5, 10.8, ...]. Similarly, before extraction... Linear regression of the natural logarithmic values ​​of the dominant eigenvalues ​​yields the slope. The calculated pulse amplitude is -2.0. .because The system has detected a pulse exceeding the threshold. The event was most intense in frame 11, with the absolute value of the slope further decreasing to 1.05. The pulse amplitude was calculated. .because Upon determining that an activation pulse peak has occurred, the system unlocks the attention mask matrix into a standard lower triangular causal mask at the 11th time step and within its preset time window, such as steps 9 to 13, for example, [[1,0,0,0], [1,1,0,0], [1,1,1,0], [1,1,1,1]], triggering the model to perform a complete dense attention computation within this region.

[0076] In one embodiment of the present invention, the knowledge fusion reasoning module is configured to perform the following steps: identify basic entities based on general visual initial features, and generate an initial entity list containing category labels; input the category labels as entry nodes into an external knowledge graph array; perform random walks in the node network of the external knowledge graph array to generate a path sequence; convert the path sequence into path embedding vectors; project the path embedding vectors into a risk prior tensor matching the dimension of the visual feature space using a multilayer perceptron; retain the feature extraction channels within the deep visual analysis architecture used to generate the native visual query matrix, and generate the native visual query matrix; extract the risk prior tensor, and... The self-attention gated query matrix is ​​reshaped by performing gated addition with the original visual query matrix. A linear transformation is performed on the general visual initial features to generate a visual key matrix and a visual value matrix. Dot product multiplication attention is then performed based on the reshaped self-attention gated query matrix, visual key matrix, and visual value matrix. During the dot product multiplication attention calculation, a common sense constraint threshold is set to generate a sparse constraint mask matrix, mapping dot product terms smaller than the threshold to negative infinity. A feature blocking mechanism is introduced: if all dot product terms corresponding to a query vector are smaller than the threshold, the attention output feature vector corresponding to the query vector is forcibly set to zero, and finally, a structured video semantic event is output.

[0077] Specifically, the process of extracting the risk prior tensor, performing cross-modal feature update operations at the visual feature level, and calculating and generating structured video semantic events with common sense verification capabilities is as follows: First, a lightweight entity recognition head is attached to the general visual initial features stored in the feature extraction cache, and the basic entities in the scene and their coordinates are initially identified with a preset recall rate. For example, an initial entity list containing category labels and location bounding boxes is output.

[0078] The external knowledge graph array is guided by the category labels of each entity in the initial entity list, which serve as the entry nodes for the graph. It should be noted that the external knowledge graph array is a graph database containing large-scale common-sense knowledge, where nodes represent entities or concepts, and edges represent the relationships between them.

[0079] Upon receiving an entry node, the external knowledge graph array triggers a biased random walk mechanism within its pre-built node network. Starting from the entry node, it explores neighboring nodes with strong logical connections. The random walk mechanism is an algorithm for information exploration on a graph structure. Its walk probability depends not only on the edge weights but also on pre-defined meta-paths, prioritizing the exploration of paths with specific semantic relationships.

[0080] The path sequences generated during the walk are converted into path embedding vectors of fixed dimension. These path embedding vectors are then fed into a pre-trained multilayer perceptron for processing. The function of this multilayer perceptron is to project and quantize the logical association information at the graph level into a risk prior tensor that matches the dimension of the visual feature space. .

[0081] In this embodiment, the risk prior tensor Logical knowledge is encoded as feature variables of the same dimension as the visual model data stream. Each row vector serves as a query feature vector guided by common sense, used to match data patterns in visual features that conform to logical rules.

[0082] Subsequently, a cross-modal feature update operation is performed, preserving the linear transformation channels of the original visual self-attention query features to obtain the original visual query matrix. Simultaneously, a risk prior tensor is extracted, and through a cross-modal feature fusion mechanism, the spatial specificity of visual features is fused with the global commonsense constraints of the knowledge graph to generate a self-attention gated query matrix with commonsense constraints. Cross-modal feature fusion mechanisms, such as broadcasting the risk prior tensor based on the spatial coordinates in the initial entity list, and then performing Hadamard product or gated addition operations with the native visual query matrix.

[0083] Specifically, the original method of generating a self-attention gating query matrix is ​​usually to perform a linear projection on the input sequence itself. However, this invention directly upgrades the "motivation" of the query from a simple "data-driven" to "data and knowledge-driven" through feature update operations.

[0084] Based on this self-attention gating query matrix reshaped by external knowledge The visual key matrix from the local window corresponding to the feature extraction cache. and visual value matrix Perform dot product multiplication attention calculation. During this calculation, by setting a commonsense constraint threshold and generating a sparse constraint mask matrix, it is combined with the risk prior tensor. The insignificant terms that contradict the common-sense rules (i.e., the dot product is less than the threshold) are mapped to negative infinity, causing the attention weights of their corresponding vector channels to be affected after passing through... After calculation, the feature is greatly suppressed. If all dot product terms are below the threshold, the feature blocking mechanism is triggered to force the output vector of the query feature to be set to zero.

[0085] It should be noted that the aforementioned logical contradictions in common-sense rules are manifested in the orthogonality of these rules. That is, in the high-dimensional embedding space learned by the model, the dot product of two vectors representing common-sense error combinations approaches zero. For example, the query vector representing "finding metal sabotage tools" generated by the knowledge graph and the feature vector representing "black long-handled umbrella" in the visual space have a significant semantic gap in the embedding space, and their dot product approaches zero, i.e., they tend to be orthogonal.

[0086] The final output is a high-confidence structured video semantic event that has been filtered through a causal logic verification layer. The structured video semantic event is a JSON object containing multiple fields, for example: {"event_type":"Vandalism","confidence":0.95,"participants":[{"entity_id":3,"role":"vandal"}],"target":"ATM_machine","timestamp":"...","reasoning_trace":"KG_path:ATM->is_target_of->Vandalism"}.

[0087] The formula for the local window self-attention mechanism is: ; In the formula, It is a query matrix incorporating a risk prior tensor, and its dimension is [dimension value missing] within each local window. , The sequence length of the local window. For feature dimensions. and These are the visual key matrix and visual value matrix, respectively obtained by linear transformation of the general initial visual features within the local window, both having dimension 1. . The feature dimension of the attention head is mapped to the aforementioned visual feature extraction dimension and serves as a scaling factor to maintain gradient stability. Represents the matrix transpose operator; For a sparse constraint mask matrix, when and The dot product is less than the set common sense constraint threshold. hour, The corresponding element is set to Otherwise, it is 0; if all dot products in a row are less than 0. If this is not the case, the feature blocking mechanism will be triggered, and the line will not be executed. Instead of normalizing, it directly forces the attention output feature vector to zero; The normalized exponential function is used to transform the scaled and masked dot product into a probability weight distribution that sums to 1; the result of matrix multiplication constitutes the attention output feature sequence of each local window.

[0088] In one example of this invention, the system initially identifies entities from the feature extraction buffer: {"person", "ATM", "night", "crowbar-like_object"}. These tags are fed into an external knowledge graph array. The graph performs a random walk starting from the "ATM" node, adjusting the edge weights based on the temporal context of "night," and discovers a high-risk path: "ATM" - (damaged) → "Vandalism" ← (using a tool) - "Crowbar". The multilayer perceptron projects this path information into a one-dimensional risk prior tensor with a feature dimension of 96. Based on the previously identified spatial coordinates of "person" and "crowbar-like_object" in the video, this risk prior tensor is broadcast and fused into the feature data of the corresponding local window, generating a query matrix with a local dimension of 392×96. Within these selected windows, The row vectors of the corresponding entities are empowered into query feature vectors specifically designed to identify "destructive actions" and "metal tools." The visual query matrix originally generated by the video Swin Transformer model is then retained, and the broadcast-generated risk prior tensor is fused with its execution-gated addition to generate the reshaped query matrix. Then attention calculations are performed. Visual key matrix. The location "crowbar-like_object" actually contains fine-grained visual features of a "black long-handled umbrella". Since the query feature vector is searching for feature patterns of "metallic reflection" and "forceful prying", and the feature vectors of the umbrella's fabric and curved frame have a significant semantic gap with this query feature vector in the embedding space, the dot product between them will fluctuate around 0 and fall below the set common-sense constraint threshold. This triggers the sparse constraint mask. The corresponding position is set to negative infinity. If all values ​​are below the threshold, the feature blocking mechanism is directly triggered, and a zero vector is output. Therefore, after... After the function, the attention weight of the erroneous visual evidence "umbrella" is reset to zero, contributing almost nothing to the final result. Combining this with contextual knowledge graph logic, the system ultimately outputs a structured video semantic event: {"event_type":"Attempted_Vandalism","confidence":0.89,...}. This reasoning mechanism reduces the semantic misjudgment rate caused by relying solely on visual recognition.

[0089] In one embodiment of the present invention, the joint training method of the cross-modal model (visual features and knowledge graph features), the design of the loss function, and the acquisition method of the core hyperparameters are as follows: First, the cross-modal training data is constructed as follows: the system builds a cross-modal training set containing a large number of video clips and their corresponding structured semantic labels. For each training video sample, not only are real event categories labeled, such as "vandalism," but basic entities are also extracted and matched with path sequences in an external knowledge graph. Then, positive and negative sample pairs are constructed through manual verification or heuristic rules: video visual features that conform to common sense logic are combined with graph paths and labeled as positive samples, such as the combination of "vandalism" and "metal crowbar"; features of irrelevant segments randomly extracted from the video or features that violate common sense logic are forged and labeled as negative samples, such as the combination of "vandalism" and "black long-handled umbrella." Triple training data containing video feature tensors, path embedding vectors, and common sense compliance labels are generated.

[0090] Secondly, the end-to-end joint training mechanism and the parameter acquisition method of the multilayer perceptron are as follows: the pre-built deep visual analysis architecture and the multilayer perceptron that generates the risk prior tensor are trained in an end-to-end joint mode. In the initial training phase, the pre-built deep visual analysis architecture can load pre-trained weights from a public dataset, while the multilayer perceptron is randomly initialized. During backpropagation, the bottom-level convolutions and shallow feature extraction operators of the pre-built deep visual analysis architecture are frozen, while the deep attention mask computation layer and the weights of the multilayer perceptron are opened for gradient updates. Through joint training, the network weights of the multilayer perceptron can adaptively project and align discrete graph path features into a continuous feature space of the same dimension as the visual feature space.

[0091] Next, the loss function design and the implementation of "orthogonal common sense rules" are as follows: the system's total training loss function is composed of a weighted average of the main task classification loss and the common sense orthogonal regularization loss. The main task classification loss adopts the standard cross-entropy loss to optimize the model's accuracy in classifying and recognizing the final semantic events in the video. To transform the concept of "orthogonal common sense rules" into a gradient penalty that the neural network can learn, this embodiment introduces a common sense orthogonal regularization loss based on supervised contrastive loss before the attention layer. Specifically: in the feature embedding space, for positive sample pairs with the common sense compliance label "true", the loss function rewards their dot product distance, causing their feature distribution to converge; for negative sample pairs with the common sense compliance label "false", the loss function acts as a penalty term, forcing the dot product between the original visual query matrix and the risk prior tensor to approach zero. After batch iteration, the network weights are constrained by this gradient penalty, causing the combination of common sense paradoxes to be forcibly stretched to a mutually orthogonal state in the geometric space. Based on this training mechanism, the logic blocking effect of mapping dot product terms less than the common sense constraint threshold to negative infinity can be stably triggered during the inference phase.

[0092] Finally, the tuning logic and selection criteria for the core hyperparameters are as follows: The preset energy threshold, smoothing scaling adjustment coefficient, and other hyperparameters mentioned in this application embodiment are not arbitrarily selected special conditions, but rather system-level hyperparameters obtained through joint tuning using grid search on the validation set. The preset energy threshold determines the proportion of dominant eigenvalues ​​of the covariance matrix retained. Its tuning logic is: while ensuring that a preset proportion (e.g., 80%) or more of static redundant background information is filtered out, the information entropy of sudden actions is maximized. Through experimental statistics, this preset energy threshold is preferably between 0.85 and 0.95. The smoothing scaling adjustment coefficient is used to scale the mapping relationship between the absolute value of the linear regression slope and the natural constant exponent. If this coefficient is too large, it will cause the spatiotemporal feature pulses to be overly sensitive, triggering frequent false wake-ups; if this coefficient is too small, it will cause the pulse peaks to be sluggish, resulting in the loss of transient events. Please refer to [link to relevant documentation]. Figure 6 The system uses the ratio of impulse-triggered recall to computational cost on the validation set (i.e., the Pareto front curve) to converge the typical value of the smoothing scaling factor to between 0.05 and 0.2, thus establishing the configuration logic of selecting 0.1 as the optimal balance point, as shown in the embodiment. Combined with the aforementioned system threshold selection based on the inflection point of the subject operating characteristic curve, this constitutes a complete and reproducible hyperparameter setting scheme for the system.

[0093] Each of the modules can be implemented in whole or in part through software, hardware, or a combination thereof. It supports hardware embedded in or independent of the processor in the computer device, and also supports software stored in the memory of the computer device, so that the processor can call and execute the operations corresponding to each of the above modules.

[0094] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A video content semantic understanding system based on deep learning, characterized in that, include: The initial matrix generation module is used to acquire the original continuous frame stream of video, perform decoding and size transformation, and generate the initial video pixel block matrix; The environmental status assessment module is used to obtain the system processing cycle and integrate it with the spatial arrangement attributes of the initial video pixel block matrix to generate environmental status assessment parameters. The control channel establishment module is used to smooth the initial video pixel block matrix based on environmental state assessment parameters to obtain video feature data, and to construct a conventional transition control channel in combination with network layer parameters. The general feature acquisition module is used to parse the conventional transition control channel, input the video feature data into the deep vision analysis architecture for feature dimensionality reduction, and generate general visual initial features; The dynamic decoding control module is used to extract the spatiotemporal feature vectors of the initial video pixel block matrix to form a data matrix, calculate the attenuation rate of the eigenvalue distribution of the covariance matrix to generate spatiotemporal feature pulses, and dynamically adjust the effective area of ​​the attention mask matrix of the deep vision analysis architecture accordingly. The knowledge fusion reasoning module is used to obtain the risk prior tensor output by the knowledge graph and fuse it with the native visual query matrix to reshape the self-attention gating query matrix; Based on the reshaped query matrix and general visual initial features, dot product attention calculation is performed by combining common sense constraint thresholds to generate structured video semantic events.

2. The video content semantic understanding system based on deep learning according to claim 1, characterized in that, The initial matrix generation module is specifically used for: Decode the original continuous video frame stream, extract the video stream state parameters, and assign a timestamp to each decoded frame. Compare the pixel size of the decoded frame with the resolution in the video stream status parameters to generate a validity flag for the initial running parameters; In response to a preset number of consecutive initial running parameter validity flags being true, the corresponding consecutive multiple frames of original pixel images are treated as a processing batch. Perform spatial size transformation and pixel value normalization on each frame in the processing batch; The pixel data, after size transformation and normalization, are stacked in chronological order to construct the initial video pixel block matrix.

3. The video content semantic understanding system based on deep learning according to claim 1, characterized in that, The environmental status assessment module is specifically used for: Query the current hardware status indicators of the computing platform and allocate contiguous memory space for the initial video pixel block matrix; Extract the dimensional information of the initial video pixel block matrix, and combine it with the allocated memory space address pointer to generate the basic running boundary conditions; The basic operating boundary conditions and system processing cycle are encapsulated to generate environmental status assessment parameters.

4. The video content semantic understanding system based on deep learning according to claim 1, characterized in that, The control channel establishment module is specifically used for: Analyze environmental status assessment parameters, read the system default resource pool status, and establish intermediate layer signal transmission rules; Load the general network layer parameters according to the intermediate layer signal transmission rules; Based on the preset filtering window radius, a sliding window filtering operation is applied to the time dimension of the initial video pixel block matrix to generate video feature data; Video feature data and general network layer parameters are written into pre-allocated video memory space to construct an intermediate buffer. The memory address pointing to the intermediate buffer, the handle of the general network layer parameters, and the system processing cycle are encapsulated to generate a regular transition control channel.

5. The video content semantic understanding system based on deep learning according to claim 1, characterized in that, The general feature acquisition module is specifically used for: Parse the data entry node of the conventional transition control channel and load the initial gating threshold of the deep vision analysis architecture; Input video feature data into the feature dimensionality reduction operator within the deep visual analysis architecture; The pixel spatial information is mapped into a sequence of feature vectors of the target dimension by using feature dimensionality reduction operators; The feature vector sequence is temporarily stored in a pre-defined circular buffer to construct a feature extraction cache. Encapsulate the memory address pointer to the feature extraction cache and the feature sequence dimension information to generate general visual initial features.

6. The video content semantic understanding system based on deep learning according to claim 1, characterized in that, The dynamic decoding control module, when extracting the spatiotemporal feature vectors of the initial video pixel block matrix to form a data matrix and calculating the attenuation rate of the covariance matrix eigenvalue distribution to generate spatiotemporal feature pulses, is specifically used for: Features are extracted independently from each frame in the initial video pixel block matrix, the spatial dimension information of the feature map is preserved, and a spatiotemporal feature map tensor is generated. Based on a preset sliding window mechanism, the feature map tensor moves along the time dimension, flattens it, and extracts the feature vectors within the window to form a data matrix. Calculate the covariance matrix of the data matrix and perform eigenvalue decomposition to obtain the eigenvalues ​​in descending order; After normalizing the eigenvalues, calculate the eigenvalue distribution decay rate; Spatiotemporal characteristic pulses are generated based on the attenuation rate of eigenvalue distribution.

7. The video content semantic understanding system based on deep learning according to claim 1, characterized in that, The dynamic decoding control module, when dynamically adjusting the effective area of ​​the attention mask matrix in the deep vision analysis architecture, is specifically used for: Monitor the amplitude at each time point in the spatiotemporal characteristic pulse; In response to the amplitude not reaching the system's set threshold, the off-diagonal elements in the attention mask matrix at the corresponding time point within the deep visual analysis architecture are cleared to zero. In response to an amplitude greater than or equal to a system-set threshold, the effective area of ​​the attention mask matrix within the corresponding time point and preset time window is unlocked, and a complete self-attention calculation is performed.

8. The video content semantic understanding system based on deep learning according to claim 1, characterized in that, The knowledge fusion reasoning module is specifically used to obtain the risk prior tensor output by the knowledge graph when: Based on the general visual initial features, basic entities are identified, and an initial entity list containing category labels is generated; Input the category labels as entry nodes into the external knowledge graph array; Random walks are performed on the node network of the external knowledge graph array to generate path sequences; Convert the path sequence into a path embedding vector; The path embedding vector is projected into a risk prior tensor that matches the dimension of the visual feature space by using a multilayer perceptron.

9. The video content semantic understanding system based on deep learning according to claim 1, characterized in that, The knowledge fusion reasoning module is specifically used to generate structured video semantic events when: Retain the feature extraction channels within the deep visual analysis architecture used to generate the native visual query matrix, and generate the native visual query matrix. Extract the risk prior tensor and reshape the self-attention gated query matrix by performing a gated addition operation with the native visual query matrix; A linear transformation is performed on the general visual initial features to generate a visual key matrix and a visual value matrix. Dot product multiplication attention calculation is then performed based on the reshaped self-attention gating query matrix, visual key matrix, and visual value matrix. In the dot product multiplication attention calculation process, a common sense constraint threshold is set to generate a sparse constraint mask matrix, and dot product terms smaller than the threshold are mapped to negative infinity. A feature blocking mechanism is introduced: if all dot product terms corresponding to a query vector are smaller than the threshold, the attention output feature vector corresponding to the query vector is forced to zero, and finally the structured video semantic event is output.

10. The video content semantic understanding system based on deep learning according to claim 6, characterized in that, After normalizing the feature values, the dynamic decoding control module calculates the feature value distribution attenuation rate. When generating spatiotemporal feature pulses based on this attenuation rate, it is specifically used for: Extract the eigenvalues ​​obtained after eigenvalue decomposition of the covariance matrix; Normalize each feature value and calculate its proportion. Before calculating the natural logarithm, add a constant to the proportion of each normalized feature value. Select the top K dominant feature values ​​whose cumulative proportion reaches the preset energy threshold and construct a two-dimensional data point set composed of the dominant feature value index and the corresponding natural logarithm value. Perform least squares linear regression on a two-dimensional data point set to obtain the slope of the regression line; Extract the absolute value of the slope, introduce a smooth scaling adjustment coefficient, and calculate the power function value with the natural constant as the base and the negative of the product of the absolute value of the slope and the coefficient as the exponent. The power function value is determined as the amplitude of the spatiotemporal characteristic pulse corresponding to that time point.

Images