Video scene understanding feature extraction method and device, equipment and medium

Abstract

The invention relates to the field of artificial intelligence, the technical scheme can be applied to the fields of financial science and technology and medical health, and discloses a video scene understanding feature extraction method, device and equipment and a medium. Then, a cross-modal dynamic attention mechanism is utilized to adaptively calculate a fusion weight and generate a preliminary fusion feature; then context information of different time granularities is captured through multi-scale time sequence pyramid fusion; and finally, a space-time entity relation graph is constructed based on the features, reasoning is performed by using a graph attention network, and scene understanding features rich in semantics are output. According to the method, the problems of modal isolation, single time sequence scale and rigid fusion strategy in the prior art are effectively solved, and the accuracy and robustness of understanding the multi-modal video content in complex scenes such as financial science and technology, medical health and the like are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, fintech, and healthcare, and particularly to a method, apparatus, device, and medium for extracting features from video scene understanding. Background Technology

[0002] Video scene understanding is one of the core challenges in computer vision and artificial intelligence. Its goal is to extract semantically rich scene information from continuous video data, including but not limited to advanced tasks such as object detection, behavior recognition, event analysis, and contextual semantic reasoning. This technology has broad application needs in key areas such as fintech and healthcare. For example, in fintech, intelligent monitoring systems in bank branches need to accurately identify abnormal transaction behaviors (such as large gatherings or emotional agitation) and understand their potential risks; in telemedicine scenarios, systems need to analyze patient behavior videos (such as rehabilitation training movements) and accompanying audio (such as breathing sounds and doctor's instructions) to assist doctors in diagnosis or treatment evaluation.

[0003] With the development of deep learning technology, especially the widespread application of convolutional neural networks (CNNs) and recurrent neural networks (RNNs), this field has made significant progress. Traditional mainstream methods typically rely on powerful CNN architectures (such as ResNet and VGG) to extract spatial static features from a single modality, usually RGB video frames, and then use temporal modeling techniques (such as 3D-CNN, LSTM, or Transformer) to capture dynamic changes within a short time frame. However, these traditional methods have gradually revealed their inherent limitations when faced with the complex and ever-changing real-world application scenarios mentioned above.

[0004] First, modal isolation is a significant problem. Video data is inherently multimodal, containing heterogeneous information from multiple sources, including visual frame sequences and synchronized audio signals. Most existing technologies employ a late-stage fusion strategy, processing the temporal features of each modality independently and then simply integrating them at the decision-making level. This approach results in a lack of early, fine-grained interactions between modalities, causing valuable cross-modal semantic relationships to be lost before fusion. In intelligent customer service quality inspection within the fintech field, this can lead to the system's inability to effectively correlate a customer's agitated tone with corresponding body language (such as slamming a table), thus missing potential complaint risks or signs of fraud.

[0005] Secondly, the temporal modeling scale is singular. Existing methods are typically fixed to a specific time window for analysis, either focusing on extremely short segments to capture instantaneous actions but lacking long-term contextual support, or using simple operations such as global average pooling to process long sequences but obscuring key temporal details. In rehabilitation training assessments in healthcare, this single-scale modeling cannot simultaneously and effectively capture both the instantaneous postural deviation of "inadequate joint mobility" and the long-term fatigue trend of "insufficient training endurance," limiting the comprehensiveness and accuracy of the assessment system.

[0006] Finally, the fusion strategy is static and rigid. Most cross-modal fusion methods rely on predefined or simply learned static weights, which cannot adaptively adjust to dynamic changes in the video scene. In scenarios such as financial remote identity authentication, when the ambient lighting changes abruptly or the background noise increases, the system cannot automatically increase the weight of the other trusted modality, leading to authentication failure or decreased security, which seriously affects user experience and system reliability.

[0007] In recent years, attention mechanisms, especially cross-modal attention and Transformer architectures, have offered new solutions to these problems. For example, cross-modal attention frameworks fuse multimodal features through a query-key mechanism, but fail to fully consider the inherent temporal scale differences between visual and audio streams; recursive joint attention models, while iteratively optimizing modal weights, still lack the ability to dynamically perceive and respond to scene complexity. Furthermore, although graph attention networks have been attempted to model the interaction relationships between multimodal entities, their input node features are usually the result of prior static fusion, failing to embed temporal dynamic changes into the relationship reasoning process.

[0008] In summary, existing technologies still suffer from three core shortcomings when addressing practical applications in fintech, healthcare, and other fields: modal isolation, single-scale limitations, and static fusion. These shortcomings fail to meet the urgent need for accurate, robust, and adaptive scene understanding in complex environments. Therefore, there is an urgent need in this field for a novel feature extraction method capable of early deep fusion, adaptive multi-scale temporal modeling, and dynamic response to scene changes. Summary of the Invention

[0009] This invention provides a method, apparatus, device, and medium for feature extraction in video scene understanding, aiming to overcome the problems of modal information isolation, single temporal modeling scale, and static and rigid fusion strategies in existing video scene understanding methods.

[0010] Firstly, a video scene understanding feature extraction method is provided, including the following steps: Temporal modeling was performed on the input visual modality data and auditory modality data respectively to extract visual temporal features and auditory temporal features; Based on the visual and auditory temporal features, the modal fusion weights related to the current scene are dynamically calculated, and the visual and auditory temporal features are fused according to the modal fusion weights to generate preliminary fusion features; Multi-scale temporal analysis is performed on the preliminary fusion features to extract sub-features at different time granularities, and the sub-features at all time granularities are fused to generate temporal fusion features rich in multi-scale contextual information. Based on the temporal fusion features or the detection results obtained from the temporal fusion features, a spatiotemporal entity relationship graph is constructed, where nodes represent entities and edges represent relationships between entities. The spatiotemporal entity relationship graph is subjected to attention-based graph reasoning to update node features and capture high-order semantic associations between entities, outputting graph embedding features for scene understanding.

[0011] Secondly, a video scene understanding feature extraction device is provided, comprising: The modal temporal feature extraction module is used to perform temporal modeling on the input visual modal data and auditory modal data respectively, and extract visual temporal features and auditory temporal features; The cross-modal dynamic fusion module is used to dynamically calculate the modal fusion weights related to the current scene based on the visual temporal features and auditory temporal features, and to fuse the visual temporal features and auditory temporal features according to the modal fusion weights to generate preliminary fusion features; The multi-scale temporal fusion module is used to perform multi-scale temporal analysis on the preliminary fusion features, extract sub-features at different time granularities, and fuse the sub-features at all time granularities to generate temporal fusion features rich in multi-scale contextual information. The spatiotemporal graph construction module is used to construct a spatiotemporal entity relationship graph based on the temporal fusion features or the detection results obtained from the temporal fusion features, wherein nodes represent entities and edges represent relationships between entities; The graph reasoning and output module is used to perform attention-based graph reasoning on the spatiotemporal entity relationship graph to update node features and capture high-order semantic associations between entities, and output graph embedding features for scene understanding.

[0012] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described video scene understanding feature extraction method.

[0013] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the video scene understanding feature extraction method described above.

[0014] The aforementioned video scene understanding feature extraction methods, devices, equipment, and media achieve adaptive fusion of visual and auditory information through dynamic calculation of modal fusion weights. This overcomes the insensitivity of traditional static weighting strategies to scene changes and significantly enhances feature representation capabilities in complex environments. The use of multi-scale temporal analysis enables parallel capture of short-term emergencies and long-term behavioral patterns, solving the problem that single-time-scale modeling cannot simultaneously address both instantaneity and continuity. By constructing a spatiotemporal entity relationship graph and performing semantic reasoning, low-level features are elevated to high-level semantic representations rich in contextual relationships. This provides a more accurate and interpretable feature foundation for the final scene understanding task, effectively reducing false alarm rates and improving system performance. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the video scene understanding feature extraction method in one embodiment of the present invention; Figure 2 This is a schematic diagram of a process for modal temporal feature extraction in one embodiment of the present invention; Figure 3 This is a schematic diagram of a process for cross-modal dynamic fusion in one embodiment of the present invention; Figure 4 This is a schematic diagram of a multi-scale temporal fusion process in one embodiment of the present invention; Figure 5 This is a schematic diagram of the reasoning and output process according to an embodiment of the present invention; Figure 6 This is a schematic diagram of a video scene understanding feature extraction device in one embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 As shown, Figure 1 A flowchart illustrating the video scene understanding feature extraction method provided in this embodiment of the invention includes the following steps: S10. Perform temporal modeling on the input visual modal data and auditory modal data respectively, and extract visual temporal features and auditory temporal features; Furthermore, such as Figure 2 As shown, in a specific embodiment, step S10 specifically includes: S11. The video frame sequence is segmented into image blocks and linearly projected to obtain an embedding vector. The embedding vector is processed by a transformer model based on a self-attention mechanism to capture the temporal dependencies between frames and output the visual temporal features. S12. After the audio spectrogram is input into a convolutional neural network for encoding, a transformer model based on a self-attention mechanism is used to process the encoding result in order to capture the temporal dependencies between audio segments and output the auditory temporal features.

[0019] In this embodiment, the visual branch uses an input rate of 16 frames per second, and the visual transformer is equipped with 4 encoder layers, each with 8 heads, to process RGB frames with a resolution of 224×224.

[0020] Auditory branch: The audio sampling rate is 16kHz, the spectrogram is generated by a 128-dimensional Mel filter bank, and the auditory transducer uses a 3-layer encoder with 4 heads per layer.

[0021] For the visual branch, the input is a sequence of video frames. ,in Indicates the time step. For spatial resolution, This refers to the number of channels. Each frame is first divided into... A number of non-overlapping patches of equal size are obtained. ,in For the visual embedding dimension, the self-attention calculation of the visual transformer is as follows: in , , , For the first Layer input embedding, The projection matrix is ​​a learnable matrix, randomly initialized, and its parameters are updated and learned through backpropagation of model gradients. There are a total of 8 outputs from the 8 heads, each with an Attention mechanism. Concatenate the output vectors of these 8 heads. The final visual branch output features are: Auditory branch processing audio spectrogram ,in This is the audio time step (which may be out of sync with the visual time step). For the frequency dimension. The auditory transducer employs a similar attention mechanism, but additionally includes a convolutional spectral encoder to handle variable temporal resolution. The characteristics of the auditory output are: In this embodiment, a modality-specific temporal transformer is used to maximize the extraction of long-range dependency features from visual and auditory sequences, avoiding information confusion that may be caused by early fusion and providing a clean and powerful single-modal feature representation for subsequent steps.

[0022] S20. Based on the visual temporal features and auditory temporal features, dynamically calculate the modal fusion weights related to the current scene, and fuse the visual temporal features and auditory temporal features according to the modal fusion weights to generate preliminary fusion features; Furthermore, such as Figure 3 As shown, in a specific embodiment, step S20 specifically includes: S21. The visual temporal features and the auditory temporal features are concatenated to form a concatenated feature vector; S22. Input the spliced ​​feature vector into a multilayer perceptron model for calculation; S23. Perform nonlinear activation mapping on the output of the multilayer perceptron model to generate dynamic weight coefficients representing the relative importance of visual and auditory modalities; S24. Using the dynamic weighting coefficients, the visual temporal features and auditory temporal features after linear transformation are weighted and summed to generate the preliminary fusion features.

[0023] In this embodiment, features from two branches are received. and Adaptive weights are generated through a meta-controller, which is designed as a 3-layer MLP. The input dimension is a concatenation of visual and auditory features (256 dimensions), and the output is dynamic weight coefficients. in, The sigmoid function is used, and MLP stands for Multilayer Perceptron. Indicates the relative importance of visual features. It involves concatenating the feature vectors from two branches. Linear is a linear layer, and the fused feature calculation is as follows: In this embodiment, the dynamic weight calculation mechanism enables the model to adaptively adjust the importance of modalities based on the scene content. For example, in a noisy surveillance scene, the weight of the visual modality is automatically increased; while in cases of blurred or occluded images, the weight of the auditory modality is enhanced, thereby ensuring the robustness of the fused features. This adaptive capability effectively overcomes the rigidity problem of fixed-weight fusion strategies, providing a more reliable feature foundation for subsequent processing.

[0024] S30. Perform multi-scale temporal analysis on the preliminary fusion features, extract sub-features at different time granularities, and fuse the sub-features at all time granularities to generate temporal fusion features rich in multi-scale contextual information. Furthermore, such as Figure 4 As shown, in a specific embodiment, step S30 specifically includes: S31. The preliminary fusion features are processed by multiple one-dimensional convolutional layers with different temporal downsampling factors to obtain a set of downscaling features corresponding to different temporal resolutions. S32. Perform upsampling operations on the set of downscaled features to restore their temporal length to be consistent with the preliminary fused features; S33. The features after upsampling are concatenated to generate the temporal fusion feature rich in multi-scale contextual information.

[0025] In this embodiment, features at different temporal granularities are captured through hierarchical processing. The multi-scale pyramid contains three levels, corresponding to the original resolution, 1 / 2 downsampling, and 1 / 4 downsampling, respectively. Specific implementation includes... There are several layers, and each layer is downsampled using stride convolution: in, For the first Layer features. Features from all layers are upsampled to their original temporal resolution and then concatenated. In this embodiment, the multi-scale temporal analysis, by processing features at different sampling rates in parallel, can simultaneously capture instantaneous events and long-term trends. Specifically, high temporal resolution (original scale) features are better at capturing rapid dynamics such as gestures and lip movements; while medium-to-low temporal resolution features (1 / 2 downsampling, 1 / 4 downsampling) are better at modeling long-term behavioral patterns such as people's movement trajectories and dialogue segments. By fusing them, the model possesses both the ability to perceive details and grasp the overall picture, significantly improving the completeness of temporal understanding.

[0026] Furthermore, in one specific embodiment, after concatenating the features after the upsampling operation to generate the temporal fusion feature rich in multi-scale contextual information, the process further includes: The features generated after splicing are added to the preliminary fused features by residual addition to enhance feature representation and avoid information degradation, and the enhanced temporal fused features are output.

[0027] In this embodiment, the timing features are finally obtained through residual connections: In this embodiment, the residual addition operation adds the multi-scale fused features to the original preliminary fused features. This not only preserves the most original information and prevents the loss of key details during downsampling, but also makes gradients easier to backpropagate, accelerates the convergence of the model training process, and further enhances the expressive power of the final features.

[0028] S40. Based on the temporal fusion features or the detection results obtained from the temporal fusion features, construct a spatiotemporal entity relationship graph, where nodes represent entities and edges represent relationships between entities; Furthermore, in one specific embodiment, the construction of the spatiotemporal entity relationship graph, wherein the entity is a target object detected from the video, and the relationship type between entities includes at least one of spatial proximity relationship, temporal continuity relationship, and semantic similarity relationship.

[0029] In this embodiment, the graph attention network node feature dimension is 128, and the relationship types include spatial proximity, temporal continuity, and semantic similarity. This module models spatiotemporal entities as a graph structure. , where nodes Represents the detected entities, edges It represents the relationship between entities.

[0030] In this embodiment, the detection result obtained from the temporal fusion features refers to the structured information obtained after inputting the temporal fusion features into a target detection network (such as Faster R-CNN, YOLO, etc.), including the target's category, spatial location, and occurrence time. The entities refer to the specific targets detected (such as people, vehicles, and animals), and the edges represent their spatial proximity, temporal continuity, or semantic similarity (e.g., "talking" is a semantic relationship). This graph-structured representation provides a foundation for advanced semantic reasoning.

[0031] S50. Perform attention-based graph reasoning on the spatiotemporal entity relationship graph to update node features and capture high-order semantic associations between entities, and output graph embedding features for scene understanding.

[0032] Furthermore, such as Figure 5 As shown, in one specific embodiment, step S50 specifically includes: S51. For each target node in the spatiotemporal entity relationship graph, calculate the attention coefficient between it and each neighbor node. The attention coefficient is calculated by linearly transforming the features of the target node and its neighbor nodes. S52. Based on the attention coefficient, the features of all neighboring nodes are weighted and summed, and then transformed by a nonlinear activation function to update the feature representation of the target node, and output the graph embedding features composed of the updated node features.

[0033] In this embodiment, the graph attention update rule is: i and j are the i-th and j-th nodes, respectively, and n is the feature vector of the node. For learnable weights, Let be the activation function, where the attention coefficient is calculated as follows: For activation function, For the transpose of a learnable attention matrix, The Euclidean distance between the dimension values ​​of two node vectors was calculated, and the final output feature was flattened. Image embedding yields: As a representational feature of the entire video, it is input into downstream tasks of subsequent object detection modules or behavior detection modules for classification.

[0034] In this embodiment, the graph attention inference mechanism allows the model to dynamically focus on the neighboring nodes most relevant to the current node. For example, when inferring that someone is opening a door, the attention coefficient between the hand node and the doorknob node will be much higher than the coefficient between the hand and the distant window node. This selective information aggregation enables the model to more accurately understand complex scene interactions, and the output graph embedding features are rich in semantic information, which can be directly used for downstream tasks such as classification, retrieval, or generation of natural language descriptions.

[0035] In one specific embodiment, the present invention can be applied to financial technology scenarios, specifically for risk behavior identification and quality inspection during bank remote video teller machines (VTMs) or online credit interviews.

[0036] In this embodiment, the system input is a video stream and synchronized audio captured by a VTM device or the user's mobile phone front-facing camera. The visual branch extracts temporal features such as facial expression changes (e.g., frequent blinking, wandering gaze) and subtle body movements (e.g., nervously rubbing hands, frequently checking the phone) from the video frame sequence using a modality-specific temporal transformer. The auditory branch processes the audio synchronously, extracting the temporal patterns of speech content (identifying keywords such as "high returns" and "guaranteed profits" by combining with an ASR model) and its paralinguistic features (e.g., sudden increase in speech rate, long pauses, or repetitive wording). The cross-modal dynamic attention module is crucial in this scenario: when the system detects that the user mentions sensitive words (e.g., "bypassing the review"), it automatically increases the attention weight to micro-expression changes in the visual modality (α_visual increases); conversely, when the image quality is poor (e.g., dim lighting), it relies more on the tense emotional features in the voice. The multi-scale temporal pyramid fusion module operates simultaneously: capturing a fleeting, stiff smile or slip of the tongue on short timescales (e.g., seconds); and analyzing the gradual transition of the user's overall posture from relaxed to tense on long timescales (e.g., the entire conversation). The spatiotemporal graph construction module uses identified entities (e.g., "User A," "Phone B," "Promotional Material C") and audio events (e.g., "Sensitive Word D") as nodes, constructing a relationship graph based on their co-occurrence and interaction relationships (e.g., "User A holding Phone B," "User A pointing to Promotional Material C when mentioning Sensitive Word D"). Finally, the graph attention reasoning module generates high-level semantic features by analyzing the relationships between nodes, enabling the system to output structured warnings such as "The system has detected a user promoting unofficial investment products while coordinating with another person's communication device, accompanied by tension, indicating a potential risk of fraudulent guidance," rather than isolated anomaly alerts. This significantly improves the accuracy and automation of risk identification at the front end of financial transactions, assisting human quality inspectors in quickly locating high-risk conversations.

[0037] In one specific embodiment, the present invention can be applied to a medical and health scenario, specifically as a video-based fall detection and remote rehabilitation training movement standardization assessment for elderly people living alone.

[0038] In this embodiment, the system is deployed in smart cameras in home environments or rehabilitation centers. Taking fall detection as an example: the visual branch analyzes video sequences, extracts the temporal trajectory of key points in human posture, and captures movement patterns that may indicate instability or sinking. The auditory branch simultaneously analyzes ambient sounds, identifying sounds such as heavy falls, impacts, or painful groans. The cross-modal dynamic attention module plays a crucial role: under normal circumstances, the system may primarily monitor behavior using the visual modality; however, when the view is briefly obstructed (e.g., a person enters the bathroom) or there is insufficient light, once an abnormal sound matching the characteristics of a fall (e.g., a sudden thud) is detected, the module immediately and dynamically increases the weight of the auditory modality, triggering a high-level alarm and effectively reducing missed detections. The multi-scale temporal pyramid can distinguish between different patterns of "sudden slip" (short-term feature) and "slowly falling while leaning against a wall due to dizziness" (long-term feature), providing more accurate contextual information for subsequent medical interventions. The spatiotemporal graph construction module uses major human joints (such as the head, shoulders, knees, and ankles) as graph nodes and skeletal connections and spatiotemporal motion relationships between joints as edges to construct a human posture graph. The graph attention reasoning module analyzes abnormal changes in the features of nodes in this graph (such as the ankle and knee joint nodes rapidly approaching each other in the vertical direction, and the head node height dropping sharply), and combines this with sound evidence to finally output a judgment result of "high confidence fall event, accompanied by impact sound," which can automatically trigger an alarm to notify medical staff or family members.

[0039] In rehabilitation training assessment scenarios, the visual branch accurately captures the movement trajectory, angle, and speed of each joint point when the patient completes a training movement (such as shoulder abduction). The auditory branch can collect the patient's inhalation sounds caused by pain or the therapist's verbal guidance. Dynamic attention ensures that when the therapist provides verbal guidance, the system can appropriately refer to audio information to understand the intention of adjusting the movement; while when the patient practices independently, it focuses on the quality of visual movements. Multi-scale temporal analysis can assess the standardization within a single movement cycle (short-term scale) and also analyze the fatigue decay trend in multiple consecutive repetitive movements (long-term scale). The graph reasoning module analyzes human posture diagrams to determine whether the movement meets the standards (such as whether the abduction angle is sufficient), whether there are compensatory movements (such as whether bending over is used to compensate for insufficient shoulder joint mobility), and generates a quantitative assessment report (such as "In this training, 30% of the movements showed scapular compensation, and the range of motion decreased by 15% in the later stages, indicating fatigue"). This provides objective and continuous monitoring methods for remote rehabilitation, greatly improving the efficiency and accessibility of medical services.

[0040] In one specific embodiment, this invention can be applied to autonomous driving scenarios. In the autonomous driving system, the input is the surrounding environment information collected by the vehicle's cameras and microphones. The visual branch identifies the vehicle ahead (visual temporal features); the auditory branch captures the urgent horn sound from the vehicle (auditory temporal features). The dynamic attention module combines visual tracking (the vehicle ahead may be slowing down) and the auditory urgency (horn sound), giving higher weight to the auditory features. Multi-scale temporal analysis captures the instantaneous nature of the horn sound on a short time scale and tracks the deceleration trend of the vehicle ahead on a long time scale. The graph reasoning module integrates nodes such as the vehicle, the vehicle ahead, and the horn sound, as well as their spatial and temporal relationships, to infer the semantic scenario of the vehicle ahead braking urgently and issuing a warning. This result provides the autonomous driving system with richer contextual information than simple visual perception, assisting it in making safer and more human-like decisions, such as timely deceleration or lane changing.

[0041] like Figure 6 As shown, this embodiment of the invention also provides a video scene understanding feature extraction device, comprising: The modal temporal feature extraction module 10 is used to perform temporal modeling on the input visual modal data and auditory modal data respectively, and extract visual temporal features and auditory temporal features; The cross-modal dynamic fusion module 20 is used to dynamically calculate the modal fusion weights related to the current scene based on the visual temporal features and auditory temporal features, and to fuse the visual temporal features and auditory temporal features according to the modal fusion weights to generate preliminary fusion features; The multi-scale temporal fusion module 30 is used to perform multi-scale temporal analysis on the preliminary fusion features, extract sub-features at different time granularities, and fuse the sub-features at all time granularities to generate temporal fusion features rich in multi-scale contextual information. The spatiotemporal graph construction module 40 is used to construct a spatiotemporal entity relationship graph based on the temporal fusion features or the detection results obtained from the temporal fusion features, wherein nodes represent entities and edges represent relationships between entities; The graph reasoning and output module 50 is used to perform attention-based graph reasoning on the spatiotemporal entity relationship graph to update node features and capture high-order semantic associations between entities, and output graph embedding features for scene understanding.

[0042] In one specific embodiment, the modal temporal feature extraction module 10 is specifically used for: The video frame sequence is segmented into image blocks and linearly projected to obtain embedding vectors. The embedding vectors are then processed using a transformer model based on a self-attention mechanism to capture the temporal dependencies between frames and output the visual temporal features. After the audio spectrogram is input into a convolutional neural network for encoding, a transformer model based on a self-attention mechanism is used to process the encoding result in order to capture the temporal dependencies between audio segments and output the auditory temporal features.

[0043] In one specific embodiment, the cross-modal dynamic fusion module 20 is specifically used for: The visual temporal features and the auditory temporal features are concatenated to form a concatenated feature vector; The concatenated feature vector is input into a multilayer perceptron model for calculation; The output of the multilayer perceptron model is subjected to nonlinear activation mapping to generate dynamic weight coefficients representing the relative importance of the visual and auditory modalities; Using the dynamic weighting coefficients, the visual temporal features and auditory temporal features after linear transformation are weighted and summed to generate the preliminary fusion features.

[0044] In one specific embodiment, the multi-scale temporal fusion module 30 is specifically used for: The preliminary fused features are processed by multiple one-dimensional convolutional layers with different temporal downsampling factors to obtain a set of downscaling features corresponding to different temporal resolutions; The set of downscaled features are upsampled to restore their temporal length to be consistent with the initial fused features. The features after upsampling are concatenated to generate the temporal fusion feature rich in multi-scale contextual information.

[0045] In one specific embodiment, after concatenating the features after upsampling to generate the temporal fusion feature rich in multi-scale contextual information, the process further includes: The features generated after splicing are added to the preliminary fused features by residual addition to enhance feature representation and avoid information degradation, and the enhanced temporal fused features are output.

[0046] In one specific embodiment, in the spatiotemporal graph construction module 40, the spatiotemporal entity relationship graph is constructed, wherein the entity is a target object detected from the video, and the relationship type between entities includes at least one of spatial proximity relationship, temporal continuity relationship and semantic similarity relationship.

[0047] In one specific embodiment, the graph reasoning and output module 50 is specifically used for: For each target node in the spatiotemporal entity relationship graph, the attention coefficient between it and each of its neighboring nodes is calculated. The attention coefficient is calculated by linearly transforming the features of the target node and its neighboring nodes. The features of all neighboring nodes are weighted and summed based on the attention coefficients, and then transformed by a nonlinear activation function to update the feature representation of the target node, outputting the graph embedding features composed of the updated node features.

[0048] Specific limitations regarding the video scene understanding feature extraction device can be found in the limitations of the video scene understanding feature extraction method described above, and will not be repeated here. Each module in the aforementioned video scene understanding feature extraction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0049] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a video scene understanding feature extraction method on the server side.

[0050] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the client-side functions or steps of a video scene understanding feature extraction method.

[0051] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: S10. Perform temporal modeling on the input visual modal data and auditory modal data respectively, and extract visual temporal features and auditory temporal features; S20. Based on the visual temporal features and auditory temporal features, dynamically calculate the modal fusion weights related to the current scene, and fuse the visual temporal features and auditory temporal features according to the modal fusion weights to generate preliminary fusion features; S30. Perform multi-scale temporal analysis on the preliminary fusion features, extract sub-features at different time granularities, and fuse the sub-features at all time granularities to generate temporal fusion features rich in multi-scale contextual information. S40. Based on the temporal fusion features or the detection results obtained from the temporal fusion features, construct a spatiotemporal entity relationship graph, where nodes represent entities and edges represent relationships between entities; S50. Perform attention-based graph reasoning on the spatiotemporal entity relationship graph to update node features and capture high-order semantic associations between entities, and output graph embedding features for scene understanding.

[0052] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: S10. Perform temporal modeling on the input visual modal data and auditory modal data respectively, and extract visual temporal features and auditory temporal features; S20. Based on the visual temporal features and auditory temporal features, dynamically calculate the modal fusion weights related to the current scene, and fuse the visual temporal features and auditory temporal features according to the modal fusion weights to generate preliminary fusion features; S30. Perform multi-scale temporal analysis on the preliminary fusion features, extract sub-features at different time granularities, and fuse the sub-features at all time granularities to generate temporal fusion features rich in multi-scale contextual information. S40. Based on the temporal fusion features or the detection results obtained from the temporal fusion features, construct a spatiotemporal entity relationship graph, where nodes represent entities and edges represent relationships between entities; S50. Perform attention-based graph reasoning on the spatiotemporal entity relationship graph to update node features and capture high-order semantic associations between entities, and output graph embedding features for scene understanding.

[0053] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0054] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0055] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0056] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for video scene understanding feature extraction, characterized in that, Includes the following steps: Temporal modeling was performed on the input visual modality data and auditory modality data respectively to extract visual temporal features and auditory temporal features; Based on the visual and auditory temporal features, the modal fusion weights related to the current scene are dynamically calculated, and the visual and auditory temporal features are fused according to the modal fusion weights to generate preliminary fusion features; Multi-scale temporal analysis is performed on the preliminary fusion features to extract sub-features at different time granularities, and the sub-features at all time granularities are fused to generate temporal fusion features rich in multi-scale contextual information. Based on the temporal fusion features or the detection results obtained from the temporal fusion features, a spatiotemporal entity relationship graph is constructed, where nodes represent entities and edges represent relationships between entities. The spatiotemporal entity relationship graph is subjected to attention-based graph reasoning to update node features and capture high-order semantic associations between entities, outputting graph embedding features for scene understanding.

2. The method of claim 1, wherein, The steps of performing temporal modeling on the input visual modality data and auditory modality data respectively, and extracting visual temporal features and auditory temporal features, specifically include: The video frame sequence is segmented into image blocks and linearly projected to obtain embedding vectors. The embedding vectors are then processed using a transformer model based on a self-attention mechanism to capture the temporal dependencies between frames and output the visual temporal features. After the audio spectrogram is input into a convolutional neural network for encoding, a transformer model based on a self-attention mechanism is used to process the encoding result in order to capture the temporal dependencies between audio segments and output the auditory temporal features.

3. The video scene understanding feature extraction method of claim 1 or 2, characterized in that, The steps of dynamically calculating modal fusion weights related to the current scene based on the visual and auditory temporal features, and fusing the visual and auditory temporal features according to the modal fusion weights to generate preliminary fused features specifically include: The visual temporal features and the auditory temporal features are concatenated to form a concatenated feature vector; The concatenated feature vector is input into a multilayer perceptron model for calculation; The output of the multilayer perceptron model is subjected to nonlinear activation mapping to generate dynamic weight coefficients representing the relative importance of the visual and auditory modalities; Using the dynamic weighting coefficients, the visual temporal features and auditory temporal features after linear transformation are weighted and summed to generate the preliminary fusion features.

4. The method of claim 1, wherein, The steps of performing multi-scale temporal analysis on the preliminary fused features, extracting sub-features at different time granularities, and fusing all sub-features at all time granularities to generate temporal fused features rich in multi-scale contextual information specifically include: The preliminary fused features are processed by multiple one-dimensional convolutional layers with different temporal downsampling factors to obtain a set of downscaling features corresponding to different temporal resolutions; The set of downscaled features are upsampled to restore their temporal length to be consistent with the initial fused features. The features after upsampling are concatenated to generate the temporal fusion feature rich in multi-scale contextual information.

5. The method of claim 4, wherein, After concatenating the features after upsampling to generate the temporal fusion feature rich in multi-scale contextual information, the process further includes: The features generated after splicing are added to the preliminary fused features by residual addition to enhance feature representation and avoid information degradation, and the enhanced temporal fused features are output.

6. The method of claim 1, wherein, The construction of the spatiotemporal entity relationship graph, wherein the entities are target objects detected from the video, and the relationship types between entities include at least one of spatial proximity relationship, temporal continuity relationship and semantic similarity relationship.

7. The method of claim 1, wherein, The steps of performing attention-based graph reasoning on the spatiotemporal entity relationship graph to update node features and capture high-order semantic relationships between entities, and outputting graph embedding features for scene understanding, specifically include: For each target node in the spatiotemporal entity relationship graph, the attention coefficient between it and each of its neighboring nodes is calculated. The attention coefficient is calculated by linearly transforming the features of the target node and its neighboring nodes. The features of all neighboring nodes are weighted and summed based on the attention coefficients, and then transformed by a nonlinear activation function to update the feature representation of the target node, outputting the graph embedding features composed of the updated node features.

8. A video scene understanding feature extraction apparatus characterized by comprising: include: The modal temporal feature extraction module is used to perform temporal modeling on the input visual modal data and auditory modal data respectively, and extract visual temporal features and auditory temporal features; The cross-modal dynamic fusion module is used to dynamically calculate the modal fusion weights related to the current scene based on the visual temporal features and auditory temporal features, and to fuse the visual temporal features and auditory temporal features according to the modal fusion weights to generate preliminary fusion features; The multi-scale temporal fusion module is used to perform multi-scale temporal analysis on the preliminary fusion features, extract sub-features at different time granularities, and fuse the sub-features at all time granularities to generate temporal fusion features rich in multi-scale contextual information. The spatiotemporal graph construction module is used to construct a spatiotemporal entity relationship graph based on the temporal fusion features or the detection results obtained from the temporal fusion features, wherein nodes represent entities and edges represent relationships between entities; The graph reasoning and output module is used to perform attention-based graph reasoning on the spatiotemporal entity relationship graph to update node features and capture high-order semantic associations between entities, and output graph embedding features for scene understanding.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the video scene understanding feature extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. When the computer program is executed by a processor, it implements the steps of the video scene understanding feature extraction method as described in any one of claims 1 to 7.

Images