A feedback-driven scene-aware temporal convolutional attention network method and device and related products

Abstract

The application provides a feedback-driven scene-aware timing convolution attention network method and device and related products, and relates to the technical field of multi-modal processing. The method comprises: performing timing alignment on multi-modal input data to obtain a first feature sequence; extracting timing features of each modality; performing cross-modal attention enhancement based on the timing features and a first modulation matrix to obtain a second feature sequence; determining a scene type and a confidence based on the second feature sequence; adjusting the first modulation matrix to obtain a second modulation matrix according to the scene type and the confidence; performing cross-modal attention enhancement again on the second feature sequence based on the second modulation matrix to obtain a third feature sequence; and performing long-range timing dependence processing on the third feature sequence to obtain a fourth feature sequence. The application can dynamically adjust the interaction weight between different modalities according to the real-time recognized scene type, thereby improving the accuracy and adaptability of multi-modal detection.

Description

Technical Field

[0001] This application relates to the field of multimodal processing technology, and in particular to a feedback-driven scene-aware temporal convolutional attention network method, device, and related products. Background Technology

[0002] With the rapid development of the live streaming industry, multimodal real-time content detection has become an important technical means to ensure compliance and regulation. Unlike general short video review, live streaming is characterized by frequent scene switching, high real-time requirements, and fine-grained compliance standards. The host may switch from product introduction to revenue promise and then to interactive Q&A within seconds, which places high demands on the response speed and scene adaptability of the detection system.

[0003] Currently, mainstream cross-modal real-time detection solutions mainly fall into two categories. One is a general cross-modal attention mechanism based on transformers, which uses a unified query, key, and value calculation mode for all modal pairs. All scenes share the same set of attention weights, but lack the ability to dynamically adjust the focus of attention modal pairs according to the current detection scene. The other is a serial pipeline architecture for scene recognition and attention calculation, which processes multimodal feature extraction and scene recognition as two independent modules sequentially. The scene recognition results are only passed unidirectionally to downstream tasks and cannot be fed back to the feature extraction stage to guide attention focus.

[0004] Therefore, improving the accuracy and adaptability of multimodal detection is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] In view of the above problems, this application is proposed to provide a feedback-driven scene-aware temporal convolutional attention network method, apparatus, and related products that overcome or at least partially solve the above problems. The technical solution is as follows: Firstly, a feedback-driven scene-aware temporal convolutional attention network method is provided, the method comprising: Temporal alignment is performed on the multimodal input data to obtain the first feature sequence; Extract the temporal features of each modality from the first feature sequence; Based on the temporal features of each modality and the first modulation matrix, the first feature sequence is subjected to cross-modal attention enhancement processing to obtain the second feature sequence. The first modulation matrix indicates the interaction weight configuration between different modal pairs. The scene type and confidence level are determined based on the second feature sequence; The first modulation matrix is ​​adjusted based on the scene type and confidence level to obtain the second modulation matrix; Based on the second modulation matrix, the second feature sequence is subjected to cross-modal attention enhancement processing to obtain the third feature sequence; The third feature sequence is processed for long-range temporal dependence to obtain the fourth feature sequence.

[0006] In one possible implementation, the multimodal input data includes audio data, video data, bullet screen text data, and screen display data.

[0007] In one possible implementation, the first modulation matrix is ​​obtained by multiplying the first modulation coefficients element by element using a static mask; The static mask is determined based on preset rules and is used to indicate the baseline value of the attention interaction intensity between different modal pairs; The first modulation coefficient is generated based on the scene embedding vector and is used to indicate the intensity adjustment value of attention interaction between different modal pairs. The scene embedding vector indicates the semantic encoding representation corresponding to the scene type.

[0008] In one possible implementation, adjusting the first modulation matrix based on the scene type and confidence level to obtain the second modulation matrix further includes: The scene embedding vector identified at the current time step is weighted and summed with the smoothed scene embedding vector from the previous time step to obtain the smoothed scene embedding vector. The smoothed scene embedding vector is used to determine the second modulation coefficient, and the second modulation coefficient is used to determine the second modulation matrix.

[0009] In one possible implementation, the first modulation matrix is ​​adjusted based on the scene type and confidence level to obtain the second modulation matrix, including: The first modulation matrix is ​​updated based on the second modulation coefficient to obtain a candidate modulation matrix, wherein the second modulation coefficient is different from the first modulation coefficient; If the confidence level is higher than a preset threshold, the second modulation matrix is ​​determined as the candidate modulation matrix; If the confidence level is lower than or equal to a preset threshold, the candidate modulation matrix is ​​updated based on the third modulation coefficient to obtain the second modulation matrix. The third modulation coefficient is different from the first and second modulation coefficients.

[0010] In one possible implementation, after performing long-range temporal dependency processing on the third feature sequence to obtain the fourth feature sequence, the following steps are also included: Scene classification or verification is performed based on the fourth feature sequence.

[0011] Secondly, a feedback-driven scene-aware temporal convolutional attention network device is provided, the device comprising: The temporal alignment unit is used to perform temporal alignment on the multimodal input data to obtain the first feature sequence; The feature extraction unit is used to extract the temporal features of each modality from the first feature sequence; The first enhancement unit is used to perform cross-modal attention enhancement processing on the first feature sequence based on the temporal features of each modality and the first modulation matrix to obtain the second feature sequence. The first modulation matrix indicates the interaction weight configuration between different modal pairs. A scene recognition unit is used to determine the scene type and confidence level based on the second feature sequence; A feedback modulation unit is used to adjust the first modulation matrix based on the scene type and confidence level to obtain the second modulation matrix; The second enhancement unit is used to perform cross-modal attention enhancement processing on the second feature sequence based on the second modulation matrix to obtain the third feature sequence; The temporal fusion unit is used to perform long-range temporal dependency processing on the third feature sequence to obtain the fourth feature sequence.

[0012] Thirdly, an electronic device is provided, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the feedback-driven scene-aware temporal convolutional attention network method described in any of the preceding claims.

[0013] Fourthly, a storage medium is provided that stores a computer program, wherein the computer program is configured to execute the feedback-driven scene-aware temporal convolutional attention network method described in any of the preceding claims at runtime.

[0014] Fifthly, a computer program product is provided, including a computer program configured to execute the feedback-driven scene-aware temporal convolutional attention network method described in any of the preceding claims at runtime.

[0015] By utilizing the above technical solutions, the feedback-driven scene-aware temporal convolutional attention network method, apparatus, and related products provided in this application include: temporally aligning multimodal input data to obtain a first feature sequence; extracting temporal features for each modality; performing cross-modal attention enhancement based on the temporal features and a first modulation matrix to obtain a second feature sequence; determining the scene type and confidence level based on the second feature sequence; adjusting the first modulation matrix according to the scene type and confidence level to obtain a second modulation matrix; performing cross-modal attention enhancement again on the second feature sequence based on the second modulation matrix to obtain a third feature sequence; and performing long-range temporal dependency processing on the third feature sequence to obtain a fourth feature sequence. This application embodiment can dynamically adjust the interaction weights between different modalities according to the real-time identified scene type, improving the accuracy and adaptability of multimodal detection. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0017] Figure 1 A flowchart of the feedback-driven scene-aware temporal convolutional attention network method provided in an embodiment of this application is shown; Figure 2 The diagram shows the structure of the feedback-driven scene-aware temporal convolutional attention network device provided in an embodiment of this application. Figure 3 A structural diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0018] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the term "comprising" and its variations should be interpreted as open-ended terms meaning "including but not limited to."

[0020] With the continuous iteration of artificial intelligence technology, various intelligent agents are gradually being integrated into diverse real-time business scenarios such as financial live streaming, online education, and telemedicine. Intelligent agents can not only independently perform system operations such as audio and video acquisition, text analysis, and image recognition, but also cooperate to complete complex cross-modal joint detection tasks. The types of data they process are becoming increasingly diverse, and the detection scenarios are becoming increasingly dynamic. This has led to a continuous increase in the demand for intelligent agents' scene perception capabilities, cross-modal feature extraction accuracy, and real-time response performance.

[0021] Currently, there are two main types of solutions for real-time multimodal detection in the industry. The first type is a fusion solution based on a general attention mechanism. This solution uses a uniform query-key-value calculation mode for all modal pairs, which can only achieve coarse-grained overall feature fusion. It cannot dynamically adjust the interaction intensity between modalities for different detection scenarios, nor does it have the ability to adapt attention when switching scenarios. The second type is a solution that processes scene recognition and feature extraction in sequence. Although this solution is based on fused features for scene determination, it does not feed the scene recognition results back to the feature extraction stage for feature refinement, and lacks a closed-loop optimization mechanism based on scene feedback.

[0022] To address the aforementioned technical problems, embodiments of this application provide a feedback-driven scene-aware temporal convolutional attention network method, such as... Figure 1 As shown, this feedback-driven scene-aware temporal convolutional attention network method may include the following steps S101 to S107: Step S101: Perform time-series alignment on the multimodal input data to obtain the first feature sequence.

[0023] In one possible implementation, the aforementioned multimodal input data may include audio data, video data, bullet screen text data, and screen display data. Furthermore, the aforementioned multimodal input data may also include one or more of the following: sensor data, infrared image data, depth image data, and text log data. The quantity and type of the aforementioned multimodal input data can be determined according to the actual situation, and are not limited here.

[0024] In one possible implementation, time-series alignment can be performed using a fixed time window as the sliding step. For example, data from each modality can be aggregated into input units by timestamp, and the first feature sequence on a unified time axis can be output to ensure temporal consistency in subsequent processing. The window length and sliding step of the aforementioned fixed time window can be determined based on the real-time requirements and data sampling rate of the actual application scenario. For example, in a financial live streaming detection scenario, a sliding window step of 1 second can be used to aggregate audio, video, bullet screen text, and screen images into input units for multiple time steps. In an offline analysis scenario, a longer window length (such as 5 seconds or 10 seconds) and a larger sliding step can be used to reduce computational load. In a low-latency mobile scenario, a shorter window length (such as 200 milliseconds or 500 milliseconds) can be used to reduce latency.

[0025] Another possible implementation involves frame-level alignment for temporal alignment. For example, nearest neighbor matching or interpolation matching can be performed based on the timestamps of each modality in the multimodal data to ensure that each sampling point of each modality corresponds precisely on the time axis. If data is missing in one or more modalities within a certain time window, the missing data can be filled in through interpolation, zero-padding, or copying from adjacent frames to ensure that the feature sequence lengths of each modality are consistent on a unified time axis. The temporal alignment method can be determined based on the actual situation and is not limited here.

[0026] In one possible implementation, the multimodal input data can be preprocessed before temporal alignment. For example, noise reduction and frame segmentation can be performed on audio data, size normalization and frame rate unification can be performed on video data, word segmentation and special character removal can be performed on bullet screen text, and key area cropping and resolution unification can be performed on the screen image.

[0027] Step S102: Extract the temporal features of each modality from the first feature sequence.

[0028] In one possible implementation, when extracting temporal features for each modality from the first feature sequence, a causal convolution operation can be used to extract local temporal patterns from the feature sequences of each modality separately. In the aforementioned causal convolution operation, zero-padding is applied to the left side of the convolution kernel, with a zero-padding length equal to the kernel size minus one. This ensures that the output of each time step depends on the input of the current time step and the time steps preceding it, thereby preventing future information leakage and guaranteeing the causality of the temporal alignment processing.

[0029] In another possible implementation, the aforementioned causal convolution can be implemented using depthwise separable convolutional structures, standard convolution, grouped convolution, or dilated convolution. The specific convolutional structure used can be determined based on hardware computing power and latency requirements. For example, on devices with strong parallel computing capabilities such as GPUs (Graphics Processing Units), standard convolution can be used to obtain higher feature representation capabilities, while on CPUs (Central Processing Units) or edge devices, depthwise separable convolution can be used to reduce computational load. The kernel size can be determined based on the length of the local temporal pattern to be extracted. For example, a smaller kernel size (such as 3 or 5) can be used to capture short-term abrupt change patterns, while a larger kernel size (such as 7 or 9) can be used to capture longer-term rhythmic change patterns. The number of layers in the aforementioned causal convolution can be determined based on the complexity of the temporal features. A single-layer convolution can be used to reduce latency, or multiple layers of stacked convolutions can be used to expand the receptive field and enhance the ability to capture longer local patterns. The implementation method of the aforementioned causal convolution can be determined according to the actual situation, and no specific restrictions are imposed here.

[0030] Step S103: Based on the temporal features of each mode and the first modulation matrix, perform cross-modal attention enhancement processing on the first feature sequence to obtain the second feature sequence. The first modulation matrix indicates the interaction weight configuration between different mode pairs.

[0031] In one possible implementation, a modality pair can indicate any pairing between two modalities, representing a cross-modal attentional interaction relationship from one modality to another. For example, an audio modality and a text modality constitute one modality pair, representing attentional attention applied from the text modality to the audio modality; an audio modality and a video modality constitute another modality pair, representing attentional attention applied from the video modality to the audio modality.

[0032] In one possible implementation, the aforementioned cross-modal attention enhancement processing includes two parts: self-attention processing and cross-attention modulation processing. For example, self-attention processing can be understood as calculating self-attention weights for the feature sequences within each modality, capturing the key temporal positions of each modality. These self-attention weights are obtained by multiplying the query vector and key vector of each modality by a dot product, scaling, and normalization, followed by multiplication with the value vector. Cross-attention modulation processing can be understood as modulating the attention scores between different modality pairs based on a first modulation matrix. This first modulation matrix is ​​a two-dimensional matrix, with its rows and columns corresponding to each modality. Each element in the matrix indicates the interaction weight configuration between the corresponding row modality and column modality. During cross-modal attention enhancement processing, the query vector, key vector, and value vector can be obtained by linearly transforming the temporal features of each modality. The projection matrix of this linear transformation can use different parameters for different modality pairs, or it can share parameters among all modality pairs. The dimensions of the query vector and key vector of each modality can be the same to ensure the effectiveness of the dot product operation.

[0033] Step S104: Determine the scene type and confidence level based on the second feature sequence.

[0034] In one possible implementation, the second feature sequence can be input into a scene classifier for classification to obtain the scene type and its corresponding confidence level. For example, the scene classifier can be a multilayer perceptron classifier, a support vector machine classifier, a Bayesian classifier, or a neural network-based classification model.

[0035] In one possible implementation, the aforementioned scenario types can be one or more from a predefined set of categories based on business needs. For example, in a financial live streaming scenario, scenario types could include product introductions, profit promises, risk disclosures, interactive Q&A, etc. The aforementioned confidence level can be understood as a measure of the reliability of the scenario type identification result, and can be expressed as the probability value output by the classifier, a normalized score, or other forms of credibility indicators.

[0036] Step S105: Adjust the first modulation matrix based on the scene type and confidence level to obtain the second modulation matrix.

[0037] Step S106: Based on the second modulation matrix, perform cross-modal attention enhancement processing on the second feature sequence to obtain the third feature sequence.

[0038] In one possible implementation, based on the aforementioned second modulation matrix, the second feature sequence is subjected to cross-attention modulation again to obtain the third feature sequence. The aforementioned second modulation matrix is ​​a modulation matrix updated by scene type feedback, and the interaction weight configuration between modal pairs matches the current smoothed scene type. Therefore, when cross-attention modulation is performed again, feature extraction can be made more focused on the key modal pair interactions in the current scene.

[0039] In one possible implementation, when performing cross-modal attention enhancement processing on the second feature sequence based on the aforementioned second modulation matrix, the cross-attention calculation between the two modalities corresponding to elements in the second modulation matrix that have values ​​of zero or close to zero can be skipped during the attention score matrix calculation process to reduce computational load. This skipping of attention calculation can be understood as skipping the dot product operation between the query vector and key vector of the corresponding modal pair before the attention score matrix calculation, based on the positions in the second modulation matrix that have values ​​of zero or below a preset sparsity threshold. Alternatively, the score at the corresponding position can be set to zero after the attention score matrix calculation. The preset sparsity threshold can be a value close to zero, such as 0.01 or 0.001.

[0040] Step S107: Perform long-range temporal dependency processing on the third feature sequence to obtain the fourth feature sequence.

[0041] In one possible implementation, when processing the long-range temporal dependencies of the third feature sequence, a one-way gated recurrent unit is used to perform temporal fusion on the third feature sequences of each modality to capture the long-range temporal dependencies within each modality. This one-way gated recurrent unit can rely on the inputs of the current time step and previous time steps, without introducing future information, thus ensuring the causality of the temporal processing. The long-range temporal dependency processing can also be implemented using long short-term memory networks, bidirectional gated recurrent units, bidirectional long short-term memory networks, or temporal convolutional networks. For example, in offline processing scenarios that require simultaneous utilization of future information to improve feature representation capabilities, bidirectional gated recurrent units or bidirectional long short-term memory networks can be used; in scenarios requiring parallel processing to improve computational efficiency, temporal convolutional networks can be used; and in real-time scenarios that require minimizing the number of parameters and computational latency, one-way gated recurrent units can be used.

[0042] This embodiment performs temporal alignment on the multimodal input data to obtain a first feature sequence; extracts temporal features for each modality; performs cross-modal attention enhancement based on the temporal features and the first modulation matrix to obtain a second feature sequence; determines the scene type and confidence level based on the second feature sequence; adjusts the first modulation matrix according to the scene type and confidence level to obtain a second modulation matrix; performs cross-modal attention enhancement again on the second feature sequence based on the second modulation matrix to obtain a third feature sequence; and performs long-range temporal dependency processing on the third feature sequence to obtain a fourth feature sequence. This embodiment can dynamically adjust the interaction weights between different modalities according to the real-time identified scene type, improving the accuracy and adaptability of multimodal detection.

[0043] This application provides a possible implementation method in which the first modulation matrix in step S103 above is obtained by multiplying the first modulation coefficient element by element using a static mask; The static mask is determined based on preset rules and is used to indicate the baseline value of the attention interaction intensity between different modal pairs; The first modulation coefficient is generated based on the scene embedding vector and is used to indicate the intensity adjustment value of attention interaction between different modal pairs. The scene embedding vector indicates the semantic encoding representation corresponding to the scene type.

[0044] In one possible implementation, the element-wise multiplication described above can be understood as multiplying the static mask and the first modulation coefficients at each corresponding position in the matrix. The static mask is a two-dimensional matrix, with its rows and columns corresponding to each mode, and each element indicating a baseline value for the attention interaction strength between the corresponding row and column modes. The first modulation coefficients are two-dimensional matrices with the same dimension as the static mask, and each element indicating an adjustment value for the attention interaction strength between corresponding mode pairs. Element-wise multiplication involves multiplying the elements at the same positions in both matrices to obtain a first modulation matrix with the same dimension as the static mask. This element-wise multiplication can be performed using standard multiplication or weighted multiplication.

[0045] In one possible implementation, the static mask is determined based on preset rules and is used to indicate the baseline value of the attention interaction intensity between different modal pairs. The static mask can take continuous values, such as any value between 0 and 1; or discrete values, such as a binary value {0, 1} representing whether the corresponding modal pair is allowed to interact, or multi-level discrete values ​​such as 0, 0.3, 0.7, and 1 representing different levels of preset interaction intensity. The preset rules can originate from the experience and knowledge of domain experts, such as financial compliance experts pre-configuring modal pair priorities based on the importance of different modalities in various scenarios; they can also originate from the statistical analysis results of historical data, such as determining the baseline value by statistically analyzing the correlation coefficients of different modal pairs in historical samples; or they can originate from mandatory requirements in regulatory laws or business specifications, such as the requirement to focus on the interaction of specific modal pairs in certain scenarios. The specific source and content of the preset rules can be determined according to the actual application scenario, and are not limited here. The static mask remains unchanged during model training and inference, enabling it to continuously and stably affect the attention calculation process. When a certain modality is completely irrelevant in a specific scenario, the elements at the corresponding positions of that modality and other modalities in the static mask can be set to zero, thereby blocking the attention interaction between the modal pairs and reducing the introduction of irrelevant noise.

[0046] In one possible implementation, the first modulation coefficient is generated based on a scene embedding vector, used to indicate the adjustment value of the attention interaction intensity between different modal pairs. The scene embedding vector indicates the semantic encoding representation corresponding to the scene type. The first modulation coefficient can be obtained by inputting the scene embedding vector into a modulation coefficient generation network and then mapping it through an activation function. The modulation coefficient generation network can be a multilayer perceptron, taking the scene embedding vector as input and outputting a modulation coefficient matrix with the same dimension as the static mask after nonlinear transformation through one or more fully connected layers; the modulation coefficient generation network can also be implemented using linear layers (i.e., a single fully connected layer) to reduce computational cost; or it can be implemented using an attention mechanism network, so that the generation process of the modulation coefficient itself also has the ability to model the relationship between modalities.

[0047] In this embodiment, a first modulation matrix is ​​generated by multiplying a static mask element-wise with a first modulation coefficient. The two work together to modulate the cross-modal attention score matrix element-wise, so that in each scenario, the interaction intensity between different modal pairs can be dynamically enhanced or suppressed according to preset rules and data adaptive signals. This integrates the advantages of rule-driven and data-driven approaches into the attention calculation mechanism, and achieves fine-grained configuration of modal interaction weights with scene type as the granularity.

[0048] This application embodiment provides a possible implementation method. Before adjusting the first modulation matrix based on scene type and confidence level to obtain the second modulation matrix in step S105 above, the following steps may be included: The scene embedding vector identified at the current time step is weighted and summed with the smoothed scene embedding vector from the previous time step to obtain the smoothed scene embedding vector. The smoothed scene embedding vector is used to determine the second modulation coefficient, and the second modulation coefficient is used to determine the second modulation matrix.

[0049] In one possible implementation, the weighted summation can be calculated as follows: multiply a preset smoothing coefficient by the original recognition vector at the current moment, and then add or subtract the preset smoothing coefficient multiplied by the smoothed vector from the previous moment to obtain the smoothed scene embedding vector. The preset smoothing coefficient is a hyperparameter ranging from 0 to 1. When the coefficient is close to 1, the smoothing result at the current moment depends more on the original recognition result at the current moment, resulting in a fast response but weak smoothing effect. When the preset smoothing coefficient is close to 0, the smoothing result at the current moment depends more on the smoothing results from previous moments, resulting in a good smoothing effect but a larger response delay. The preset smoothing coefficient can be determined based on the scene switching speed and noise level. For example, in scenes with frequent scene switching, a smaller smoothing coefficient can be used to quickly respond to changes, while in relatively stable scenes, a larger smoothing coefficient can be used to effectively suppress jitter.

[0050] In one possible implementation, the smoothed scene embedding vector is used to determine the second modulation coefficients, which in turn are used to determine the second modulation matrix. Specifically, the smoothed scene embedding vector is input into a modulation coefficient generation network, mapped by an activation function, and the output is the second modulation coefficients. Then, a static mask is multiplied element-wise with the second modulation coefficients to obtain the second modulation matrix. Because the smoothed scene embedding vector incorporates scene information from historical moments, the second modulation matrix generated based on the smoothed scene embedding vector has continuity in the time dimension, avoiding abrupt changes in the modulation matrix between adjacent moments.

[0051] In one possible implementation, if the current time is the starting time or the smoothed scene embedding vector from the previous time does not exist, the smoothed scene embedding vector from the previous time can be set to zero or the original scene embedding vector at the current time.

[0052] This embodiment uses a weighted summation of the scene embedding vector obtained at the current moment and the smoothed scene embedding vector from the previous moment. This allows for the temporal smoothing and suppression of potential misjudgments or fluctuations in the single-point recognition results, resulting in a more stable and continuous scene embedding vector. The smoothed scene embedding vector is used to determine the second modulation coefficient, which in turn determines the second modulation matrix. This ensures that the update of the modulation matrix is ​​not based on the recognition results at isolated moments, but rather integrates scene information from historical moments. This avoids abrupt changes in the modulation matrix caused by noise in single-frame recognition or jitter during rapid scene switching, making the adjustment process of intermodal interaction weights smoother and more stable, and improving the robustness of cross-modal attention modulation in complex dynamic scenes.

[0053] This application embodiment provides a possible implementation method. The above step S105 adjusts the first modulation matrix based on the scene type and confidence level to obtain the second modulation matrix, which may specifically include the following steps: The first modulation matrix is ​​updated based on the second modulation coefficient to obtain a candidate modulation matrix, wherein the second modulation coefficient is different from the first modulation coefficient; If the confidence level is higher than a preset threshold, the second modulation matrix is ​​determined as a candidate modulation matrix; If the confidence level is lower than or equal to a preset threshold, the candidate modulation matrix is ​​updated based on the third modulation coefficient to obtain the second modulation matrix. The third modulation coefficient is different from the first and second modulation coefficients.

[0054] In one possible implementation, updating the first modulation matrix based on the second modulation coefficient can be understood as multiplying the static mask element-wise with the second modulation coefficient to obtain a candidate modulation matrix. This candidate modulation matrix is ​​an intermediate result obtained by updating the modulation coefficients based on the scene type identified at the current time, building upon the first modulation matrix. The first modulation matrix before the update is generated based on the scene type from the previous time step or an initial default setting, while the candidate modulation matrix after the update is generated based on the scene type identified at the current time step. The modality pair interaction weight distributions of the two differ depending on the scene type.

[0055] In one possible implementation, the aforementioned preset threshold is used to measure the reliability of the scene recognition result, and the comparison result between the aforementioned confidence level and the aforementioned preset threshold is used to control the number of feedback iterations. When the confidence level is higher than the preset threshold, it indicates that the recognition result of the current scene type is relatively reliable, and the candidate modulation matrix is ​​determined as the second modulation matrix without the need for additional feedback iterations. The value of the aforementioned preset threshold can be determined according to the business requirements for recognition reliability. For example, a higher threshold can be set in high-risk scenarios such as financial compliance detection, and a lower threshold can be set in general scenarios. Specific limitations are not specified here.

[0056] In one possible implementation, when the confidence level is lower than or equal to the aforementioned preset threshold, it indicates that the recognition result for the current scene type is not reliable enough, and a single feedback iteration may not be able to fully adapt the modulation matrix to the current scene, thus triggering an additional feedback iteration. The aforementioned third modulation coefficient is a further updated modulation coefficient relative to the second modulation coefficient, and the generation method of the third modulation coefficient is the same as that of the second modulation coefficient. Since the scene embedding vector may have been smoothed or weighted by confidence level in the second feedback iteration, the specific value of the third modulation coefficient is different from that of the second modulation coefficient. The total number of times the modulation matrix is ​​updated is not limited to one or two times; the number of iterations can also be dynamically determined based on the difference between the confidence level and the preset threshold. For example, the lower the confidence level, the more iterations are needed, but an upper limit can be set for the total number of updates to avoid infinite loops leading to uncontrollable delays.

[0057] In this embodiment, the number of feedback iterations can be adaptively controlled at the confidence level to achieve real-time scene perception updates of the attention modulation matrix. When the scene type changes, the modulation coefficients are regenerated and the modulation matrix is ​​updated to match the currently identified scene type, avoiding the problem of cross-modal attention focus deviation caused by the mismatch between the modulation matrix and the scene type. In addition, the number of updates is dynamically determined by comparing the confidence level with a preset threshold, which allows for flexible adjustment of the feedback depth based on the degree of confidence in scene recognition. This ensures the quality of feature modulation while avoiding the increase in latency caused by invalid iterations, achieving an adaptive balance between feedback modulation efficiency and accuracy.

[0058] This application embodiment provides a possible implementation method. After performing long-range temporal dependency processing on the third feature sequence in step S107 above to obtain the fourth feature sequence, it may further include the following steps: Scene classification or verification is performed based on the fourth feature sequence.

[0059] In one possible implementation, the aforementioned fourth feature sequence is the final multimodal temporal feature output after long-range temporal dependency processing, used for downstream tasks. These downstream tasks include, but are not limited to, at least one of scenario classification and verification tasks. The specific application of the fourth feature sequence can be determined based on actual business needs. For example, in a financial live-streaming detection scenario, it can be used for compliance and violation determination; in a general scenario classification scenario, it can be used for scenario tag recognition; and in a content review scenario, it can be used for risk content identification. The specific type of the downstream task can be determined based on the actual application scenario, and is not limited here.

[0060] In one possible implementation, the scene classification described above can be understood as determining the scene category of the current multimodal input data based on the fourth feature sequence. This scene classification task can be achieved by inputting the fourth feature sequence into a downstream scene classifier for classification processing. The downstream scene classifier can be implemented using a multilayer perceptron, support vector machine, Bayesian classifier, or a neural network-based classification model. The scene classifier described above and the scene classifier used to determine the scene type in step S104 can be the same classifier or independently deployed different classifiers. For example, when the scene classifier in step S104 is used for feedback modulation, a lightweight classifier can be used to ensure real-time performance, while the scene classifier in this embodiment can use a more complex classification model to improve classification accuracy.

[0061] In one possible implementation, the above verification can be understood as determining whether the current multimodal input data conforms to preset compliance rules based on the fourth feature sequence. For example, the verification can be implemented by inputting the fourth feature sequence into a compliance detector for detection processing. The compliance detector can include at least one of a binary classifier, a multi-classifier, or a rule engine. For example, in a financial live streaming detection scenario, the verification can be used to determine violations such as promised returns, misleading advertising, and missing risk disclosures. In an online education scenario, the verification can be used to determine whether the teaching content contains prohibited vocabulary or exceeds the scope of the teaching syllabus. Furthermore, in an insurance sales scenario, the verification can be used to determine whether the insurance terms are fully disclosed and whether there is exaggerated returns.

[0062] This embodiment uses long-range temporal dependency processing to link preparatory statements, promises, and subsequent explanations scattered across different time steps into a unified temporal feature. This enables downstream detectors to identify the intent to violate regulations within the complete context, thereby significantly improving the detection rate of complex violations and reducing the false alarm rate.

[0063] The above introduces Figure 1The embodiments shown have various implementation methods for each stage. The following will further illustrate a feedback-driven scene-aware temporal convolutional attention network method of this application through specific embodiments.

[0064] This specific embodiment is applied to a scenario of detecting violations of improper return promises in securities investment advisory live broadcasts. The implementation process is described below: I. Multimodal Timing Alignment Stage 1. Receive multimodal input data Receive a 10-second live clip (timestamps t to t+10s) from a securities investment advisor's live broadcast room. The four modal input contents are as follows: (1) Audio data: The host’s voice said, “Our internal analysis shows that this stock will definitely rise by 20% in the next three months. It is absolutely safe to enter the market now.” (2) Video data: The anchor appears face-up on camera, pointing to the right side of the screen; (3) Screen data: Displays individual stock candlestick charts and "buy signal" technical indicators, but does not simultaneously display risk disclosure pop-ups; (4) Bullet screen text data: "Can I make money by buying according to the teacher's instructions?", "Request code".

[0065] 2. Execution timing alignment Using a fragment-level alignment method with a sliding window step size of 1 second, the four-modal data is aggregated into 10 time-step input units, and the multimodal feature sequence on a unified time axis is output as the first feature sequence to ensure the temporal consistency of subsequent cross-modal attention calculations.

[0066] II. Local Temporal Feature Extraction Stage Temporal features were extracted from the four modalities in the first feature sequence by using depthwise separable causal convolution with residual connections. The kernel size was set to 3, and the zero-padding length on the left side of the kernel was 2 to ensure the causality of the convolution operation and not to introduce future information.

[0067] The local temporal features extracted from each modality are as follows: (1) Audio modality: Extract speech rhythm change features (from a smooth explanation to an emphatic intonation); (2) Video modality: Extract local patterns of the anchor's hand gestures and facial orientation changes; (3) Screen modality: Extract the interface change features when the screen switches from "Overview of the Market" to "Individual Stock Recommendation"; (4) Text modality: The temporal clustering pattern of keywords such as "make money" and "request code" extracted from the bullet comments.

[0068] III. Cross-modal attention enhancement stage The initial moment corresponds to the product introduction scenario. Based on the product introduction scenario, the first modulation matrix is ​​generated, and the first cross-modal attention enhancement is performed.

[0069] 1. Generate the first modulation matrix The first modulation matrix is ​​obtained by element-wise multiplication of the static mask and the first modulation coefficient. The static mask is configured based on preset rules for financial compliance, and the first modulation coefficient is generated by initial scenario embedding vector operations and is used to fine-tune the static mask. The values ​​of the first modulation matrix of the main mode pair in the product introduction scenario are shown below: Modal pairs (video) screen: (1) Static mask value: 0.9; (2) First modulation coefficient: 0.944; (3) First modulation matrix value: 0.85; Modal pair audio text: (1) Static mask value: 0.3; (2) First modulation coefficient: 0.8; (3) First modulation matrix value: 0.24; Modal pairs audio video: (1) Static mask value: 0.5; (2) First modulation coefficient: 1.0; (3) First modulation matrix value: 0.5; 2. Perform cross-modal attention enhancement First, intramodal self-attention captures key temporal positions of each modality. Then, intermodal cross-attention completes information interaction. The cross-attention score is modulated element-wise by the first modulation matrix to obtain enhanced features, and finally, the second feature sequence is output. At this point, the cross-attention mainly focuses on verifying whether the content explained by the anchor is consistent with the product information displayed on the screen.

[0070] IV. Scene Recognition Stage Perform scene recognition based on the second feature sequence and output the recognition result: (1) Scenario type: Revenue commitment; (2) Confidence level: 0.62; (3) Preset reliability threshold: 0.75; The current confidence level of 0.62 is lower than the preset threshold of 0.75.

[0071] V. Modulation Matrix Feedback Adjustment Stage 1. Scene embedding vector smoothing processing Perform cross-time-step smoothing on the scene embedding vector, with a preset smoothing coefficient alpha = 0.7. The smoothing calculation formula is as follows: Smoothed scene embedding vector = alpha × current time-time revenue commitment scene embedding vector + (1-alpha) × previous time-time product introduction scene embedding vector; Substituting the values, we get: Smoothed scene embedding vector = 0.7 × revenue commitment scene embedding vector + 0.3 × product introduction scene embedding vector; The smoothed scene embedding vector is used to generate subsequent modulation coefficients to avoid abrupt changes in attention weights; 2. Modulation matrix iterative update The second modulation coefficients are generated based on the smoothed scene embedding vector, and the first modulation matrix is ​​updated to obtain the candidate modulation matrix. Since the current confidence level of 0.62 is lower than or equal to the preset threshold of 0.75, the candidate modulation matrix is ​​updated a second time based on the third modulation coefficients to obtain the final second modulation matrix. The iterative values ​​of the modulation matrix for the main mode pair in the revenue commitment scenario are shown below: Modal pairs audio text: (1) Static mask value: 0.95 (2) Second modulation coefficient: 0.957 (3) Second modulation matrix value: 0.92 Modal pairs audio video: (1) Static mask value: 0.1 (2) Second modulation coefficient: 0.8 (3) Second modulation matrix value: 0.08 Modal pairs video screen: (1) Static mask value: 0.3 (2) Second modulation coefficient: 0.9 (3) Second modulation matrix value: 0.27 The interaction weights of each modality pair in the second modulation matrix are adapted to the priority of the promised benefits scenario, which differs significantly from the weight distribution of the first modulation matrix.

[0072] VI. Secondary Cross-Modal Attention Enhancement Stage Based on the second modulation matrix, cross-modal attention enhancement processing is performed again on the second feature sequence. The second modulation matrix amplifies the audio. The intensity of attentional interactions between text modal pairs suppressed audio. The interaction strength of the video modal pairs is used to output a third feature sequence after further cross-modal attention enhancement. This third feature sequence focuses more on the semantic consistency features between speech and text, and significantly reduces irrelevant modal noise.

[0073] VII. Long-range time-series fusion and downstream application stage 1. Handling long-term time dependencies A unidirectional GRU (Gated Recurrent Unit) is used to model the long-range temporal dependency of the third feature sequence, capture the long-range temporal correlation of each modality feature, and output the final fourth feature sequence. 2. Downstream compliance verification Perform compliance verification based on the fourth feature sequence and output the detection result: (1) Type of violation: Promising of improper benefits; (2) Detection confidence level: 0.94; (3) Risk level: High; (4) Processing action: Trigger manual review alarm.

[0074] This specific embodiment can dynamically adjust the interaction weights between different modalities based on the real-time identified scene type, thereby improving the accuracy and adaptability of multimodal detection.

[0075] It should be noted that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In practical applications, all the above possible implementation methods can be arbitrarily combined in a combined manner to form possible embodiments of this application, which will not be described in detail here.

[0076] Based on the feedback-driven scene-aware temporal convolutional attention network method provided in the above embodiments, and based on the same inventive concept, this application also provides a feedback-driven scene-aware temporal convolutional attention network device.

[0077] Figure 2 This is a structural diagram of the feedback-driven scene-aware temporal convolutional attention network device provided in an embodiment of this application. Figure 2 As shown, the feedback-driven scene-aware temporal convolutional attention network device may specifically include a temporal alignment unit 210, a feature extraction unit 220, a first enhancement unit 230, a scene recognition unit 240, a feedback modulation unit 250, a second enhancement unit 260, and a temporal fusion unit 270.

[0078] The temporal alignment unit 210 is used to perform temporal alignment on the multimodal input data to obtain the first feature sequence; The feature extraction unit 220 is used to extract the temporal features of each modality from the first feature sequence; The first enhancement unit 230 is used to perform cross-modal attention enhancement processing on the first feature sequence based on the temporal features of each modality and the first modulation matrix to obtain the second feature sequence. The first modulation matrix indicates the interaction weight configuration between different modal pairs. Scene recognition unit 240 is used to determine scene type and confidence level based on the second feature sequence; Feedback modulation unit 250 is used to adjust the first modulation matrix based on scene type and confidence level to obtain the second modulation matrix; The second enhancement unit 260 is used to perform cross-modal attention enhancement processing on the second feature sequence based on the second modulation matrix to obtain the third feature sequence; The temporal fusion unit 270 is used to perform long-range temporal dependency processing on the third feature sequence to obtain the fourth feature sequence.

[0079] This application embodiment provides a possible implementation, wherein the feedback modulation unit 250 is further configured to: The scene embedding vector identified at the current time step is weighted and summed with the smoothed scene embedding vector from the previous time step to obtain the smoothed scene embedding vector. The smoothed scene embedding vector is used to determine the second modulation coefficient, and the second modulation coefficient is used to determine the second modulation matrix.

[0080] This application embodiment provides a possible implementation, wherein the feedback modulation unit 250 is further configured to: The first modulation matrix is ​​updated based on the second modulation coefficient to obtain a candidate modulation matrix, wherein the second modulation coefficient is different from the first modulation coefficient; If the confidence level is higher than a preset threshold, the second modulation matrix is ​​determined as a candidate modulation matrix; If the confidence level is lower than or equal to a preset threshold, the candidate modulation matrix is ​​updated based on the third modulation coefficient to obtain the second modulation matrix. The third modulation coefficient is different from the first and second modulation coefficients.

[0081] Based on the same inventive concept, embodiments of this application also provide an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute a feedback-driven scene-aware temporal convolutional attention network method of any of the above embodiments.

[0082] In an exemplary embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3The illustrated electronic device 300 includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in practical applications, the transceiver 304 is not limited to one type, and the structure of this electronic device 300 does not constitute a limitation on the embodiments of this application.

[0083] Processor 301 may be a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0084] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 302 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0085] The memory 303 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0086] The memory 303 stores computer program code that executes the scheme of this application, and its execution is controlled by the processor 301. The processor 301 executes the computer program code stored in the memory 303 to implement the content shown in the foregoing method embodiments.

[0087] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0088] Based on the same inventive concept, this application also provides a storage medium storing a computer program, wherein the computer program is configured to execute a feedback-driven scene-aware temporal convolutional attention network method of any of the above embodiments at runtime.

[0089] Based on the same inventive concept, this application also provides a computer program product, including a computer program configured to execute a feedback-driven scene-aware temporal convolutional attention network method of any of the above embodiments at runtime.

[0090] Those skilled in the art will clearly understand that the specific working process of the systems, devices, and modules described above can be referred to the corresponding process in the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0091] Those skilled in the art will understand that the technical solution of this application, or all or part of it, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several program instructions to cause an electronic device (e.g., a personal computer, server, or network device) to execute all or part of the steps of the methods described in the embodiments of this application when running the program instructions. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0092] Alternatively, all or part of the steps of the foregoing method embodiments can be implemented by hardware (such as electronic devices like personal computers, servers, or network devices) associated with program instructions. The program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by the processor of the electronic device, the electronic device executes all or part of the steps of the methods described in the embodiments of this application.

[0093] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principles of this application, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the corresponding technical solutions to leave the protection scope of this application.

Claims

1. A feedback-driven scene-aware temporal convolutional attention network method, characterized in that, The method includes: The multimodal input data is time-aligned to obtain a first feature sequence; wherein the multimodal input data includes audio data, video data, bullet screen text data, and screen image data; Extract the temporal features of each modality from the first feature sequence; Based on the temporal features of each modality and the first modulation matrix, the first feature sequence is subjected to cross-modal attention enhancement processing to obtain the second feature sequence. The first modulation matrix indicates the interaction weight configuration between different modal pairs. The first modulation matrix is ​​obtained by multiplying a static mask by a first modulation coefficient element-wise. The static mask is determined based on a preset rule and is used to indicate the baseline value of the attention interaction intensity between the different modal pairs. The first modulation coefficient is generated based on a scene embedding vector and is used to indicate the adjustment value of the attention interaction intensity between the different modal pairs. The scene embedding vector indicates the semantic encoding representation corresponding to the scene type. The scene type and confidence level are determined based on the second feature sequence; The first modulation matrix is ​​adjusted based on the scene type and the confidence level to obtain the second modulation matrix; Based on the second modulation matrix, the second feature sequence is subjected to cross-modal attention enhancement processing to obtain the third feature sequence; The third feature sequence is processed by long-range temporal dependency processing to obtain the fourth feature sequence; The step of adjusting the first modulation matrix based on the scene type and the confidence level to obtain the second modulation matrix includes: The first modulation matrix is ​​updated based on the second modulation coefficient to obtain a candidate modulation matrix, wherein the second modulation coefficient is different from the first modulation coefficient; If the confidence level is higher than a preset threshold, the second modulation matrix is ​​determined to be the candidate modulation matrix; If the confidence level is lower than or equal to a preset threshold, the candidate modulation matrix is ​​updated based on the third modulation coefficient to obtain the second modulation matrix, wherein the third modulation coefficient is different from the first modulation coefficient and the second modulation coefficient; After performing long-range temporal dependency processing on the third feature sequence to obtain the fourth feature sequence, the method further includes: performing scene classification or verification based on the fourth feature sequence.

2. The method according to claim 1, characterized in that, Before adjusting the first modulation matrix based on the scene type and the confidence level to obtain the second modulation matrix, the method further includes: The scene embedding vector identified at the current time is weighted and summed with the smoothed scene embedding vector at the previous time to obtain the smoothed scene embedding vector; The smoothed scene embedding vector is used to determine the second modulation coefficient, and the second modulation coefficient is used to determine the second modulation matrix.

3. A feedback-driven scene-aware temporal convolutional attention network device, characterized in that, The device includes: A temporal alignment unit is used to perform temporal alignment on multimodal input data to obtain a first feature sequence; wherein, the multimodal input data includes audio data, video data, bullet screen text data, and screen image data; The feature extraction unit is used to extract the temporal features of each modality from the first feature sequence; A first enhancement unit is configured to perform cross-modal attention enhancement processing on the first feature sequence based on the temporal features of each modality and a first modulation matrix to obtain a second feature sequence. The first modulation matrix indicates the interaction weight configuration between different modal pairs. The first modulation matrix is ​​obtained by element-wise multiplication of a static mask with a first modulation coefficient. The static mask is determined based on a preset rule and is used to indicate a baseline value for the attention interaction intensity between the different modal pairs. The first modulation coefficient is generated based on a scene embedding vector and is used to indicate an adjustment value for the attention interaction intensity between the different modal pairs. The scene embedding vector indicates the semantic encoding representation corresponding to the scene type. The scene recognition unit is used to determine the scene type and confidence level based on the second feature sequence; A feedback modulation unit is used to adjust the first modulation matrix based on the scene type and the confidence level to obtain a second modulation matrix; The second enhancement unit is used to perform cross-modal attention enhancement processing on the second feature sequence based on the second modulation matrix to obtain the third feature sequence; A time-series fusion unit is used to perform long-range time-series dependency processing on the third feature sequence to obtain a fourth feature sequence; The feedback modulation unit is further used for: The first modulation matrix is ​​updated based on the second modulation coefficient to obtain a candidate modulation matrix, wherein the second modulation coefficient is different from the first modulation coefficient; If the confidence level is higher than a preset threshold, the second modulation matrix is ​​determined to be the candidate modulation matrix; If the confidence level is lower than or equal to a preset threshold, the candidate modulation matrix is ​​updated based on the third modulation coefficient to obtain the second modulation matrix, wherein the third modulation coefficient is different from the first modulation coefficient and the second modulation coefficient; After the temporal fusion unit performs long-range temporal dependency processing on the third feature sequence to obtain the fourth feature sequence, it further includes: performing scene classification or verification based on the fourth feature sequence.

4. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the feedback-driven scene-aware temporal convolutional attention network method according to any one of claims 1 to 2.

5. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the feedback-driven scene-aware temporal convolutional attention network method according to any one of claims 1 to 2 at runtime.

6. A computer program product, comprising a computer program, characterized in that, The computer program is configured to execute the feedback-driven scene-aware temporal convolutional attention network method according to any one of claims 1 to 2 at runtime.

Images