Intelligent scene recognition method and device of intelligent interaction control unit

Abstract

The application relates to the technical field of intelligent scene recognition, and discloses an intelligent scene recognition method and equipment of an intelligent interaction control unit, which comprises the following steps: collecting and preprocessing the operation sequence, video content and audio data of the intelligent interaction control unit in a digital set-top box to obtain a standard multi-modal data set; inputting the standard multi-modal data set into a multi-branch deep neural network model to extract time sequence, visual and audio features through parallel processing and obtain a fusion scene feature vector; performing dimension reduction and semantic mapping on the fusion scene feature vector to obtain a compressed semantic feature vector; performing similarity calculation on the compressed semantic feature vector and a predefined scene query vector to obtain a scene matching score; and based on the scene matching score, comprehensively outputting the outputs of multiple sub-classifiers to obtain a target scene classification result; and the application can efficiently run on a resource-limited digital set-top box and realize real-time scene recognition.

Description

Technical Field

[0001] The present invention relates to the field of intelligent scene recognition technology, and in particular to an intelligent scene recognition method and device for an intelligent interactive control unit. Background Art

[0002] Traditional digital set-top box interaction methods often lack the ability to intelligently identify user usage scenarios, resulting in a less than smooth user experience. While some research has attempted to perform scenario recognition using single-modal data (such as remote control operation sequences), this approach often struggles to accurately capture complex usage scenarios.

[0003] Multimodal data fusion offers a new approach to solving this problem, but effectively processing and fusing heterogeneous data from different modalities remains a challenge. Traditional feature extraction and fusion methods often struggle to capture the complex relationships between data, resulting in low scene recognition accuracy. Furthermore, existing scene recognition models typically require large amounts of labeled data for training, which is often difficult to achieve in practical applications. Summary of the Invention

[0004] The present invention provides an intelligent scene recognition method and device for an intelligent interactive control unit. The present invention can run efficiently on a digital set-top box with limited resources and realize real-time scene recognition.

[0005] In a first aspect, the present invention provides an intelligent scene recognition method for an intelligent interactive control unit, the intelligent scene recognition method for the intelligent interactive control unit comprising:

[0006] The operation sequence, video content and audio data of the intelligent interactive control unit in the digital set-top box are collected and preprocessed to obtain a standard multimodal dataset;

[0007] Inputting the standard multimodal dataset into a multi-branch deep neural network model, extracting temporal, visual, and audio features through parallel processing, and obtaining a fused scene feature vector;

[0008] Using a variational autoencoder to perform dimensionality reduction and semantic mapping on the fusion scene feature vector to obtain a compressed semantic feature vector;

[0009] Calculating the similarity between the compressed semantic feature vector and the predefined scene query vector, and obtaining a scene matching score through an attention mechanism;

[0010] Based on the scene matching score, an ensemble learning method is used to synthesize the outputs of multiple sub-classifiers to obtain a target scene classification result.

[0011] In a second aspect, the present invention provides an intelligent scene recognition device of an intelligent interaction control unit, wherein the intelligent scene recognition device of the intelligent interaction control unit comprises:

[0012] The acquisition module is used to collect and preprocess the operation sequence, video content and audio data of the intelligent interactive control unit in the digital set-top box to obtain a standard multimodal data set;

[0013] An extraction module, configured to input the standard multimodal dataset into a multi-branch deep neural network model, extract temporal, visual, and audio features through parallel processing, and obtain a fused scene feature vector;

[0014] A mapping module, configured to perform dimensionality reduction and semantic mapping on the fusion scene feature vector using a variational autoencoder to obtain a compressed semantic feature vector;

[0015] a calculation module, configured to calculate a similarity between the compressed semantic feature vector and a predefined scene query vector, and obtain a scene matching score through an attention mechanism;

[0016] The output module is used to synthesize the outputs of multiple sub-classifiers based on the scene matching score using an ensemble learning method to obtain a target scene classification result.

[0017] The third aspect of the present invention provides a computer device comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the intelligent scene recognition method of the above-mentioned intelligent interaction control unit.

[0018] A fourth aspect of the present invention provides a computer-readable storage medium, which stores instructions that, when executed on a computer, enable the computer to execute the intelligent scene recognition method of the intelligent interaction control unit.

[0019] In the technical solution provided by the present invention, by simultaneously processing operation sequences, video content and audio data, this method can comprehensively capture the characteristics of user usage scenarios and improve the accuracy and robustness of scene recognition. The multi-branch deep neural network model is adopted, which can process data of different modalities in parallel, effectively extract temporal, visual and audio features, and improve the efficiency and quality of feature extraction. The use of variational autoencoders to reduce the dimension and semantic map the fusion features not only reduces the data dimension, but also captures potential semantic information, which helps to improve the accuracy of subsequent scene recognition. The introduction of the attention mechanism in the similarity calculation process can adaptively focus on important features and improve the accuracy of scene matching. The use of multiple sub-classifiers combined with the Dempster-Shafer evidence theory can combine the advantages of different classifiers and improve the generalization ability and robustness of the model. Through the lightweight optimization of the task flow graph, the method can run efficiently on resource-constrained digital set-top boxes and realize real-time scene recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 Schematic diagram of the steps of the intelligent scene recognition method of the intelligent interactive control unit in an embodiment of the present invention;

[0022] Figure 2 Schematic diagram of the structure of the intelligent scene recognition device of the intelligent interaction control unit in an embodiment of the present invention;

[0023] Figure 3 It is a schematic block diagram of the structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] An embodiment of the present invention provides an intelligent scene recognition method and device for an intelligent interactive control unit. The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products or devices.

[0025] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 An embodiment of the intelligent scene recognition method of the intelligent interactive control unit in the embodiment of the present invention includes:

[0026] Step S1: collecting and preprocessing the operation sequence, video content, and audio data of the intelligent interactive control unit in the digital set-top box to obtain a standard multimodal data set;

[0027] It is understandable that the execution subject of the present invention can be the intelligent scene recognition device of the intelligent interaction control unit, or it can be a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking the server as the execution subject as an example.

[0028] Specifically, the intelligent interactive control unit's remote control button operations are captured in real time to obtain raw operation sequence data. This data exhibits certain volatility and randomness due to factors such as user habits and reaction time. This raw operation sequence data is segmented using a sliding time window, dividing the continuous operation sequence into multiple time segments to obtain a standardized operation feature sequence. Simultaneously, raw video streams are collected from the video output port of the digital set-top box. The video content can reflect the specific scene currently being played, such as the type of TV program, the content being played, and the main characters or objects on the screen. These raw video streams are captured through the video output port to obtain raw visual information. Multi-scale image segmentation technology is used to process the raw video streams, refining different regions within the video frame. This allows the system to identify important objects, background, and other visual features in the video. By segmenting the video content at different scales, standardized video content features are obtained. Raw audio streams are collected from the audio output port of the digital set-top box to obtain the sound information of the currently playing content. The audio data contains a rich variety of content, such as speech, music, and ambient sound effects. A joint time-frequency domain analysis is performed on the raw audio streams. Joint time-frequency analysis is a method that processes audio signals simultaneously in the time and frequency domains, effectively extracting the temporal characteristics and frequency components of audio signals, and providing a better understanding of the structure and characteristics of the audio content. This analysis yields standardized audio content features. By aligning the standardized operational feature sequences, standardized video content features, and standardized audio content features in the temporal dimension, a standardized multimodal dataset is generated.

[0029] Step S2: Input the standard multimodal dataset into a multi-branch deep neural network model, extract temporal, visual, and audio features through parallel processing, and obtain a fused scene feature vector;

[0030] Specifically, the standard multimodal dataset was divided into three subsets: operation sequence data, video content data, and audio data. For the operation sequence data subset, a long short-term memory (LSTM) network was used to extract temporal features. Operation sequence data exhibits certain temporal dependencies, meaning that user operations exhibit regularity and contextual associations. The LSTM network structure, which can capture long-term and short-term dependencies, is well-suited for this data type. An LSTM network with three hidden layers, 128 neurons per hidden layer, and the tanh function as the activation function was used, was used. This structure effectively extracts temporal features of the operation sequence and uses this information for multimodal feature fusion. Furthermore, for the video content data subset, a residual network (ResNet) was used to extract visual features. Video content contains rich visual information, such as scenes, characters, and objects. Residual networks can effectively alleviate the vanishing gradient problem and enable deeper model training. The residual network consists of 50 convolutional layers, with ReLU activation functions used in each residual block. Batch normalization is also performed to improve training stability and convergence speed. The final layer of the network uses global average pooling to compress the features of the entire video frame into a fixed-length visual feature vector. A one-dimensional convolutional neural network is used to extract audio features for the audio data subset. The audio data contains different types of signals, such as speech, music, and ambient sound. The one-dimensional convolutional network effectively captures the temporal and spectral characteristics of the audio signals. The one-dimensional convolutional network consists of four convolutional layers, each using the ELU activation function, which produces smaller negative outputs in negative regions, accelerating network convergence. Each layer also uses max pooling to reduce feature dimensionality and suppress noise. The final layer of the network uses global max pooling to compress the entire audio sequence into a fixed-length feature vector that effectively represents the key information in the audio data. The temporal features of the operation sequence, the visual features of the video content, and the audio content features are concatenated to form a complete multimodal feature vector. An attention mechanism is used to weight the multimodal features by their importance. The attention mechanism consists of a two-layer feedforward neural network that uses a softmax function to calculate the weight of each feature, highlighting the most important features for the current scene and enhancing the model's ability to focus on key information. The attention-weighted multimodal features are then fed into a cross-modal fusion network. The cross-modal fusion network consists of three fully connected layers, each using the LeakyReLU activation function to mitigate the "dead neuron" problem and improve the model's robustness and generalization. Furthermore, to prevent overfitting, Dropout regularization is added after each fully connected layer to randomly discard some neurons and enhance the model's generalization. After processing the cross-modal fusion network, preliminary fused features are obtained. An adaptive feature aggregation algorithm is applied to these preliminary fused features.The adaptive feature aggregation algorithm processes features through dynamic threshold screening and nonlinear transformation, dynamically adjusts the weight of features in different scenarios, enhances the expression ability of key features, and finally obtains a fused scene feature vector.

[0031] Step S3: Using a variational autoencoder to perform dimensionality reduction and semantic mapping on the fused scene feature vector to obtain a compressed semantic feature vector;

[0032] Specifically, the fused scene feature vector is input into the encoder network of the variational autoencoder. The encoder network consists of three fully connected layers, each using the LeakyReLU activation function. LeakyReLU effectively addresses the "dead neuron" problem and enhances the network's feature representation capabilities. Through layer-by-layer processing in the encoder network, the original fused scene features are mapped to an intermediate feature representation. This intermediate feature representation is then processed through two parallel fully connected layers to generate a mean vector and a log-variance vector, respectively. The mean and log-variance vectors determine the distribution parameters of the latent variable. In the variational autoencoder, the latent variable is generated using a reparameterization technique: noise is sampled from a standard normal distribution. The sampled noise is then combined with the mean and variance to generate latent variable samples. This process involves linearly transforming the noise from the standard normal distribution. The mean vector controls the center position of the latent variable, while the log-variance vector controls the spread of the variable. This reparameterization technique effectively introduces randomness into the model while maintaining its differentiability. The latent variable samples are input into the decoder network of the variational autoencoder. The decoder network also consists of three fully connected layers, each of which uses the ReLU activation function. The ReLU activation function has excellent gradient propagation properties, ensuring that the network can effectively decode the latent variables. Through layer-by-layer processing in the decoder network, a reconstructed feature vector is obtained. The reconstruction error between the reconstructed feature vector and the fused scene feature vector is calculated and incorporated into the overall loss function for optimization. The reconstruction error is measured using mean squared error or binary cross entropy, reflecting the difference between the decoder-reconstructed features and the original input. The KL divergence (Kullback-Leibler divergence) of the latent variable is also calculated. This loss measures the difference between the distribution of the latent variable and the standard normal distribution. By considering both the reconstruction error and the KL divergence, the variational autoencoder maximizes the preservation of the original information of the input features while maintaining the continuity of the latent space. The overall loss function consists of a weighted sum of the reconstruction error and the KL divergence. By optimizing the overall loss, a more accurate representation of the latent variables is obtained. Based on the overall loss function, adjusted latent variable samples are generated to improve feature quality and robustness. Spherical projection is performed on the adjusted latent variable samples, mapping the points in the high-dimensional space onto the unit hypersphere. The purpose of spherical projection is to normalize the latent variables, making them more evenly distributed in the high-dimensional space, thereby avoiding skewness and imbalance in the feature space and obtaining normalized semantic features. The normalized semantic features are then transformed into feature space to optimize the feature representation, making them more compact and discriminative, resulting in a compressed semantic feature vector.

[0033] The intermediate feature representation is input into the first fully connected layer, which uses a linear activation function to maintain a linear relationship between input and output, resulting in the original mean vector. Batch normalization is then applied to the original mean vector. This process adjusts the feature distribution to a standard normal distribution, concentrating the mean vector's value range. This reduces instability during network training and accelerates convergence, resulting in a standardized mean vector. Simultaneously, the intermediate feature representation is input into the second fully connected layer, which generates the log-variance vector. To ensure that the variance is non-negative, the second fully connected layer uses the Softplus activation function, a smoothed variant of the ReLU function that converts negative input values ​​to positive values, effectively avoiding the problem of negative variance. The Softplus activation function is then applied to the original log-variance vector, converting the log-variance to variance, resulting in a variance vector that reflects the uncertainty of the feature distribution. A random noise vector with the same dimensions as the standardized mean vector is generated. To ensure the source of random noise is standardized, the random noise vector follows a standard normal distribution with mean 0 and variance 1. This introduces appropriate randomness into the generated latent variable samples and improves the model's generalization ability. The random noise vector is scaled to match the variance vector. The random noise vector is multiplied by the square root of the variance vector to align the noise vector's variance with the variance, resulting in a scaled noise vector. The scaled noise vector is then added to the standardized mean vector to obtain the initial latent variable samples. This generated latent variable sample not only contains essential information about the intermediate feature representation but also imparts a certain degree of randomness through the introduction of noise. A spherical projection function is applied to the initial latent variable samples, projecting the sample points onto the unit hypersphere. Spherical projection normalizes the latent variables, unifying their moduli to 1 so that all latent variable samples are distributed on the unit hypersphere. Normalization ensures a uniform distribution of latent variables in high-dimensional space, avoiding imbalances caused by differences in numerical magnitude. The final latent variable samples are obtained by projecting the latent variable samples onto the unit hypersphere.

[0034] Step S4: Calculate the similarity between the compressed semantic feature vector and the predefined scene query vector, and obtain a scene matching score through the attention mechanism;

[0035] Specifically, predefined scene categories are text-encoded to generate an initial set of scene query vectors. A pre-trained text encoding model, such as a natural language processing model like BERT or GPT, is used to encode the scene categories described in the text into semantically rich vector representations. Scene feature enhancement is performed on the initial set of scene query vectors to enrich the expressive power of the scene categories. For example, feature enhancement of the scene query vectors can be achieved through a self-attention mechanism or contextual information supplementation to obtain an enhanced set of scene query vectors. The compressed semantic feature vectors and the enhanced set of scene query vectors are dimensionally aligned to calculate similarity in the same dimensional space. This alignment is achieved through linear projection or a fully connected layer, ensuring that the compressed semantic feature vectors and scene query vectors have the same dimensionality. Cosine similarity is calculated based on the aligned feature vectors to obtain an initial similarity matrix. The degree of similarity between the two vectors is measured by calculating the cosine of the angle between them. In the initial similarity matrix, each element represents the similarity between the compressed semantic feature vector and a specific scene query vector. The initial similarity matrix is ​​input into a multi-head attention mechanism for processing. The multi-head attention mechanism uses multiple independent attention heads to capture different feature relationships. Each attention head weights different features, thereby learning the dependencies between features in different subspaces. This results in an attention-weighted similarity matrix. This attention-weighted similarity matrix is ​​then nonlinearly transformed to obtain a scene matching probability distribution. Using an activation function, such as the softmax function, each element of the similarity matrix is ​​converted into a probability value, ensuring that the matching probability of each scene category lies between 0 and 1, and the sum of all probabilities is 1. This allows the model to intuitively represent the matching probability of each scene category with the compressed semantic feature vector. The scene matching probability distribution reflects the degree of match between different scene categories and the current feature vector. Based on the scene matching probability distribution, the K scene matching results with the highest probabilities are selected, effectively narrowing the target range and improving matching accuracy. These K selected scene matching results are weighted and summed to obtain the final scene matching score. During this weighted summation, each scene matching result is assigned a different weight based on its corresponding matching probability, ensuring that the most relevant scene categories contribute the most to the final matching result, resulting in the scene matching score.

[0036] Step S5: Based on the scene matching score, an ensemble learning method is used to synthesize the outputs of multiple sub-classifiers to obtain a target scene classification result.

[0037] Specifically, the scene matching scores are normalized to standardize their numerical range. This ensures that scores from different sources are compared and calculated on the same scale, avoiding bias caused by score differences. The resulting normalized scene matching scores are then obtained. A feature vector is constructed based on the normalized scores. This feature vector is then input into a random forest classifier for classification. The random forest classifier is an ensemble method based on multiple decision trees. By integrating the classification results of multiple decision trees, it effectively improves the robustness and generalization of the model. The random forest classifier contains 100 decision trees, each with a maximum depth of 10, ensuring that each tree has a moderate complexity and avoids overfitting the data. Furthermore, by integrating a large number of decision trees, the model captures the complex relationships between features and obtains the first sub-classification result. The same feature vector is then input into a support vector machine classifier for classification. The support vector machine is an effective binary classification model that achieves efficient classification by finding an optimal hyperplane that maximizes the margin between different classes. The support vector machine uses a radial basis function (RBF) kernel, which maps the original features into a high-dimensional space, making nonlinear features linearly separable. The penalty parameter C is set to 1.0, which is used to balance model complexity and classification accuracy. The second sub-classification results are obtained through processing using a support vector machine classifier. The feature vector is then classified using the gradient boosted decision tree (GBDT) algorithm. The gradient boosted decision tree is an ensemble learning method based on an additive model and a feedforward distribution algorithm. It gradually builds a series of weak learners, each correcting the error from the previous round to continuously improve model performance. The GBDT contains 200 weak learners, with a learning rate of 0.1. The learning rate determines the contribution of each weak learner to the final model. A smaller learning rate allows the model to gradually approach the optimal solution, reducing the risk of overfitting. The third sub-classification results are obtained through training and prediction using the GBDT algorithm. The first, second, and third sub-classification results are encoded and merged into a unified representation to obtain the encoded sub-classification result set. The Dempster-Shafer evidence theory is applied to this encoded sub-classification result set. The Dempster-Shafer evidence theory is a mathematical theory used to deal with uncertainty and fuse multiple information sources. By calculating the confidence interval for each category, the degree of support for each category by different sub-classifiers is quantified to obtain a fused evidence matrix. Based on the fused evidence matrix, the optimal category is selected to obtain a preliminary scene category. In the evidence matrix, the confidence interval of each category represents the probability that the category is considered correct. Selecting the optimal category means selecting the category with the highest confidence as the preliminary classification result. Markov smoothing is performed on the preliminary scene category. Markov smoothing is a smoothing method based on time series. By considering the states of the previous and next moments, the current state is adjusted to ensure the continuity and rationality of the scene classification results.Through Markov smoothing, the classification instability caused by instantaneous errors is eliminated, so that the final target scene classification results are more consistent with the logic of actual scene changes.

[0038] In an embodiment of the present invention, by simultaneously processing operation sequences, video content and audio data, the method can comprehensively capture the characteristics of user usage scenarios and improve the accuracy and robustness of scene recognition. The multi-branch deep neural network model is adopted, which can process data of different modalities in parallel, effectively extract temporal, visual and audio features, and improve the efficiency and quality of feature extraction. The use of variational autoencoders to reduce the dimension and semantically map the fused features not only reduces the data dimension, but also captures potential semantic information, which helps to improve the accuracy of subsequent scene recognition. Introducing the attention mechanism in the similarity calculation process can adaptively focus on important features and improve the accuracy of scene matching. The use of multiple sub-classifiers combined with the Dempster-Shafer evidence theory can combine the advantages of different classifiers and improve the generalization ability and robustness of the model. Through the lightweight optimization of the task flow graph, the method can run efficiently on resource-constrained digital set-top boxes and realize real-time scene recognition.

[0039] In a specific embodiment, the process of executing step S1 may specifically include the following steps:

[0040] The remote control button operations of the intelligent interactive control unit are captured in real time to obtain the original operation sequence data, and the original operation sequence data is segmented into sliding time windows to obtain a standardized operation feature sequence;

[0041] Collect the original video stream from the video output port of the digital set-top box, perform multi-scale image segmentation on the original video stream, and obtain standardized video content features;

[0042] Collecting the original audio stream from the audio output port of the digital set-top box, and performing a joint time-frequency domain analysis on the original audio stream to obtain standardized audio content features;

[0043] The standardized operation feature sequence, standardized video content features and standardized audio content features are aligned in the time dimension to obtain a standard multimodal dataset.

[0044] Specifically, the remote control button operations of the intelligent interactive control unit are captured in real time. In the interaction between the user and the digital set-top box, the specific actions of each remote control button are recorded in real time, including the type of button pressed, the duration, and the order of the buttons pressed. The original operation sequence data obtained is usually discrete time series data. Since the user's operation behavior has a certain degree of randomness and irregularity, the sliding time window method is used to segment the operation sequence data. Assume that the original operation sequence data is ,in Indicates a time point, dividing the entire operation sequence into multiple time windows, each containing a certain number of operation events. Set the time window size to , the sliding step length is , then at time point The window content is represented as . In this way, multiple overlapping windows are obtained, each window contains a continuous period of operation events, and these windows can obtain standardized operation feature sequences after feature extraction. At the same time, the original video stream is collected from the video output port of the digital set-top box. The real-time output video signals are captured, and these video signals contain the specific content that the user is watching, including TV programs, movies, etc. Multi-scale image segmentation is performed on the collected original video stream to extract content that can represent the characteristics of the scene. The video frame is divided into regions of different scales to identify important objects and background information in the image. For example, if the video frame is ,in and is the spatial coordinate of the image, is the time, and the image region obtained after segmentation is represented as a set of sub-regions of different scales , each sub-region Represents the feature information of different parts in the video frame. By segmenting regions of different scales, the detailed features and overall features of the video content are extracted to obtain standardized video content features. Multi-scale segmentation can effectively enhance the model's ability to understand the scene, enabling it to capture meaningful visual information at different scales. In the collection of audio data, the original audio stream is collected from the audio output port of the digital set-top box to obtain the sound information of the content the user is watching. The audio data contains various types of information such as speech, music, and environmental sound effects. In order to extract features from the audio data, the original audio stream is subjected to a joint analysis of the time and frequency domains. Joint analysis of the time and frequency domains is a technology that can simultaneously describe audio signals in time and frequency, using short-time Fourier transform (STFT) and the like. Assume that the original audio signal is , its short-time Fourier transform is expressed as:

[0045] ;

[0046] in, Represents the audio signal at time and frequency The amplitude of is a sliding window function. A short-time Fourier transform (SFT) is used to obtain the time-frequency spectrum of the audio signal, which displays the frequency distribution of the signal at different time points. Joint analysis in the time and frequency domains yields standardized audio content features that reflect the temporal and frequency variations of the audio signal, facilitating the training of scene recognition models. The standardized operational feature sequences, standardized video content features, and standardized audio content features are aligned in the temporal dimension to construct a complete multimodal dataset.

[0047] In a specific embodiment, the process of executing step S2 may specifically include the following steps:

[0048] The standard multimodal dataset is divided into an operation sequence data subset, a video content data subset, and an audio data subset;

[0049] Extract temporal features from a subset of the operation sequence data and process them through a long short-term memory network. The long short-term memory network contains three hidden layers, each with 128 neurons, and uses the tanh activation function to obtain the temporal features of the operation sequence.

[0050] Visual features are extracted from a subset of video content data and processed through a residual network. The residual network consists of 50 convolutional layers. Each residual block uses the ReLU activation function and batch normalization. The last layer uses global average pooling to obtain visual features of the video content.

[0051] Audio features are extracted from a subset of the audio data and processed using a one-dimensional convolutional neural network. The one-dimensional convolutional neural network contains four convolutional layers, each using the ELU activation function and maximum pooling, and the last layer uses global maximum pooling to obtain audio content features.

[0052] Connect the operation sequence temporal features, video content visual features, and audio content features to form a multimodal feature vector;

[0053] The multimodal feature vector is weighted by feature importance through the attention mechanism. The attention mechanism consists of a two-layer feedforward neural network, which uses the softmax function to calculate the weight of each feature to obtain the weighted multimodal features.

[0054] The weighted multimodal features are input into the cross-modal fusion network. The cross-modal fusion network consists of three fully connected layers. Each layer uses the LeakyReLU activation function and Dropout regularization to obtain preliminary fusion features.

[0055] An adaptive feature aggregation algorithm is applied to the preliminary fusion features, and the fusion scene feature vector is obtained through dynamic threshold screening and nonlinear transformation.

[0056] Specifically, the time series feature extraction is performed on the subset of operation sequence data. User operation data has a strong time dependency, and the long short-term memory network (LSTM) is used to process these operation sequences. Assume that the operation sequence is ,in Indicates time Operation, The operation sequence is input into an LSTM network with 3 hidden layers, each layer contains 128 neurons, and uses the tanh activation function. LSTM captures long-term dependencies in the time series and outputs the feature representation of each time step. By formula ,in is the weight matrix, is the bias term, Is the activation function, get the hidden state of each time step , and obtain the overall operation sequence temporal features. Visual feature extraction is performed on the subset of video content data. Video data contains rich scene and object information, and the residual network (ResNet) is used to extract visual features. Assume that the input video frame is , processed by 50 convolutional layers in the residual network. Each residual block in the residual network contains multiple convolutional layers, uses the ReLU activation function and performs batch normalization. Through batch normalization, the features remain stable during training, thereby accelerating the convergence of the network. In the last layer of the residual network, a global average pooling operation is used to calculate the average value of each feature map to form a fixed-length visual feature vector. The formula for global average pooling is:

[0057] ;

[0058] in, represents the visual features after pooling, and Represent the height and width of the feature map respectively, is the position in the feature map The value of . Through the processing of the residual network, the standardized visual features containing scene and object information are extracted to reflect the core visual information in the video content. Audio feature extraction is performed on the audio data subset. Audio signals usually contain information such as language, background music, and environmental sound effects. A one-dimensional convolutional neural network (1D-CNN) is used to process audio data. Assume that the input audio signal is ,in Indicates time The audio amplitude is denoted by . The one-dimensional convolutional network consists of four convolutional layers, each of which uses the ELU activation function, which produces a smaller negative output in the negative part, thereby enhancing the expressiveness of the feature. At the same time, a maximum pooling operation is performed after each convolution layer to reduce the feature dimension and computational complexity. In the final layer, global maximum pooling is used to compress the features into a fixed-length feature vector, using the formula:

[0059] ;

[0060] in, represents the audio features after pooling, Indicates the maximum value in the time dimension. In this way, the standardized audio features are obtained, which effectively reflect the main information in the audio signal. The operation sequence timing features, video content visual features and audio content features are connected to form a complete multimodal feature vector. Let the operation feature be , the visual features are , the audio features are , then the multimodal feature vector Expressed as , which enables the model to use operation, visual and audio information for scene recognition at the same time. In order to highlight the importance of each modal feature, the multimodal feature vector is weighted by the attention mechanism. The attention mechanism contains a two-layer feedforward neural network, which calculates the weight of each feature through the softmax function, thereby weighting the features of different modalities. Assume that the attention weight is , then the weighted multimodal feature vector is expressed as:

[0061] ;

[0062] in, Indicates the The characteristics of a mode, is the weight calculated by softmax, . In this way. The model can pay more attention to the modal features that are most important for the current scene recognition. The weighted multimodal features are input into the cross-modal fusion network for processing. The cross-modal fusion network contains 3 fully connected layers. Each layer uses the LeakyReLU activation function and uses Dropout regularization to prevent overfitting. The LeakyReLU activation function effectively solves the "dead neuron" problem, while Dropout regularization enhances the generalization ability of the network by randomly discarding a part of the neurons. Preliminary fusion features are obtained through the processing of the cross-modal fusion network. An adaptive feature aggregation algorithm is applied to the preliminary fusion features to optimize the feature representation. The adaptive feature aggregation algorithm selects the most important features and performs nonlinear combination of features through dynamic threshold screening and nonlinear transformation. Assume that the preliminary fusion features are , the process of dynamic threshold screening is represented as selecting characteristics, among which is the dynamic threshold. Then the final fusion scene feature vector is obtained through nonlinear transformation .

[0063] In a specific embodiment, the process of executing step S3 may specifically include the following steps:

[0064] The fused scene feature vector is input into the encoder network of the variational autoencoder. The encoder network consists of three fully connected layers, each of which uses the LeakyReLU activation function to obtain an intermediate feature representation.

[0065] The intermediate feature representation is processed through two parallel fully connected layers to obtain the mean vector and log variance vector respectively. Based on the mean vector and log variance vector, a reparameterization technique is used to generate latent variable samples. The reparameterization process involves sampling noise from a standard normal distribution and combining it with the mean and variance.

[0066] Input the latent variable sample into the decoder network of the variational autoencoder. The decoder network consists of three fully connected layers, each layer uses the ReLU activation function to obtain the reconstructed feature vector;

[0067] Calculate the reconstruction error between the reconstructed feature vector and the fused scene feature vector, and calculate the KL divergence of the latent variables to obtain the overall loss function value;

[0068] Based on the overall loss function value, an adjusted latent variable sample is generated. The adjusted latent variable sample is subjected to a spherical projection operation to map the points in the high-dimensional space onto the unit hypersphere to obtain normalized semantic features. The normalized semantic features are then transformed into a feature space to obtain a compressed semantic feature vector.

[0069] Specifically, the fusion scene feature vector is input into the encoder network consisting of 3 fully connected layers, and each layer uses the LeakyReLU activation function. The LeakyReLU activation function can effectively solve the problem of dead neurons and enhance the learning ability and stability of the network by retaining a small non-zero gradient for negative inputs. After layer-by-layer processing of the encoder network, the intermediate feature representation is obtained. The intermediate feature representation is processed to generate the mean vector and logarithmic variance vector required by the variational autoencoder. The intermediate feature representation is input into two parallel fully connected layers respectively. The first layer outputs the mean vector and the second layer outputs the logarithmic variance vector. The mean vector represents the center position of the latent variable, while the logarithmic variance vector is used to calculate the degree of diffusion of the latent variable. Assume that the intermediate feature representation is , after two parallel fully connected layers, the mean vectors are obtained respectively and the logarithmic variance vector In order to generate latent variable samples, the reparameterization technique is used to maintain the differentiability of the model by introducing randomness and achieve end-to-end training. The reparameterization process is expressed as:

[0070] ;

[0071] in, represents the standard deviation, Represents a noise vector sampled from a standard normal distribution. By combining the mean and random noise, a latent variable sample is generated. . The latent variable samples are input into the decoder network of the variational autoencoder for reconstruction. The decoder network also contains 3 fully connected layers, each layer uses the ReLU activation function. The ReLU activation function can maintain a large gradient when the output is a positive number, making it easier for the model to converge. After processing by the decoder network, the reconstructed feature vector is obtained. In order to measure the reconstruction effect, the reconstruction error between the reconstructed feature vector and the fused scene feature vector is calculated using the mean square error, which is as follows:

[0072] ;

[0073] in, The first character vector of the original fused scene elements, represents the corresponding reconstruction feature, is the length of the feature vector. The reconstruction error reflects the difference between the reconstructed features and the original features. The smaller the error, the more accurate the decoder's reconstruction of the original features. At the same time, the KL divergence (Kullback-Leibler divergence) of the latent variable is calculated to measure the difference between the generated latent distribution and the standard normal distribution. The calculation formula of KL divergence is:

[0074] ;

[0075] in, and Respectively represent The mean and variance of the latent variables, represents the dimension of the latent space. The KL divergence term is used to encourage the distribution of latent variables to be close to the standard normal distribution, so that the latent space has better structured properties. By combining the reconstruction error and KL divergence, the overall loss function value is obtained:

[0076] ;

[0077] in, Is a weight parameter used to control the trade-off between reconstruction error and KL divergence. During the training process, by minimizing the overall loss function and optimizing the parameters of the encoder and decoder, the reconstructed features are close to the original features while ensuring the continuity and regularization effect of the latent space. The adjusted latent variable samples are generated based on the overall loss function value. The adjusted latent variable samples are subjected to a spherical projection operation to map the points in the high-dimensional space onto the unit hypersphere. The purpose of spherical projection is to normalize the latent variable samples to ensure that their distribution in the high-dimensional space is more uniform and orderly. Assume that the adjusted latent variable samples are , then the sample after spherical projection is expressed as:

[0078] ;

[0079] in, Represents a vector The Euclidean norm of the latent variable samples is projected onto the unit hypersphere to ensure that the modulus of all samples is 1, avoiding the impact of numerical amplitude differences on subsequent feature processing. The normalized semantic features are transformed into feature space to obtain a compressed semantic feature vector. The latent variables are compressed and optimized through linear or nonlinear transformations, so that the final feature vector has higher discrimination and more compact representation capabilities. For example, through the linear projection matrix Mapping the normalized latent variables to a lower dimensional space is expressed as:

[0080] ;

[0081] in, is the transformation matrix, Represents the compressed semantic feature vector. In this way, the final compressed semantic feature vector is obtained.

[0082] In a specific embodiment, the execution step processes the intermediate feature representation through two parallel fully connected layers to obtain a mean vector and a logarithmic variance vector, respectively, and generates a latent variable sample based on the mean vector and the logarithmic variance vector using a reparameterization technique. The reparameterization process includes sampling noise from a standard normal distribution and combining it with the mean and variance, which can specifically include the following steps:

[0083] The intermediate feature representation is input into the first fully connected layer. The first fully connected layer uses a linear activation function to obtain the original mean vector, and batch normalization is applied to the original mean vector to standardize the feature distribution and obtain the standardized mean vector.

[0084] The intermediate feature representation is input into the second fully connected layer. The second fully connected layer uses the Softplus activation function to obtain the original logarithmic variance vector, and applies the exponential function transformation to the original logarithmic variance vector to convert the logarithmic variance into variance to obtain the variance vector;

[0085] Based on the dimension of the standardized mean vector, generate a random noise vector with the same dimension. The random noise vector obeys the standard normal distribution, and multiply the random noise vector by the square root of the variance vector to obtain the scaled noise vector.

[0086] The scaled noise vector is added to the standardized mean vector to obtain the initial latent variable sample, and the spherical projection function is applied to the initial latent variable sample to project the sample point onto the unit hypersphere to obtain the latent variable sample.

[0087] Specifically, the intermediate feature representation is input into the first fully connected layer to obtain the original mean vector. Assume that the intermediate feature representation is , represents the main features extracted by the encoder network of the variational autoencoder. Input to the first fully connected layer, which uses a linear activation function and outputs the mean vector after linear combination of features, denoted as The original mean vector Batch normalization is applied to features with a certain offset and imbalanced distribution. The feature distribution is standardized so that the mean is 0 and the variance is 1, which improves the stability and convergence speed of model training. The formula for batch normalization is expressed as:

[0088] ;

[0089] in, represents the normalized mean vector elements, and Var denote the mean and variance respectively, Is a small positive value used to avoid the denominator being zero. After batch normalization, the standardized mean vector is obtained At the same time, the intermediate features are represented as Input to the second fully connected layer to obtain the original logarithmic variance vector. In order to ensure that the variance is always non-negative, the fully connected layer uses the Softplus activation function. Softplus is a smoothed ReLU variant defined as:

[0090] ;

[0091] in, represents the intermediate feature representation elements, Represents the elements of the corresponding logarithmic variance vector. The value obtained after Softplus activation is considered to be the logarithmic variance. Applying the exponential function converts it to the actual variance value:

[0092] ;

[0093] in, The standard deviation of the variance is represented by the exponential function, which is used to convert the logarithmic variance into the actual variance value. After processing, the obtained variance vector is used to describe the uncertainty of the feature and help construct the latent variable. Based on the standardized mean vector and variance vector Generate a random noise vector of the same dimension as , where the noise vector obeys the standard normal distribution, that is In order to make the size of the noise consistent with the feature distribution, the random noise vector is multiplied element by element by the square root of the variance vector to scale the noise and obtain the scaled noise vector:

[0094] ;

[0095] in, represents the scaled noise vector, represents random noise sampled from a standard normal distribution, is the standard deviation of the variance. This process ensures that the generated latent variable samples have a certain degree of randomness in distribution, while being controlled by the mean and variance, so that the sample distribution is reasonable. Add together to get the initial latent variable sample

[0096] ;

[0097] in, represents the initial latent variable sample, and Represent the normalized mean vector and the scaled noise vector respectively. The generated latent variable samples contain the main information of the intermediate features, and also introduce a certain degree of randomness, which helps the model learn the distribution structure of the latent space. Apply the spherical projection function to map the points in the high-dimensional space onto the unit hypersphere to ensure that the modulus of the latent variable sample is 1. The formula for spherical projection is expressed as:

[0098] ;

[0099] in, represents the latent variable sample after spherical projection, is a vector By projecting the initial latent variable samples onto the unit hypersphere, we ensure that all samples have the same modulus length, improve the uniformity of the latent space, and reduce the impact of numerical differences on subsequent analysis to obtain the latent variable samples.

[0100] In a specific embodiment, the process of executing step S4 may specifically include the following steps:

[0101] Performing text encoding on predefined scene categories to obtain an initial scene query vector set, and performing scene feature enhancement on the initial scene query vector set to obtain an enhanced scene query vector set;

[0102] Perform dimension alignment on the compressed semantic feature vector and the enhanced scene query vector set to obtain aligned feature vectors, and calculate the cosine similarity between the aligned feature vectors to obtain the initial similarity matrix;

[0103] The initial similarity matrix is ​​input into the multi-head attention mechanism for processing to obtain the attention-weighted similarity matrix, and the attention-weighted similarity matrix is ​​subjected to nonlinear transformation to obtain the scene matching probability distribution;

[0104] Based on the scene matching probability distribution, K scene matching results with the highest probability are selected, and the weighted sum of the selected K scene matching results is performed to obtain the scene matching score.

[0105] Specifically, the predefined scene categories are encoded in text, and the text description of each scene category is mapped into a high-dimensional vector through a natural language processing model, such as a pre-trained BERT model or other semantic embedding model, to obtain the initial scene query vector set. Assume that the initial scene query vector set is ,in is the number of scene categories, each vector is a high-dimensional vector representation of the corresponding scene category. The initial scene query vector set is enhanced with scene features to obtain an enhanced scene query vector set. This is achieved through the self-attention mechanism or by adding additional context information. For example, for each scene query vector Perform self-attention calculation to obtain the enhanced query vector This process is expressed as:

[0106] ;

[0107] in, Represents the attention weight, by calculating the scene With scene The attention weight is calculated by the softmax function to ensure that the sum of all weights is 1. The enhanced scene query vector set after processing is The enhanced scene query vector set is dimensionally aligned with the compressed semantic feature vector so that similarity can be calculated in the same feature space. Assume that the compressed semantic feature vector is , in order to align with the enhanced scene query vector set, z is projected to the same dimensional space as the query vector set through a linear transformation. Assume that the linear transformation matrix is , then the aligned feature vector is expressed as:

[0108] ;

[0109] in, is the representation of the compressed semantic feature vector in the aligned space, is a linear transformation matrix used to match dimensions. By aligning the dimensions, we ensure that the compressed semantic features and the enhanced scene query vector are calculated in the same space for similarity. The cosine similarity is calculated between the aligned feature vectors to obtain the initial similarity matrix. Cosine similarity is used to measure the similarity between two vectors. Its value varies between -1 and 1, where 1 means that the two vectors are completely similar, 0 means they are irrelevant, and -1 means they are completely opposite. The formula for calculating cosine similarity is:

[0110] ;

[0111] in, Represents the compressed semantic feature vector and the enhanced scene query vector The cosine similarity between and Represent the Euclidean norm of the vector respectively. The initial similarity matrix is ​​calculated in this way , where each element Represents the similarity between the semantic feature and the scene query vector. The initial similarity matrix is ​​input into the multi-head attention mechanism for processing. The multi-head attention mechanism captures the relationship between different features through multiple sets of independent attention heads. Each attention head focuses on a different feature subspace, thereby learning the dependency between features in multiple subspaces. Attention heads are used to process the initial similarity matrix through the multi-head attention mechanism to obtain the attention-weighted similarity matrix. This process is expressed as:

[0112] ;

[0113] in, The attention-weighted similarity matrix is ​​represented by MultiHeadAttention, which represents the operation of the multi-head attention mechanism. This enhances the important relationships in the similarity matrix, highlighting the most important parts for scene matching. A nonlinear transformation is performed on the attention-weighted similarity matrix to obtain the scene matching probability distribution. This nonlinear transformation is implemented using the softmax function to ensure that the matching probability of each scene is between 0 and 1, and that the sum of all probabilities is 1. The softmax function is calculated as follows:

[0114] ;

[0115] in, Representation scene The matching probability, Represents the similarity matrix after attention weighting The similarity value of each scene. The scene matching probability distribution obtained by the softmax function It can intuitively represent the matching degree between each scene category and the current semantic feature. Based on the scene matching probability distribution, the previous The scene matching results with the highest probability are selected to ensure that the selected scene category is most relevant to the current semantic features. The weighted sum of the scene matching results is used to obtain the final scene matching score. The scene matching results are , then the scene matching score is expressed as:

[0116] ;

[0117] Among them, Score is the scene matching score, Indicates the The matching probability of each scene, Represents the corresponding similarity value. By taking a weighted sum of the matching probability and similarity, we get a scene matching score that comprehensively considers similarity and matching probability.

[0118] In a specific embodiment, the process of executing step S5 may specifically include the following steps:

[0119] Normalizing the scene matching score to obtain a normalized scene matching score, and constructing a feature vector based on the normalized scene matching score;

[0120] The feature vector is input into the random forest classifier, which contains 100 decision trees with a maximum depth of 10 for each tree, to obtain the first sub-classification result;

[0121] The feature vector is input into the support vector machine classifier, which uses a radial basis function kernel and a penalty parameter C set to 1.0 to obtain the second sub-classification result;

[0122] Apply the gradient boosting decision tree algorithm to the feature vector. The gradient boosting decision tree algorithm contains 200 weak learners and the learning rate is set to 0.1 to obtain the third sub-classification result.

[0123] The first sub-classification results, the second sub-classification results, and the third sub-classification results are encoded to obtain an encoded sub-classification result set. The Dempster-Shafer evidence theory is applied to the encoded sub-classification result set to calculate the confidence interval of each category and obtain a fused evidence matrix.

[0124] Based on the fused evidence matrix, the optimal category is selected to obtain the preliminary scene category, and the preliminary scene category is Markov smoothed to obtain the target scene classification result.

[0125] Specifically, the scene matching score is normalized to obtain a normalized scene matching score. A feature vector is constructed based on the normalized scene matching score. The feature vector is input into a random forest classifier. Random forest is an ensemble learning method that improves the robustness and generalization ability of the model by integrating multiple decision trees. A random forest classifier containing 100 decision trees is used, and the maximum depth of each tree is 10 to prevent overfitting and ensure that the complexity of the model is moderate. The first sub-classification result is obtained through the random forest classifier. Assuming that the feature vector The output category is ,in , represents a category in the scene category set. The feature vector is input into the support vector machine (SVM) classifier for classification. Support vector machine is an algorithm applied to classification tasks, which can effectively find the optimal hyperplane in the feature space to maximize the interval between different categories. The radial basis function kernel (RBF kernel) is used as the kernel function of SVM, and the penalty parameter is set to Set to 1.0 to control the model's tolerance to errors. After passing the SVM classifier, the second sub-classification result is obtained, which is recorded as ,same Also belongs to the scene category collection To improve the accuracy of classification, the Gradient Boosting Decision Tree (GBDT) algorithm is applied to the feature vector for classification. The Gradient Boosting Decision Tree is an ensemble learning method that improves the overall model performance by gradually adding weak learners. 200 weak learners are used and the learning rate is set to 0.1 to ensure that the update step of each learner is small, thereby reducing the risk of overfitting. Feature vector After inputting into GBDT, the third sub-classification result is obtained, which is recorded as . For the first sub-category results , the second sub-classification results and the third sub-category results Encode and get the encoded sub-classification result set. Assume that the encoded result set is , where each sub-classification result corresponds to the encoding of the scene category. In order to integrate the outputs of different sub-classifiers, the Dempster-Shafer evidence theory is used to calculate the confidence interval of each category and obtain the fused evidence matrix. The Dempster-Shafer evidence theory is a method for dealing with uncertainty and fusing multiple information sources. By combining the evidence of multiple classifiers, the probability of each category can be better estimated. Let the confidence interval be , indicating the category The credibility of , then the fused evidence matrix is ​​expressed as:

[0126] ;

[0127] in, Representation category The reliability, Indicates the Classifiers for categories By fusing the support of all sub-classifiers, the comprehensive reliability of each category is obtained. After obtaining the fused evidence matrix, the category with the highest reliability is selected as the preliminary scene category, which is recorded as The selection of the preliminary scene category is based on the maximum value of the confidence interval, which means that the most likely category is selected as the current scene prediction result. Markov smoothing is performed on the preliminary scene category. Markov smoothing is a time series-based method that smoothes the current prediction result by considering the category status of the previous and next moments to reduce the instability of the classification. Assume that the preliminary scene category at the current moment is , the scene category at the previous moment is , then the target scene classification result after Markov smoothing is expressed as:

[0128] ;

[0129] in, represents the probability of the smoothed scene category at the current moment, is a smoothing parameter used to control the balance between the current prediction result and the state at the previous moment. In this way, the continuity and rationality of the target scene classification results in the time dimension can be ensured, thereby improving the system's responsiveness to changes in user scenarios.

[0130] The above describes the intelligent scene recognition method of the intelligent interactive control unit in the embodiment of the present invention. The following describes the intelligent scene recognition device of the intelligent interactive control unit in the embodiment of the present invention. Figure 2 In one embodiment of the present invention, an intelligent scene recognition device of an intelligent interactive control unit includes:

[0131] The acquisition module is used to collect and preprocess the operation sequence, video content and audio data of the intelligent interactive control unit in the digital set-top box to obtain a standard multimodal data set;

[0132] The extraction module is used to input the standard multimodal dataset into the multi-branch deep neural network model, extract the temporal, visual and audio features through parallel processing, and obtain the fused scene feature vector;

[0133] A mapping module is used to perform dimensionality reduction and semantic mapping on the fusion scene feature vector using a variational autoencoder to obtain a compressed semantic feature vector;

[0134] The calculation module is used to calculate the similarity between the compressed semantic feature vector and the predefined scene query vector, and obtain the scene matching score through the attention mechanism;

[0135] The output module is used to integrate the outputs of multiple sub-classifiers based on the scene matching score using an ensemble learning method to obtain the target scene classification result.

[0136] Through the collaborative efforts of these components and by simultaneously processing operation sequences, video content, and audio data, this method can comprehensively capture the characteristics of user usage scenarios and improve the accuracy and robustness of scene recognition. The multi-branch deep neural network model enables parallel processing of data from different modalities, effectively extracting temporal, visual, and audio features, and improving the efficiency and quality of feature extraction. The use of a variational autoencoder for dimensionality reduction and semantic mapping of fused features not only reduces data dimensionality but also captures potential semantic information, helping to improve the accuracy of subsequent scene recognition. The introduction of an attention mechanism during similarity calculation can adaptively focus on important features and improve the accuracy of scene matching. The use of multiple sub-classifiers combined with the Dempster-Shafer evidence theory can combine the advantages of different classifiers and improve the model's generalization and robustness. Through lightweight optimization of the task flow graph, this method can run efficiently on resource-constrained digital set-top boxes, achieving real-time scene recognition.

[0137] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, memory, display screen, input device, network interface and database connected via a system bus. The processor of the computer design is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above method is implemented.

[0138] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0139] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the above-described method when executed by a processor. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0140] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware using a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media provided herein and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM.

[0141] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0142] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0143] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent scene recognition method for an intelligent interactive control unit, characterized in that: The method comprises: The operation sequence, video content and audio data of the intelligent interactive control unit in the digital set-top box are collected and preprocessed to obtain a standard multimodal dataset; specifically, the method includes: real-time capture of the remote control button operations of the intelligent interactive control unit to obtain original operation sequence data, and sliding time window segmentation of the original operation sequence data to obtain a standardized operation feature sequence; collecting the original video stream from the video output port of the digital set-top box, and performing multi-scale image segmentation on the original video stream to obtain standardized video content features; collecting the original audio stream from the audio output port of the digital set-top box, and performing time-frequency domain joint analysis on the original audio stream to obtain standardized audio content features; aligning the standardized operation feature sequence, the standardized video content features and the standardized audio content features in the time dimension to obtain a standard multimodal dataset; The standard multimodal dataset is input into a multi-branch deep neural network model, and the timing, visual and audio features are extracted through parallel processing to obtain a fusion scene feature vector; specifically comprising: dividing the standard multimodal dataset into an operation sequence data subset, a video content data subset and an audio data subset; performing timing feature extraction on the operation sequence data subset, processing it through a long short-term memory network, the long short-term memory network comprising 3 hidden layers, 128 neurons in each layer, and using a tanh activation function to obtain operation sequence timing features; performing visual feature extraction on the video content data subset, processing it through a residual network, the residual network comprising 50 convolutional layers, each residual block using a ReLU activation function and batch normalization, and the last layer using global average pooling to obtain video content visual features; performing audio feature extraction on the audio data subset, processing it through a one-dimensional convolutional neural network, the one The method comprises the following steps: a 3D convolutional neural network comprising 4 convolutional layers, each layer using the ELU activation function and maximum pooling, and the last layer using the global maximum pooling to obtain audio content features; the operation sequence temporal features, the video content visual features and the audio content features are connected to form a multimodal feature vector; the multimodal feature vector is weighted by feature importance through an attention mechanism, the attention mechanism comprising a two-layer feedforward neural network, and the softmax function is used to calculate the weight of each feature to obtain weighted multimodal features; the weighted multimodal features are input into a cross-modal fusion network, the cross-modal fusion network comprising 3 fully connected layers, each layer using the LeakyReLU activation function and Dropout regularization to obtain preliminary fusion features; an adaptive feature aggregation algorithm is applied to the preliminary fusion features, and a fusion scene feature vector is obtained through dynamic threshold screening and nonlinear transformation; A variational autoencoder is used to perform dimensionality reduction and semantic mapping on the fusion scene feature vector to obtain a compressed semantic feature vector; specifically comprising: inputting the fusion scene feature vector into the encoder network of the variational autoencoder, the encoder network comprising 3 fully connected layers, each layer using a LeakyReLU activation function to obtain an intermediate feature representation; inputting the intermediate feature representation into the first fully connected layer, the first fully connected layer using a linear activation function to obtain an original mean vector, and applying batch normalization to the original mean vector to standardize the feature distribution to obtain a standardized mean vector; inputting the intermediate feature representation into the second fully connected layer, the second fully connected layer using a Softplus activation function to obtain an original logarithmic variance vector, and applying an exponential function transformation to the original logarithmic variance vector to convert the logarithmic variance into a variance to obtain a variance vector; based on the dimension of the standardized mean vector, generating a random noise vector with the same dimension, the random noise vector obeys a standard normal distribution, and multiplying the random noise vector by the square root of the variance vector to obtain a scaled noise vector; adding the scaled noise vector to the standardized mean vector to obtain an initial latent variable sample, and applying a spherical projection function to the initial latent variable sample to project the sample point onto a unit hypersphere to obtain a latent variable sample; inputting the latent variable sample into the decoder network of the variational autoencoder, the decoder network comprising three fully connected layers, each layer using a ReLU activation function to obtain a reconstructed feature vector; calculating the reconstruction error between the reconstructed feature vector and the fusion scene feature vector, and simultaneously calculating the KL divergence of the latent variable to obtain an overall loss function value; generating an adjusted latent variable sample based on the overall loss function value, and performing a spherical projection operation on the adjusted latent variable sample to map the points in the high-dimensional space onto a unit hypersphere to obtain a normalized semantic feature, and performing a feature space transformation on the normalized semantic feature to obtain a compressed semantic feature vector; Calculate the similarity between the compressed semantic feature vector and the predefined scene query vector, and obtain a scene matching score through the attention mechanism; specifically include: text encoding the predefined scene categories to obtain an initial scene query vector set, and performing scene feature enhancement on the initial scene query vector set to obtain an enhanced scene query vector set; perform dimension alignment operation on the compressed semantic feature vector and the enhanced scene query vector set to obtain aligned feature vectors, and calculate the cosine similarity between the aligned feature vectors to obtain an initial similarity matrix; input the initial similarity matrix into the multi-head attention mechanism for processing to obtain an attention-weighted similarity matrix, and perform nonlinear transformation on the attention-weighted similarity matrix to obtain a scene matching probability distribution; based on the scene matching probability distribution, select K scene matching results with the highest probability, and perform weighted summation on the selected K scene matching results to obtain a scene matching score; Based on the scene matching score, an ensemble learning method is used to synthesize the outputs of multiple sub-classifiers to obtain a target scene classification result; specifically, the method includes: normalizing the scene matching score to obtain a normalized scene matching score, and constructing a feature vector based on the normalized scene matching score; inputting the feature vector into a random forest classifier, wherein the random forest classifier contains 100 decision trees, and the maximum depth of each tree is 10, to obtain a first sub-classification result; inputting the feature vector into a support vector machine classifier, wherein the support vector machine classifier uses a radial basis function kernel and the penalty parameter C is set to 1.0, to obtain a second sub-classification result; and performing a weighted average on the feature vector. A gradient boosting decision tree algorithm is applied, wherein the gradient boosting decision tree algorithm includes 200 weak learners and the learning rate is set to 0.1 to obtain the third sub-classification result; the first sub-classification result, the second sub-classification result and the third sub-classification result are encoded to obtain an encoded sub-classification result set, and the Dempster-Shafer evidence theory is applied to the encoded sub-classification result set to calculate the confidence interval of each category to obtain a fused evidence matrix; based on the fused evidence matrix, the optimal category is selected to obtain a preliminary scene category, and the preliminary scene category is Markov smoothed to obtain the target scene classification result.

2. An intelligent scene recognition device for an intelligent interactive control unit, characterized in that: The device for executing the intelligent scene recognition method of the intelligent interactive control unit according to claim 1 comprises: The acquisition module is used to collect and preprocess the operation sequence, video content and audio data of the intelligent interactive control unit in the digital set-top box to obtain a standard multimodal data set; An extraction module, configured to input the standard multimodal dataset into a multi-branch deep neural network model, extract temporal, visual, and audio features through parallel processing, and obtain a fused scene feature vector; A mapping module, configured to perform dimensionality reduction and semantic mapping on the fusion scene feature vector using a variational autoencoder to obtain a compressed semantic feature vector; a calculation module, configured to calculate a similarity between the compressed semantic feature vector and a predefined scene query vector, and obtain a scene matching score through an attention mechanism; The output module is used to synthesize the outputs of multiple sub-classifiers based on the scene matching score using an ensemble learning method to obtain a target scene classification result.

3. A computer device, characterized in that: The intelligent scene recognition method of the intelligent interactive control unit according to claim 1 is implemented when the processor executes the computer program.

4. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the processor is caused to execute the intelligent scene recognition method of the intelligent interactive control unit as claimed in claim 1.

Images