Intelligent scene identification method and equipment of intelligent interaction control unit

Abstract

The invention relates to the technical field of intelligent scene recognition, and discloses an intelligent scene recognition method and equipment for an intelligent interaction control unit, and the method comprises the steps: carrying out the collection and preprocessing of an operation sequence, video content and audio data of the intelligent interaction control unit in a digital set-top box, and obtaining a standard multi-mode data set; inputting the standard multi-modal data set into a multi-branch deep neural network model, and extracting time sequence, visual and audio features through parallel processing to obtain a fusion scene feature vector; carrying out dimensionality reduction and semantic mapping on the fused scene feature vector to obtain a compressed semantic feature vector; performing similarity calculation on the compressed semantic feature vector and a predefined scene query vector to obtain a scene matching score; and based on the scene matching score, integrating the output of the plurality of sub-classifiers to obtain a target scene classification result, and the method can be efficiently operated on the digital set top box with limited resources to realize real-time scene recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent scene recognition, and in particular to an intelligent scene recognition method and device for an intelligent interactive control unit. Background Art

[0002] Traditional digital set-top box interaction methods often lack the ability to intelligently identify user usage scenarios, resulting in a less than smooth user experience. Although some studies have attempted to perform scene recognition through single-modality data (such as remote control operation sequences), this approach often fails to accurately capture complex usage scenarios.

[0003] Multimodal data fusion provides a new approach to solving this problem, but how to effectively process and fuse heterogeneous data from different modalities remains a challenge. Traditional feature extraction and fusion methods often have difficulty capturing the complex relationships between data, resulting in low scene recognition accuracy. In addition, existing scene recognition models usually require a large amount of labeled data for training, which is often difficult to achieve in practical applications. Summary of the invention

[0004] The present invention provides an intelligent scene recognition method and device for an intelligent interactive control unit. The present invention can run efficiently on a digital set-top box with limited resources to achieve real-time scene recognition.

[0005] In a first aspect, the present invention provides an intelligent scene recognition method for an intelligent interaction control unit, the intelligent scene recognition method for the intelligent interaction control unit comprising: The operation sequence, video content and audio data of the intelligent interactive control unit in the digital set-top box are collected and preprocessed to obtain a standard multimodal data set; Inputting the standard multimodal dataset into a multi-branch deep neural network model, extracting temporal, visual and audio features through parallel processing, and obtaining a fused scene feature vector; Using a variational autoencoder to perform dimension reduction and semantic mapping on the fusion scene feature vector to obtain a compressed semantic feature vector; Calculate the similarity between the compressed semantic feature vector and the predefined scene query vector, and obtain a scene matching score through an attention mechanism; Based on the scene matching score, an ensemble learning method is used to integrate the outputs of multiple sub-classifiers to obtain a target scene classification result.

[0006] In a second aspect, the present invention provides an intelligent scene recognition device of an intelligent interaction control unit, the intelligent scene recognition device of the intelligent interaction control unit comprising: The acquisition module is used to collect and preprocess the operation sequence, video content and audio data of the intelligent interactive control unit in the digital set-top box to obtain a standard multimodal data set; An extraction module, used for inputting the standard multimodal data set into a multi-branch deep neural network model, extracting time series, visual and audio features through parallel processing, and obtaining a fused scene feature vector; A mapping module, used for performing dimension reduction and semantic mapping on the fusion scene feature vector by using a variational autoencoder to obtain a compressed semantic feature vector; A calculation module, used for calculating the similarity between the compressed semantic feature vector and the predefined scene query vector, and obtaining a scene matching score through an attention mechanism; The output module is used to obtain a target scene classification result by integrating the outputs of multiple sub-classifiers based on the scene matching score using an ensemble learning method.

[0007] The third aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the intelligent scene recognition method of the above-mentioned intelligent interaction control unit.

[0008] A fourth aspect of the present invention provides a computer-readable storage medium, in which instructions are stored, which, when executed on a computer, enable the computer to execute the intelligent scene recognition method of the intelligent interaction control unit described above.

[0009] In the technical solution provided by the present invention, by simultaneously processing the operation sequence, video content and audio data, the method can comprehensively capture the characteristics of the user's usage scenario and improve the accuracy and robustness of scene recognition. The multi-branch deep neural network model is adopted, which can process data of different modalities in parallel, effectively extract time series, visual and audio features, and improve the efficiency and quality of feature extraction. The use of variational autoencoders to reduce the dimension and semantically map the fused features not only reduces the data dimension, but also captures potential semantic information, which helps to improve the accuracy of subsequent scene recognition. Introducing the attention mechanism in the similarity calculation process can adaptively focus on important features and improve the accuracy of scene matching. The use of multiple sub-classifiers combined with the Dempster-Shafer evidence theory can combine the advantages of different classifiers and improve the generalization ability and robustness of the model. Through the lightweight optimization of the task flow graph, the method can run efficiently on resource-constrained digital set-top boxes to achieve real-time scene recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0011] Figure 1 A schematic diagram of the steps of an intelligent scene recognition method of an intelligent interactive control unit in an embodiment of the present invention; Figure 2 It is a structural schematic diagram of an intelligent scene recognition device of an intelligent interactive control unit in an embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0012] Embodiments of the present invention provide an intelligent scene recognition method and device for an intelligent interactive control unit. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0013] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the intelligent scene recognition method of the intelligent interactive control unit in the embodiment of the present invention includes: Step S1, collecting and preprocessing the operation sequence, video content and audio data of the intelligent interactive control unit in the digital set-top box to obtain a standard multimodal data set; It is understandable that the execution subject of the present invention may be an intelligent scene recognition device of an intelligent interactive control unit, or may be a terminal or a server, which is not specifically limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0014] Specifically, the key operations of the remote control of the intelligent interactive control unit are captured in real time to obtain the original operation sequence data, which has certain volatility and randomness due to factors such as user habits and reaction time. The original operation sequence data is segmented by a sliding time window, and the continuous operation sequence is divided into multiple time periods to obtain a standardized operation feature sequence. At the same time, the original video stream is collected from the video output port of the digital set-top box. The video content can reflect the specific scene of the current playback, such as the type of TV program, the content of the playback, the main characters or objects on the screen, and other information. These original video streams are collected through the video output port to obtain the original visual information. The original video stream is processed by multi-scale image segmentation technology, and different areas in the video frame are refined, so that the system can identify important objects, backgrounds and other visual features in the video. By segmenting video content at different scales, standardized video content features are obtained. The original audio stream is collected from the audio output port of the digital set-top box to obtain the sound information of the current playback content. The audio data contains rich content such as voice, music, and environmental sound effects. The original audio stream is jointly analyzed in the time and frequency domains. Time-frequency domain joint analysis is a method that can process audio signals in both the time domain and the frequency domain, effectively extracting the time-varying characteristics and frequency components of audio signals, and better understanding the structure and characteristics of audio content. Through analysis, standardized audio content features are obtained. The standardized operation feature sequence, standardized video content features, and standardized audio content features are aligned in the time dimension to obtain a standard multimodal data set.

[0015] Step S2: input the standard multimodal data set into a multi-branch deep neural network model, extract the temporal, visual and audio features through parallel processing, and obtain a fused scene feature vector; Specifically, the standard multimodal dataset is divided into an operation sequence data subset, a video content data subset, and an audio data subset. For the operation sequence data subset, a long short-term memory network (LSTM) is used to extract temporal features. The operation sequence data shows a certain temporal dependency, that is, the user's operation behavior has regularity and contextual association. The network structure of LSTM that can capture long-term and short-term dependencies is suitable for this data type. An LSTM network with three hidden layers is used, each hidden layer contains 128 neurons, and the activation function uses the tanh function. Through the processing of this structure, the temporal features of the operation sequence can be effectively extracted, and this feature information can be used for the fusion of multimodal features. At the same time, for the processing of the video content data subset, the residual network (ResNet) is used to extract visual features. Video content contains rich visual information, such as scenes, characters, objects, etc. The residual network can effectively alleviate the gradient vanishing problem and enable the model to be trained deeper. The residual network contains 50 convolutional layers, and the ReLU activation function is used in each residual block, and batch normalization is performed to improve the stability and convergence speed of training. In the last layer of the network, global average pooling is used to compress the features of the entire video frame into a visual feature vector of fixed length. For the processing of the audio data subset, a one-dimensional convolutional neural network is used to extract audio features. The audio data contains different types of signals such as speech, music, and ambient sound. The processing of the one-dimensional convolutional network effectively captures the timing and spectral characteristics of the audio signal. The one-dimensional convolutional network contains four convolutional layers, each of which uses the ELU activation function. The ELU activation function can produce a small negative output in the negative value area, thereby accelerating the convergence of the network. At the same time, each layer uses the maximum pooling operation to reduce the feature dimension and suppress noise. The last layer of the network uses global maximum pooling to compress the entire audio sequence into a fixed-length feature vector, which can effectively represent the key information in the audio data. The operation sequence timing features, video content visual features, and audio content features are connected to form a complete multimodal feature vector. The attention mechanism is used to weight the feature importance of multimodal features. The attention mechanism contains a two-layer feedforward neural network. The softmax function is used to calculate the weight of each feature, highlighting the most important features for the current scene and enhancing the model's ability to focus on key information. The attention-weighted multimodal features are input into the cross-modal fusion network. The cross-modal fusion network contains three fully connected layers. Each layer uses the LeakyReLU activation function to alleviate the "dead neuron" problem and improve the robustness and generalization ability of the model. At the same time, in order to prevent the model from overfitting, Dropout regularization is added after each fully connected layer to randomly discard some neurons and enhance the generalization ability of the model. After being processed by the cross-modal fusion network, preliminary fusion features are obtained. The adaptive feature aggregation algorithm is applied to the preliminary fusion features.The adaptive feature aggregation algorithm processes features through dynamic threshold screening and nonlinear transformation, dynamically adjusts the weight of features in different scenarios, enhances the expression ability of key features, and finally obtains a fused scene feature vector.

[0016] Step S3: Using a variational autoencoder to perform dimension reduction and semantic mapping on the fusion scene feature vector to obtain a compressed semantic feature vector; Specifically, the fusion scene feature vector is input into the encoder network of the variational autoencoder. The structure of the encoder network consists of three fully connected layers, each of which uses the LeakyReLU activation function. LeakyReLU can effectively solve the "dead neuron" problem and enhance the network's ability to represent features. Through the layer-by-layer processing of the encoder network, the original fusion scene features are mapped to an intermediate feature representation. The intermediate feature representation is processed by two parallel fully connected layers to generate a mean vector and a logarithmic variance vector respectively. The role of the mean vector and the logarithmic variance vector is to determine the distribution parameters of the latent variable. In the variational autoencoder, the generation of latent variables adopts a reparameterization technique, that is, sampling noise from a standard normal distribution, and then combining the sampled noise with the mean and variance to generate latent variable samples. This process involves linearly transforming the noise in the standard normal distribution, the mean vector controls the central position of the latent variable, and the logarithmic variance vector is used to control the degree of diffusion of the variable. Through the reparameterization technique, randomness can be effectively introduced into the model while maintaining the differentiability of the model. The latent variable samples are input into the decoder network of the variational autoencoder. The decoder network also contains three fully connected layers, and each fully connected layer in the decoder network uses the ReLU activation function. The ReLU activation function has good gradient propagation characteristics, which ensures that the network can effectively decode the latent variables. The reconstructed feature vector is obtained by layer-by-layer processing of the decoder network. The reconstruction error between the reconstructed feature vector and the fused scene feature vector is calculated and incorporated into the overall loss function for optimization. The reconstruction error is measured using mean square error or binary cross entropy, reflecting the gap between the features reconstructed by the decoder and the original input. At the same time, the KL divergence (Kullback-Leibler divergence) of the latent variable is calculated. This loss is used to measure the difference between the distribution of the latent variable and the standard normal distribution. By considering both the reconstruction error and the KL divergence, the variational autoencoder maximizes the original information of the input features while maintaining the continuity of the latent space. The overall loss function value contains the weighted sum of the reconstruction error and the KL divergence. By optimizing the overall loss, a more accurate representation of the latent variables is obtained. Based on the overall loss function value, the adjusted latent variable samples are generated to improve the quality and robustness of the features. The adjusted latent variable samples are projected onto the unit hypersphere through spherical projection. The purpose of spherical projection is to normalize the latent variables so that they are more evenly distributed in the high-dimensional space, thereby avoiding the skewness and imbalance in the feature space and obtaining normalized semantic features. The normalized semantic features are transformed into feature space to optimize the representation of the features, making them more compact and discriminative in semantics, and obtaining compressed semantic feature vectors.

[0017] The intermediate feature representation is input into the first fully connected layer, which uses a linear activation function to maintain the linear relationship between input and output, and obtains the original mean vector. Batch normalization is applied to the original mean vector. Through batch normalization, the feature distribution is adjusted to a standard normal distribution, so that the numerical range of the mean vector is more concentrated, thereby reducing the instability in network training and accelerating the convergence speed, and obtaining a standardized mean vector. At the same time, the intermediate feature representation is input into the second fully connected layer, which is responsible for generating the logarithmic variance vector. To ensure that the variance is non-negative, the second fully connected layer uses the Softplus activation function, which is a smoothed ReLU function variant that can convert the negative part of the input into a positive value, effectively avoiding the problem of negative variance. The original logarithmic variance vector is obtained by processing the Softplus activation function. The exponential function is applied to the original logarithmic variance vector to convert the logarithmic variance into variance, and a variance vector reflecting the uncertainty of the feature distribution is obtained. Based on the dimension of the standardized mean vector, a random noise vector with the same dimension is generated. In order to ensure the standardization of the source of random noise, the random noise vector follows the standard normal distribution, that is, the distribution with mean 0 and variance 1, so as to introduce appropriate randomness to the generated latent variable samples and improve the generalization ability of the model. The random noise vector is scaled to match the variance vector. The random noise vector is multiplied by the square root of the variance vector so that the degree of change of the noise vector is consistent with the size of the variance, and the scaled noise vector is obtained. The scaled noise vector is added to the standardized mean vector to obtain the initial latent variable sample. The latent variable sample generated in this way not only contains the basic information of the intermediate feature representation, but also gives the sample a certain randomness through the introduction of noise. The spherical projection function is applied to the initial latent variable sample to project the sample points onto the unit hypersphere. The role of spherical projection is to normalize the latent variables, unify their modulus length to 1, so that all latent variable samples are distributed on a unit hypersphere. Normalization ensures the uniform distribution of latent variables in high-dimensional space and avoids the imbalance caused by the difference in numerical amplitude. The final latent variable sample is obtained by projecting the latent variable sample onto the unit hypersphere.

[0018] Step S4, calculating the similarity between the compressed semantic feature vector and the predefined scene query vector, and obtaining a scene matching score through an attention mechanism; Specifically, a predefined scene category is encoded in text to generate an initial scene query vector set. A pre-trained text encoding model, such as a natural language processing model such as BERT or GPT, is used to encode the scene category described in the text into a semantically rich vector representation. A scene feature enhancement operation is performed on the initial scene query vector set to enrich the expression ability of the scene category. For example, the feature enhancement of the scene query vector is achieved by a self-attention mechanism or context information supplementation to obtain an enhanced scene query vector set. The compressed semantic feature vector and the enhanced scene query vector set are dimensionally aligned to perform similarity calculation in the same dimensional space. The alignment operation is achieved by linear projection or a fully connected layer so that the compressed semantic feature vector and the scene query vector have the same dimension. The cosine similarity is calculated based on the aligned feature vector to obtain an initial similarity matrix. The similarity between the two vectors is measured by calculating the cosine value of the angle between them. In the initial similarity matrix, each element represents the similarity between the compressed semantic feature vector and the specific scene query vector. The initial similarity matrix is ​​input into the multi-head attention mechanism for processing. The multi-head attention mechanism captures different feature relationships through multiple sets of independent attention heads. Each attention head can weight different features, so as to learn the dependency between features in different subspaces and obtain the attention-weighted similarity matrix. The attention-weighted similarity matrix is ​​transformed nonlinearly to obtain the scene matching probability distribution. Through an activation function, such as the softmax function, each element of the similarity matrix is ​​converted into a probability value, so that the matching probability of each scene category is between 0 and 1, and the sum of all probabilities is 1. In this way, the model intuitively represents the matching probability of each scene category with the compressed semantic feature vector, and the scene matching probability distribution reflects the matching degree between different scene categories and the current feature vector. Based on the scene matching probability distribution, K scene matching results with the highest probability are selected, which effectively narrows the target range and improves the matching accuracy. The selected K scene matching results are weighted and summed to obtain the final scene matching score. In the weighted summation process, each scene matching result is assigned different weights according to its corresponding matching probability, ensuring that the most relevant scene category contributes the most to the final matching result and obtains the scene matching score.

[0019] Step S5: Based on the scene matching score, an ensemble learning method is used to integrate the outputs of multiple sub-classifiers to obtain a target scene classification result.

[0020] Specifically, the scene matching scores are normalized, and the numerical range of the scene matching scores is standardized so that the scores from different sources are compared and calculated on the same scale to avoid deviations caused by score differences, and the normalized scene matching scores are obtained. A feature vector is constructed based on the normalized scores. The feature vector is input into the random forest classifier for classification. The random forest classifier is an integration method based on multiple decision trees. By integrating the classification results of multiple decision trees, the robustness and generalization ability of the model are effectively improved. The random forest contains 100 decision trees, and the maximum depth of each tree is 10, which ensures that the complexity of each tree is moderate and does not overfit the data. At the same time, through the integration of a large number of decision trees, the complex relationship between features is captured to obtain the first sub-classification result. The same feature vector is input into the support vector machine classifier for classification. The support vector machine is an effective binary classification model that maximizes the interval between different categories by finding the optimal hyperplane to achieve efficient classification. The support vector machine uses the radial basis function kernel (RBF kernel), which can map the original features to a high-dimensional space, so that nonlinear features can be linearly separable. At the same time, the penalty parameter C is set to 1.0, which is used to balance the model complexity and classification accuracy. The second sub-classification result is obtained through the processing of the support vector machine classifier. The gradient boosting decision tree (GBDT) algorithm is applied to the feature vector for classification. The gradient boosting decision tree is an ensemble learning method based on the additive model and the forward distribution algorithm. It gradually builds a series of weak learners and corrects the error of the previous round each time, thereby continuously improving the performance of the model. GBDT contains 200 weak learners, and the learning rate is set to 0.1. The learning rate determines the contribution of each weak learner to the final model. A smaller learning rate makes the model gradually approach the optimal solution, thereby reducing the risk of overfitting. The third sub-classification result is obtained through the training and prediction of the GBDT algorithm. The first sub-classification result, the second sub-classification result, and the third sub-classification result are encoded and merged into a unified representation to obtain the encoded sub-classification result set. The Dempster-Shafer evidence theory is applied to the encoded sub-classification result set. Dempster-Shafer evidence theory is a mathematical theory used to deal with uncertainty and integrate multiple information sources. By calculating the confidence interval of each category, the support of different sub-classifiers for each category is quantified to obtain the fused evidence matrix. Based on the fused evidence matrix, the optimal category is selected to obtain a preliminary scene category. In the evidence matrix, the confidence interval of each category indicates the possibility that the category is considered correct. Selecting the optimal category means selecting the category with the highest confidence as the preliminary classification result. Markov smoothing is performed on the preliminary scene category. Markov smoothing is a smoothing method based on time series. It adjusts the current state by considering the state of the previous and next moments, thereby ensuring the continuity and rationality of the scene classification results.Through Markov smoothing, the classification instability caused by instantaneous errors is eliminated, so that the final target scene classification result is more consistent with the logic of actual scene changes.

[0021] In an embodiment of the present invention, by simultaneously processing operation sequences, video content and audio data, the method can comprehensively capture the characteristics of user usage scenarios and improve the accuracy and robustness of scene recognition. A multi-branch deep neural network model is adopted, which can process data of different modalities in parallel, effectively extract temporal, visual and audio features, and improve the efficiency and quality of feature extraction. The use of variational autoencoders to reduce the dimension and semantically map the fused features not only reduces the data dimension, but also captures potential semantic information, which helps to improve the accuracy of subsequent scene recognition. Introducing the attention mechanism in the similarity calculation process can adaptively focus on important features and improve the accuracy of scene matching. The use of multiple sub-classifiers combined with the Dempster-Shafer evidence theory can combine the advantages of different classifiers and improve the generalization ability and robustness of the model. Through the lightweight optimization of the task flow graph, the method can run efficiently on resource-constrained digital set-top boxes to achieve real-time scene recognition.

[0022] In a specific embodiment, the process of executing step S1 may specifically include the following steps: The remote control button operations of the intelligent interactive control unit are captured in real time to obtain the original operation sequence data, and the original operation sequence data is segmented by sliding time windows to obtain a standardized operation feature sequence; Collecting the original video stream from the video output port of the digital set-top box, and performing multi-scale image segmentation on the original video stream to obtain standardized video content features; Collecting the original audio stream from the audio output port of the digital set-top box, and performing a joint analysis of the original audio stream in the time and frequency domains to obtain standardized audio content features; The standardized operation feature sequence, the standardized video content features and the standardized audio content features are aligned in the time dimension to obtain a standard multimodal dataset.

[0023] Specifically, the remote control button operations of the intelligent interactive control unit are captured in real time. In the interaction between the user and the digital set-top box, the specific actions of each remote control button are recorded in real time, including the type of button pressed, the duration, and the order of the buttons. The original operation sequence data obtained is usually discrete time series data. Since the user's operation behavior has a certain degree of randomness and irregularity, the sliding time window method is used to segment the operation sequence data. Assume that the original operation sequence data is ,in Indicates a time point, dividing the entire operation sequence into multiple time windows, each of which contains a certain number of operation events. Set the time window size to , the sliding step length is , then at time point The window content is represented as . In this way, multiple overlapping windows are obtained, each window contains a continuous section of operation events, and these windows can obtain standardized operation feature sequences after feature extraction. At the same time, the original video stream is collected from the video output port of the digital set-top box. The real-time output video signals are captured. These video signals contain the specific content that the user is watching, including TV programs, movies, etc. Multi-scale image segmentation is performed on the collected original video stream to extract content that can represent the characteristics of the scene. The video frame is divided into regions of different scales to identify important objects and background information in the image. For example, suppose the video frame is ,in and is the spatial coordinate of the image, is the time, and the image region obtained after segmentation is represented as a set of sub-regions of different scales , each sub-region Represents the feature information of different parts in the video frame. By segmenting regions of different scales, the detailed features and overall features of the video content are extracted to obtain standardized video content features. Multi-scale segmentation can effectively enhance the model's ability to understand the scene, enabling it to capture meaningful visual information at different scales. In the collection of audio data, the original audio stream is collected from the audio output port of the digital set-top box to obtain the sound information of the content the user is watching. The audio data contains various types of information such as speech, music, and environmental sound effects. In order to extract features from the audio data, the original audio stream is jointly analyzed in the time-frequency domain. Joint analysis in the time-frequency domain is a technology that can simultaneously describe audio signals in time and frequency, using short-time Fourier transform (STFT) and the like. Assume that the original audio signal is , its short-time Fourier transform is expressed as: ; in, Represents the audio signal at time and frequency The amplitude of is a sliding window function. Through short-time Fourier transform, the time-frequency spectrum of the audio signal is obtained, which can show the frequency distribution of the signal at different time points. After joint analysis in the time and frequency domains, standardized audio content features are obtained, which reflect the changing rules of the audio signal in time and frequency, and are helpful for the training of the scene recognition model. The standardized operation feature sequence, standardized video content features, and standardized audio content features are aligned in the time dimension to construct a complete multimodal dataset.

[0024] In a specific embodiment, the process of executing step S2 may specifically include the following steps: The standard multimodal dataset is divided into an operation sequence data subset, a video content data subset and an audio data subset; The time series features of the operation sequence data subset are extracted and processed through the long short-term memory network. The long short-term memory network contains 3 hidden layers, each with 128 neurons, and uses the tanh activation function to obtain the time series features of the operation sequence; Visual features are extracted from a subset of video content data and processed through a residual network. The residual network contains 50 convolutional layers. Each residual block uses the ReLU activation function and batch normalization. The last layer uses global average pooling to obtain the visual features of the video content. Audio feature extraction is performed on the audio data subset and processed through a one-dimensional convolutional neural network. The one-dimensional convolutional neural network contains 4 convolutional layers. Each layer uses the ELU activation function and maximum pooling. The last layer uses global maximum pooling to obtain audio content features. Connecting the operation sequence temporal features, video content visual features and audio content features to form a multimodal feature vector; The multimodal feature vector is weighted by feature importance through the attention mechanism. The attention mechanism includes a two-layer feedforward neural network, which uses the softmax function to calculate the weight of each feature to obtain the weighted multimodal feature. The weighted multimodal features are input into the cross-modal fusion network. The cross-modal fusion network contains 3 fully connected layers. Each layer uses the LeakyReLU activation function and Dropout regularization to obtain preliminary fusion features. An adaptive feature aggregation algorithm is applied to the preliminary fusion features, and the fusion scene feature vector is obtained through dynamic threshold screening and nonlinear transformation.

[0025] Specifically, the time series features are extracted from the subset of operation sequence data. User operation data has strong time dependence, and the long short-term memory network (LSTM) is used to process these operation sequences. Assume that the operation sequence is ,in Indicates at time Operation, The operation sequence is input into an LSTM network with 3 hidden layers, each layer contains 128 neurons, and uses the tanh activation function. LSTM captures long-term dependencies in the time series and outputs the feature representation of each time step. By formula ,in is the weight matrix, is the bias term, is the activation function, and the hidden state of each time step is obtained , and obtain the overall operation sequence temporal features. Visual feature extraction is performed on the subset of video content data. Video data contains rich scene and object information, and the residual network (ResNet) is used to extract visual features. Assume that the input video frame is , processed by 50 convolutional layers in the residual network. Each residual block in the residual network contains multiple convolutional layers, uses the ReLU activation function and performs batch normalization. Through batch normalization, the features remain stable during training, thereby accelerating the convergence of the network. In the last layer of the residual network, a global average pooling operation is used to calculate the average value of each feature map to form a visual feature vector of fixed length. The formula for global average pooling is: ; in, represents the visual features after pooling, and Represent the height and width of the feature map respectively, is the position in the feature map The value of . Through the processing of the residual network, the standardized visual features containing scene and object information are extracted to reflect the core visual information in the video content. Audio feature extraction is performed on the audio data subset. Audio signals usually contain information such as language, background music, and environmental sound effects. A one-dimensional convolutional neural network (1D-CNN) is used to process audio data. Assume that the input audio signal is ,in Indicates time The audio amplitude. The one-dimensional convolutional network contains 4 convolutional layers, each of which uses the ELU activation function, which can produce a small negative output in the negative part, thereby enhancing the expressiveness of the feature. At the same time, a maximum pooling operation is performed after each convolution layer to reduce the feature dimension and reduce the computational complexity. In the last layer, the global maximum pooling is used to compress the feature into a fixed-length feature vector. The formula is: ; in, represents the audio features after pooling, Indicates the maximum value in the time dimension. In this way, the standardized audio features are obtained, which effectively reflect the main information in the audio signal. The operation sequence timing features, video content visual features and audio content features are connected to form a complete multimodal feature vector. Let the operation feature be , the visual features are , the audio features are , then the multimodal feature vector Expressed as , so that the model can use operation, visual and audio information for scene recognition at the same time. In order to highlight the importance of each modal feature, the multimodal feature vector is weighted by the attention mechanism. The attention mechanism contains a two-layer feedforward neural network, which calculates the weight of each feature through the softmax function, thereby weighting the features of different modalities. Assume that the attention weight is , then the weighted multimodal feature vector is expressed as: ; in, Indicates The characteristics of a mode, is the weight calculated by softmax, . In this way. The model can pay more attention to the modal features that are most important for the current scene recognition. The weighted multimodal features are input into the cross-modal fusion network for processing. The cross-modal fusion network contains 3 fully connected layers, each layer uses the LeakyReLU activation function, and uses Dropout regularization to prevent overfitting. The LeakyReLU activation function effectively solves the "dead neuron" problem, while the Dropout regularization enhances the generalization ability of the network by randomly discarding a part of the neurons. Preliminary fusion features are obtained through the processing of the cross-modal fusion network. An adaptive feature aggregation algorithm is applied to the preliminary fusion features to optimize the feature representation. The adaptive feature aggregation algorithm selects the most important features and performs nonlinear combinations of features through dynamic threshold screening and nonlinear transformation. Assume that the preliminary fusion features are , the process of dynamic threshold screening is represented by selecting The characteristics of is the dynamic threshold. Then the final fusion scene feature vector is obtained through nonlinear transformation .

[0026] In a specific embodiment, the process of executing step S3 may specifically include the following steps: The fused scene feature vector is input into the encoder network of the variational autoencoder. The encoder network contains three fully connected layers, each of which uses the LeakyReLU activation function to obtain the intermediate feature representation. The intermediate feature representation is processed through two parallel fully connected layers to obtain the mean vector and log variance vector respectively. Based on the mean vector and log variance vector, a reparameterization technique is used to generate latent variable samples. The reparameterization process includes sampling noise from a standard normal distribution and combining it with the mean and variance. Input the latent variable sample into the decoder network of the variational autoencoder. The decoder network contains 3 fully connected layers, each layer uses the ReLU activation function to obtain the reconstructed feature vector; Calculate the reconstruction error between the reconstructed feature vector and the fused scene feature vector, and calculate the KL divergence of the latent variables to obtain the overall loss function value; Based on the overall loss function value, an adjusted latent variable sample is generated. The adjusted latent variable sample is subjected to a spherical projection operation to map the points in the high-dimensional space onto the unit hypersphere to obtain normalized semantic features. The normalized semantic features are transformed in feature space to obtain a compressed semantic feature vector.

[0027] Specifically, the fused scene feature vector is input into the encoder network consisting of 3 fully connected layers, and each layer uses the LeakyReLU activation function. The LeakyReLU activation function can effectively solve the problem of dead neurons and enhance the learning ability and stability of the network by retaining a small non-zero gradient for negative inputs. After layer-by-layer processing of the encoder network, the intermediate feature representation is obtained. The intermediate feature representation is processed to generate the mean vector and log variance vector required by the variational autoencoder. The intermediate feature representation is input into two parallel fully connected layers respectively. The first layer outputs the mean vector and the second layer outputs the log variance vector. The mean vector represents the center position of the latent variable, while the log variance vector is used to calculate the degree of diffusion of the latent variable. Assume that the intermediate feature representation is , after two parallel fully connected layers, the mean vectors are obtained respectively and the logarithmic variance vector In order to generate latent variable samples, the reparameterization technique is used to maintain the differentiability of the model by introducing randomness and achieve end-to-end training. The reparameterization process is expressed as: ; in, represents the standard deviation, represents a noise vector sampled from a standard normal distribution. By combining the mean and random noise, a latent variable sample is generated. . Input the latent variable samples into the decoder network of the variational autoencoder for reconstruction. The decoder network also contains 3 fully connected layers, each of which uses the ReLU activation function. The ReLU activation function can maintain a large gradient when the output is a positive number, making it easier for the model to converge. After being processed by the decoder network, the reconstructed feature vector is obtained. In order to measure the reconstruction effect, the reconstruction error between the reconstructed feature vector and the fused scene feature vector is calculated, using the mean square error, and the formula is: ; in, The first elements, represents the corresponding reconstruction feature, is the length of the feature vector. The reconstruction error reflects the gap between the reconstructed features and the original features. The smaller the error, the more accurate the decoder is in reconstructing the original features. At the same time, the KL divergence (Kullback-Leibler divergence) of the latent variables is calculated to measure the difference between the generated latent distribution and the standard normal distribution. The calculation formula of KL divergence is: ; in, and Respectively represent The mean and variance of the latent variables, represents the dimension of the latent space. The KL divergence term is used to encourage the distribution of latent variables to be close to the standard normal distribution, so that the latent space has better structured properties. By combining the reconstruction error and KL divergence, the overall loss function value is obtained: ; in, Is a weight parameter used to control the trade-off between reconstruction error and KL divergence. During the training process, by minimizing the overall loss function and optimizing the parameters of the encoder and decoder, the reconstructed features are close to the original features while ensuring the continuity and regularization effect of the latent space. Generate adjusted latent variable samples based on the overall loss function value. The adjusted latent variable samples are projected onto the unit hypersphere through a spherical projection operation. The purpose of spherical projection is to normalize the latent variable samples to ensure that their distribution in the high-dimensional space is more uniform and orderly. Assume that the adjusted latent variable samples are , then the sample after spherical projection is expressed as: ; in, Representation vector The Euclidean norm of the latent variable samples is projected onto the unit hypersphere to ensure that the modulus of all samples is 1, avoiding the impact of numerical amplitude differences on subsequent feature processing. The normalized semantic features are transformed into feature space to obtain a compressed semantic feature vector. The latent variables are compressed and optimized through linear or nonlinear transformations, so that the final feature vector has higher discrimination and more compact representation capabilities. For example, through the linear projection matrix Mapping the normalized latent variables to a lower dimensional space is expressed as: ; in, is the transformation matrix, Represents the compressed semantic feature vector. In this way, the final compressed semantic feature vector is obtained.

[0028] In a specific embodiment, the execution step processes the intermediate feature representation through two parallel fully connected layers to obtain a mean vector and a logarithmic variance vector respectively, and based on the mean vector and the logarithmic variance vector, a reparameterization technique is used to generate a latent variable sample, and the reparameterization process includes sampling noise from a standard normal distribution and combining it with the mean and variance, which can specifically include the following steps: The intermediate feature representation is input into the first fully connected layer. The first fully connected layer uses a linear activation function to obtain the original mean vector, and batch normalization is applied to the original mean vector to standardize the feature distribution and obtain a standardized mean vector. The intermediate feature representation is input into the second fully connected layer. The second fully connected layer uses the Softplus activation function to obtain the original logarithmic variance vector, and applies an exponential function transformation to the original logarithmic variance vector to convert the logarithmic variance into variance to obtain the variance vector. Based on the dimension of the standardized mean vector, a random noise vector with the same dimension is generated, the random noise vector obeys the standard normal distribution, and the random noise vector is multiplied by the square root of the variance vector to obtain a scaled noise vector; The scaled noise vector is added to the standardized mean vector to obtain the initial latent variable sample, and the spherical projection function is applied to the initial latent variable sample to project the sample points onto the unit hypersphere to obtain the latent variable sample.

[0029] Specifically, the intermediate feature representation is input into the first fully connected layer to obtain the original mean vector. Assume that the intermediate feature representation is , represents the main features extracted by the encoder network of the variational autoencoder. Input to the first fully connected layer, which uses a linear activation function and outputs the mean vector after linear combination of features, denoted as The original mean vector Batch normalization is applied to features with certain offset and imbalanced distribution. The feature distribution is standardized so that the mean is 0 and the variance is 1, which improves the stability and convergence speed of model training. The formula for batch normalization is: ; in, represents the normalized mean vector elements, and Var denote the mean and variance respectively, is a small positive value used to avoid the situation where the denominator is zero. After batch normalization, the standardized mean vector is obtained At the same time, the intermediate features are represented as Input to the second fully connected layer to obtain the original logarithmic variance vector. In order to ensure that the variance is always non-negative, the fully connected layer uses the Softplus activation function. Softplus is a smoothed ReLU variant defined as: ; in, represents the intermediate feature representation elements, Represents the elements of the corresponding logarithmic variance vector. The value obtained after Softplus activation is considered to be the logarithmic variance. Apply the exponential function to convert it to the actual variance value: ; in, The standard deviation of the variance is represented by the exponential function, which is used to convert the logarithmic variance into the actual variance value. After processing, the obtained variance vector is used to describe the uncertainty of the feature and help construct the latent variable. Based on the standardized mean vector and variance vector Generate a random noise vector of the same dimension as , where the noise vector follows the standard normal distribution, that is, In order to make the size of the noise consistent with the feature distribution, the random noise vector is multiplied element by element by the square root of the variance vector to scale the noise and obtain the scaled noise vector: ; in, represents the scaled noise vector, represents random noise sampled from a standard normal distribution, is the standard deviation of the variance. This process ensures that the generated latent variable samples have a certain degree of randomness in distribution, while being controlled by the mean and variance, making the sample distribution reasonable. Add together to get the initial latent variable sample ; in, represents the initial latent variable sample, and Represent the standardized mean vector and the scaled noise vector respectively. The generated latent variable samples contain the main information of the intermediate features, and also introduce a certain degree of randomness, which helps the model learn the distribution structure of the latent space. Apply the spherical projection function to map the points in the high-dimensional space onto the unit hypersphere to ensure that the modulus of the latent variable samples is 1. The formula for spherical projection is expressed as: ; in, represents the latent variable sample after spherical projection, For vector By projecting the initial latent variable samples onto the unit hypersphere, we ensure that all samples have the same modulus length, improve the uniformity of the latent space, and reduce the impact of numerical differences on subsequent analysis to obtain latent variable samples.

[0030] In a specific embodiment, the process of executing step S4 may specifically include the following steps: Performing text encoding on predefined scene categories to obtain an initial scene query vector set, and performing scene feature enhancement on the initial scene query vector set to obtain an enhanced scene query vector set; Performing dimension alignment operation on the compressed semantic feature vector and the enhanced scene query vector set to obtain aligned feature vectors, and calculating the cosine similarity between the aligned feature vectors to obtain an initial similarity matrix; The initial similarity matrix is ​​input into the multi-head attention mechanism for processing to obtain the attention-weighted similarity matrix, and the attention-weighted similarity matrix is ​​nonlinearly transformed to obtain the scene matching probability distribution; Based on the scene matching probability distribution, K scene matching results with the highest probability are selected, and the selected K scene matching results are weighted summed to obtain the scene matching score.

[0031] Specifically, the predefined scene categories are encoded in text, and the text description of each scene category is mapped into a high-dimensional vector through a natural language processing model, such as a pre-trained BERT model or other semantic embedding model, to obtain an initial scene query vector set. Assume that the initial scene query vector set is ,in is the number of scene categories, each vector is a high-dimensional vector representation of the corresponding scene category. The initial scene query vector set is enhanced with scene features to obtain an enhanced scene query vector set. This is achieved through a self-attention mechanism or by adding additional contextual information. For example, for each scene query vector Perform self-attention calculation to obtain the enhanced query vector This process is expressed as: ; in, Represents the attention weight, by calculating the scene With scene The attention weight is calculated by the softmax function to ensure that the sum of all weights is 1. The processed enhanced scene query vector set is The enhanced scene query vector set is dimensionally aligned with the compressed semantic feature vector so as to perform similarity calculation in the same feature space. Assume that the compressed semantic feature vector is , in order to align with the enhanced scene query vector set, z is projected to the same dimensional space as the query vector set through a linear transformation. Assume that the linear transformation matrix is , then the aligned feature vector is expressed as: ; in, is the representation of the compressed semantic feature vector in the aligned space, is a linear transformation matrix used to match dimensions. By aligning the dimensions, we ensure that the compressed semantic features and the enhanced scene query vector are calculated in the same space for similarity. The cosine similarity is calculated between the aligned feature vectors to obtain the initial similarity matrix. Cosine similarity is used to measure the similarity between two vectors, and its value varies between -1 and 1, where 1 means that the two vectors are completely similar, 0 means they are irrelevant, and -1 means they are completely opposite. The formula for calculating cosine similarity is: ; in, Represents the compressed semantic feature vector and the enhanced scene query vector The cosine similarity between and Respectively represent the Euclidean norm of the vector. The initial similarity matrix is ​​calculated in this way , where each element Represents the similarity between the semantic feature and the scene query vector. The initial similarity matrix is ​​input into the multi-head attention mechanism for processing. The multi-head attention mechanism captures the relationship between different features through multiple sets of independent attention heads. Each attention head focuses on a different feature subspace, thereby learning the dependency between features in multiple subspaces. Attention heads are added to process the initial similarity matrix through the multi-head attention mechanism to obtain the attention-weighted similarity matrix. This process is expressed as: ; in, is the attention-weighted similarity matrix, and MultiHeadAttention represents the operation of the multi-head attention mechanism. In this way, the important relationships in the similarity matrix are enhanced, and the most important parts for scene matching are highlighted. The attention-weighted similarity matrix is ​​nonlinearly transformed to obtain the scene matching probability distribution. The nonlinear transformation is implemented through the softmax function to ensure that the matching probability of each scene is between 0 and 1, and the sum of all probabilities is 1. The calculation formula of the softmax function is: ; in, Representation scene The matching probability, Represents the similarity matrix after attention weighting The similarity value of the scene. Through the softmax function, the scene matching probability distribution is obtained It can intuitively represent the matching degree between each scene category and the current semantic feature. Based on the scene matching probability distribution, the previous The scene matching results with the highest probability are selected to ensure that the selected scene category is most relevant to the current semantic features. The weighted sum of the scene matching results is used to obtain the final scene matching score. The scene matching results are , then the scene matching score is expressed as: ; Among them, Score is the scene matching score, Indicates The matching probability of each scene, Represents the corresponding similarity value. By taking a weighted sum of the matching probability and similarity, we get a scene matching score that comprehensively considers similarity and matching probability.

[0032] In a specific embodiment, the process of executing step S5 may specifically include the following steps: Normalizing the scene matching score to obtain a normalized scene matching score, and constructing a feature vector based on the normalized scene matching score; The feature vector is input into a random forest classifier, which contains 100 decision trees, and the maximum depth of each tree is 10, to obtain the first sub-classification result; The feature vector is input into the support vector machine classifier, the support vector machine classifier uses a radial basis function kernel, and the penalty parameter C is set to 1.0 to obtain the second sub-classification result; Apply the gradient boosting decision tree algorithm to the feature vector. The gradient boosting decision tree algorithm contains 200 weak learners and the learning rate is set to 0.1 to obtain the third sub-classification result. The first sub-classification result, the second sub-classification result and the third sub-classification result are encoded to obtain an encoded sub-classification result set, and the Dempster-Shafer evidence theory is applied to the encoded sub-classification result set to calculate the confidence interval of each category to obtain a fused evidence matrix; Based on the fused evidence matrix, the optimal category is selected to obtain the preliminary scene category, and the preliminary scene category is Markov smoothed to obtain the target scene classification result.

[0033] Specifically, the scene matching score is normalized to obtain a normalized scene matching score. A feature vector is constructed based on the normalized scene matching score. The feature vector is input into a random forest classifier. Random forest is an ensemble learning method that improves the robustness and generalization ability of the model by integrating multiple decision trees. A random forest classifier containing 100 decision trees is used, and the maximum depth of each tree is 10 to prevent overfitting and ensure that the complexity of the model is moderate. The first sub-classification result is obtained through the random forest classifier. Assume that the feature vector The output category is ,in , represents a category in the scene category set. The feature vector is input into the support vector machine (SVM) classifier for classification. Support vector machine is an algorithm applied to classification tasks, which can effectively find the optimal hyperplane in the feature space to maximize the interval between different categories. The radial basis function kernel (RBF kernel) is used as the kernel function of SVM, and the penalty parameter Set to 1.0 to control the model's tolerance to errors. After passing through the SVM classifier, the second sub-classification result is obtained, which is recorded as ,same Also belongs to the scene category collection . In order to improve the accuracy of classification, the gradient boosted decision tree (GBDT) algorithm is applied to the feature vector for classification. The gradient boosted decision tree is an ensemble learning method that gradually adds weak learners to improve the overall model performance. 200 weak learners are used and the learning rate is set to 0.1 to ensure that the update step of each learner is small, thereby reducing the risk of overfitting. Feature vector After inputting into GBDT, the third sub-classification result is obtained, which is recorded as . For the first sub-category results , Second sub-category results And the third sub-category results Encode and obtain the encoded sub-classification result set. Assume that the encoded result set is , where each sub-classification result corresponds to the encoding of the scene category. In order to integrate the outputs of different sub-classifiers, the Dempster-Shafer evidence theory is used to calculate the confidence interval of each category and obtain the fused evidence matrix. The Dempster-Shafer evidence theory is a method for dealing with uncertainty and fusing multiple information sources. By combining the evidence of multiple classifiers, the probability of each category can be better estimated. Let the confidence interval be , indicating the category The reliability of , then the fused evidence matrix is ​​expressed as: ; in, Indicates category The reliability of Indicates Classifiers for categories By integrating the support of all sub-classifiers, the comprehensive reliability of each category is obtained. After obtaining the fused evidence matrix, the category with the highest reliability is selected as the preliminary scene category, denoted as The selection of the preliminary scene category is based on the maximum value of the confidence interval, which means that the most likely category is selected as the current scene prediction result. Markov smoothing is performed on the preliminary scene category. Markov smoothing is a time series-based method that smoothes the current prediction result by considering the category status of the previous and next moments to reduce the instability of the classification. Assume that the preliminary scene category at the current moment is , the scene category at the previous moment is , then the target scene classification result after Markov smoothing is expressed as: ; in, represents the probability of the smoothed scene category at the current moment, is a smoothing parameter used to control the balance between the current prediction result and the state at the previous moment. In this way, the continuity and rationality of the target scene classification results in the time dimension can be ensured, thereby improving the system's responsiveness to changes in user scenarios.

[0034] The above describes the intelligent scene recognition method of the intelligent interaction control unit in the embodiment of the present invention. The following describes the intelligent scene recognition device of the intelligent interaction control unit in the embodiment of the present invention. Figure 2 , an embodiment of the intelligent scene recognition device of the intelligent interactive control unit in the embodiment of the present invention includes: The acquisition module is used to collect and preprocess the operation sequence, video content and audio data of the intelligent interactive control unit in the digital set-top box to obtain a standard multimodal data set; The extraction module is used to input the standard multimodal dataset into the multi-branch deep neural network model, extract the temporal, visual and audio features through parallel processing, and obtain the fused scene feature vector; A mapping module, used to perform dimension reduction and semantic mapping on the fusion scene feature vector using a variational autoencoder to obtain a compressed semantic feature vector; The calculation module is used to calculate the similarity between the compressed semantic feature vector and the predefined scene query vector, and obtain the scene matching score through the attention mechanism; The output module is used to obtain the target scene classification result by integrating the outputs of multiple sub-classifiers based on the scene matching score using an ensemble learning method.

[0035] Through the synergy of the above components, by simultaneously processing the operation sequence, video content and audio data, this method can fully capture the characteristics of the user's usage scenario and improve the accuracy and robustness of scene recognition. The multi-branch deep neural network model can process data of different modalities in parallel, effectively extract temporal, visual and audio features, and improve the efficiency and quality of feature extraction. The use of variational autoencoders to reduce the dimension and semantic map the fusion features not only reduces the data dimension, but also captures potential semantic information, which helps to improve the accuracy of subsequent scene recognition. Introducing the attention mechanism in the similarity calculation process can adaptively focus on important features and improve the accuracy of scene matching. The use of multiple sub-classifiers combined with the Dempster-Shafer evidence theory can combine the advantages of different classifiers and improve the generalization ability and robustness of the model. Through the lightweight optimization of the task flow graph, this method can run efficiently on resource-constrained digital set-top boxes and realize real-time scene recognition.

[0036] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0037] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0038] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0039] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0040] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0041] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0042] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent scene recognition method for an intelligent interactive control unit, characterized in that: The method comprises: The operation sequence, video content and audio data of the intelligent interactive control unit in the digital set-top box are collected and preprocessed to obtain a standard multimodal data set; Inputting the standard multimodal dataset into a multi-branch deep neural network model, extracting temporal, visual and audio features through parallel processing, and obtaining a fused scene feature vector; Using a variational autoencoder to perform dimension reduction and semantic mapping on the fusion scene feature vector to obtain a compressed semantic feature vector; Calculate the similarity between the compressed semantic feature vector and the predefined scene query vector, and obtain a scene matching score through an attention mechanism; Based on the scene matching score, an ensemble learning method is used to integrate the outputs of multiple sub-classifiers to obtain a target scene classification result.

2. The intelligent scene recognition method of the intelligent interactive control unit according to claim 1, characterized in that: The operation sequence, video content and audio data of the intelligent interactive control unit in the digital set-top box are collected and preprocessed to obtain a standard multimodal data set, including: The remote control button operation of the intelligent interactive control unit is captured in real time to obtain original operation sequence data, and the original operation sequence data is segmented by a sliding time window to obtain a standardized operation feature sequence; Collecting an original video stream from a video output port of a digital set-top box, and performing multi-scale image segmentation on the original video stream to obtain standardized video content features; Collecting an original audio stream from an audio output port of a digital set-top box, and performing a time-frequency domain joint analysis on the original audio stream to obtain standardized audio content features; The standardized operation feature sequence, the standardized video content feature and the standardized audio content feature are aligned in a time dimension to obtain a standard multimodal data set.

3. The intelligent scene recognition method of the intelligent interactive control unit according to claim 1, characterized in that: The method of inputting the standard multimodal data set into a multi-branch deep neural network model, extracting time series, visual and audio features through parallel processing, and obtaining a fused scene feature vector comprises: Dividing the standard multimodal data set into an operation sequence data subset, a video content data subset and an audio data subset; Extracting time series features from the operation sequence data subset, processing through a long short-term memory network, wherein the long short-term memory network comprises three hidden layers, each layer has 128 neurons, and uses a tanh activation function to obtain time series features of the operation sequence; Extract visual features from the subset of video content data and process them through a residual network, wherein the residual network includes 50 convolutional layers, each residual block uses a ReLU activation function and batch normalization, and the last layer uses global average pooling to obtain visual features of the video content; Performing audio feature extraction on the audio data subset, and processing it through a one-dimensional convolutional neural network, wherein the one-dimensional convolutional neural network comprises four convolutional layers, each layer uses an ELU activation function and maximum pooling, and the last layer uses a global maximum pooling to obtain audio content features; Connecting the operation sequence temporal features, the video content visual features and the audio content features to form a multimodal feature vector; The multimodal feature vector is weighted by feature importance through an attention mechanism, wherein the attention mechanism includes a two-layer feedforward neural network, and a softmax function is used to calculate the weight of each feature to obtain a weighted multimodal feature; Inputting the weighted multimodal features into a cross-modal fusion network, wherein the cross-modal fusion network comprises three fully connected layers, each layer using a LeakyReLU activation function and Dropout regularization, to obtain preliminary fusion features; An adaptive feature aggregation algorithm is applied to the preliminary fusion features to obtain a fusion scene feature vector through dynamic threshold screening and nonlinear transformation.

4. The intelligent scene recognition method of the intelligent interactive control unit according to claim 1, characterized in that: The adopting of a variational autoencoder to perform dimension reduction and semantic mapping on the fusion scene feature vector to obtain a compressed semantic feature vector includes: Inputting the fused scene feature vector into an encoder network of a variational autoencoder, wherein the encoder network comprises three fully connected layers, each layer using a LeakyReLU activation function, to obtain an intermediate feature representation; Processing the intermediate feature representation through two parallel fully connected layers to obtain a mean vector and a logarithmic variance vector respectively, and generating a latent variable sample based on the mean vector and the logarithmic variance vector using a reparameterization technique, wherein the reparameterization process includes sampling noise from a standard normal distribution and combining it with the mean and variance; Input the latent variable sample into the decoder network of the variational autoencoder, wherein the decoder network comprises three fully connected layers, each layer uses a ReLU activation function, and obtains a reconstructed feature vector; Calculating a reconstruction error between the reconstructed feature vector and the fused scene feature vector, and calculating the KL divergence of the latent variables to obtain an overall loss function value; Based on the overall loss function value, an adjusted latent variable sample is generated, and the adjusted latent variable sample is subjected to a spherical projection operation to map points in a high-dimensional space onto a unit hypersphere to obtain normalized semantic features, and the normalized semantic features are transformed in feature space to obtain a compressed semantic feature vector.

5. The intelligent scene recognition method of the intelligent interactive control unit according to claim 4, characterized in that: The intermediate feature representation is processed through two parallel fully connected layers to obtain a mean vector and a logarithmic variance vector respectively, and based on the mean vector and the logarithmic variance vector, a latent variable sample is generated using a reparameterization technique, wherein the reparameterization process includes sampling noise from a standard normal distribution and combining it with the mean and variance, including: Inputting the intermediate feature representation into a first fully connected layer, the first fully connected layer uses a linear activation function to obtain an original mean vector, and applying batch normalization processing to the original mean vector to standardize the feature distribution to obtain a standardized mean vector; Input the intermediate feature representation into a second fully connected layer, the second fully connected layer uses a Softplus activation function to obtain an original logarithmic variance vector, and applies an exponential function transformation to the original logarithmic variance vector to convert the logarithmic variance into a variance to obtain a variance vector; Based on the dimension of the standardized mean vector, generate a random noise vector with the same dimension, the random noise vector obeys a standard normal distribution, and multiply the random noise vector by the square root of the variance vector to obtain a scaled noise vector; The scaled noise vector is added to the standardized mean vector to obtain an initial latent variable sample, and a spherical projection function is applied to the initial latent variable sample to project the sample points onto a unit hypersphere to obtain a latent variable sample.

6. The intelligent scene recognition method of the intelligent interactive control unit according to claim 1, characterized in that: The similarity calculation between the compressed semantic feature vector and the predefined scene query vector is performed, and a scene matching score is obtained through an attention mechanism, including: Performing text encoding on predefined scene categories to obtain an initial scene query vector set, and performing scene feature enhancement on the initial scene query vector set to obtain an enhanced scene query vector set; Performing a dimension alignment operation on the compressed semantic feature vector and the enhanced scene query vector set to obtain aligned feature vectors, and calculating cosine similarities between the aligned feature vectors to obtain an initial similarity matrix; Inputting the initial similarity matrix into the multi-head attention mechanism for processing to obtain an attention-weighted similarity matrix, and performing a nonlinear transformation on the attention-weighted similarity matrix to obtain a scene matching probability distribution; Based on the scene matching probability distribution, K scene matching results with the highest probabilities are selected, and a weighted sum is performed on the selected K scene matching results to obtain a scene matching score.

7. The intelligent scene recognition method of the intelligent interactive control unit according to claim 1, characterized in that: Based on the scene matching score, an ensemble learning method is used to integrate the outputs of multiple sub-classifiers to obtain a target scene classification result, including: Normalizing the scene matching score to obtain a normalized scene matching score, and constructing a feature vector based on the normalized scene matching score; Inputting the feature vector into a random forest classifier, wherein the random forest classifier includes 100 decision trees, and the maximum depth of each tree is 10, to obtain a first sub-classification result; Inputting the feature vector into a support vector machine classifier, wherein the support vector machine classifier uses a radial basis function kernel and a penalty parameter C is set to 1.0, to obtain a second sub-classification result; Applying a gradient boosting decision tree algorithm to the feature vector, wherein the gradient boosting decision tree algorithm includes 200 weak learners and the learning rate is set to 0.1, to obtain a third sub-classification result; Encoding the first sub-classification result, the second sub-classification result, and the third sub-classification result to obtain an encoded sub-classification result set, and applying the Dempster-Shafer evidence theory to the encoded sub-classification result set to calculate the confidence interval of each category to obtain a fused evidence matrix; Based on the fused evidence matrix, the optimal category is selected to obtain a preliminary scene category, and Markov smoothing is performed on the preliminary scene category to obtain a target scene classification result.

8. An intelligent scene recognition device for an intelligent interactive control unit, characterized in that: The device is used to execute the intelligent scene recognition method of the intelligent interactive control unit according to any one of claims 1 to 7, the device comprising: The acquisition module is used to collect and preprocess the operation sequence, video content and audio data of the intelligent interactive control unit in the digital set-top box to obtain a standard multimodal data set; An extraction module, used for inputting the standard multimodal data set into a multi-branch deep neural network model, extracting time series, visual and audio features through parallel processing, and obtaining a fused scene feature vector; A mapping module, used for performing dimension reduction and semantic mapping on the fusion scene feature vector by using a variational autoencoder to obtain a compressed semantic feature vector; A calculation module, used for calculating the similarity between the compressed semantic feature vector and the predefined scene query vector, and obtaining a scene matching score through an attention mechanism; The output module is used to obtain a target scene classification result by integrating the outputs of multiple sub-classifiers using an ensemble learning method based on the scene matching score.

9. A computer device, characterized in that: It comprises a memory and a processor, the memory stores a computer program that can be run on the processor, and is characterized in that when the processor executes the computer program, the intelligent scene recognition method of the intelligent interactive control unit described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor is enabled to execute the intelligent scene recognition method of the intelligent interactive control unit according to any one of claims 1 to 7.

Images