Television box intelligent interaction control unit image processing method and equipment

Abstract

The invention relates to a television box intelligent interaction control unit image processing method and device. The method comprises the following steps: carrying out feature analysis on an input video stream of the television box to obtain an enhanced video stream; the method comprises the following steps: collecting visual and voice interaction information of a user based on an intelligent interaction control unit, and inputting the visual and voice interaction information into a double-branch Transform model for multi-modal information fusion to obtain a fusion feature vector; performing conjoint analysis to obtain a user interaction intention and an interaction response strategy; performing local contrast enhancement and motion compensation to obtain a high-quality video frame sequence; and performing scene adaptive color space mapping and dynamic color temperature adjustment on the high-quality video frame sequence according to the environment light information acquired in real time and the pre-stored color characteristics of the display device to obtain an output video stream. According to the implementation of the method, limited computing resources can be fully utilized, efficient video processing and natural man-machine interaction are achieved, and meanwhile the display effect is dynamically optimized according to environmental changes.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method and device for an intelligent interactive control unit of a TV box. Background Art

[0002] With the rapid development of smart home technology, TV boxes, as an important device for home entertainment centers, are gaining more and more attention from users for their intelligent interaction and image processing capabilities. Traditional TV boxes have obvious deficiencies in processing high dynamic range videos, user interactions in complex scenes, and adapting to different ambient light conditions. Existing technologies often use a single-modal interaction method, which makes it difficult to accurately understand the user's complex intentions, resulting in a poor interactive experience. At the same time, image processing algorithms have high computational complexity when processing high-resolution, high-frame-rate videos, making it difficult to run in real time on resource-constrained embedded devices.

[0003] In addition, most of the current image enhancement technologies for TV boxes use global processing methods, which makes it difficult to adaptively optimize image content in different areas, resulting in unsatisfactory processing results. The consistency of display effects under different lighting environments is also an issue that needs to be addressed. Traditional methods cannot dynamically adjust display parameters according to real-time ambient light changes, resulting in overexposure or underexposure of images in certain scenes, affecting the user's viewing experience. Summary of the invention

[0004] The main purpose of the present invention is to provide an image processing method and device for an intelligent interactive control unit of a TV box. The present invention can make full use of limited computing resources to achieve efficient video processing and natural human-computer interaction, while dynamically optimizing the display effect according to environmental changes.

[0005] To achieve the above object, the present invention provides an image processing method for an intelligent interactive control unit of a TV box, comprising the following steps: Perform feature analysis on the input video stream of the TV box to obtain the video feature data structure; According to the video feature data structure, the input video stream is divided into overlapping image blocks, independently processed and weightedly reorganized to obtain an enhanced video stream; Based on the intelligent interaction control unit, the user's visual and voice interaction information is collected, and the visual and voice interaction information is input into the dual-branch Transformer model for multimodal information fusion to obtain a fused feature vector; Performing joint analysis on the fused feature vector and the enhanced video stream to obtain a user interaction intention and an interaction response strategy; Based on the user interaction intention and the interaction response strategy, performing local contrast enhancement and motion compensation on the enhanced video stream to obtain a high-quality video frame sequence; The high-quality video frame sequence is subjected to scene-adaptive color space mapping and dynamic color temperature adjustment according to the ambient light information acquired in real time and the pre-stored color characteristics of the display device to obtain an output video stream.

[0006] The present invention also provides an image processing device for an intelligent interactive control unit of a TV box, comprising: The feature analysis module is used to perform feature analysis on the input video stream of the TV box to obtain a video feature data structure; A processing module, used for dividing the input video stream into overlapping image blocks, independently processing and weighted reorganizing according to the video feature data structure to obtain an enhanced video stream; A fusion module is used to collect the user's visual and voice interaction information based on the intelligent interaction control unit, and input the visual and voice interaction information into a dual-branch Transformer model for multimodal information fusion to obtain a fusion feature vector; A joint analysis module, used for jointly analyzing the fused feature vector and the enhanced video stream to obtain a user interaction intention and an interaction response strategy; A compensation module, configured to perform local contrast enhancement and motion compensation on the enhanced video stream based on the user interaction intention and the interaction response strategy to obtain a high-quality video frame sequence; The adjustment module is used to perform scene-adaptive color space mapping and dynamic color temperature adjustment on the high-quality video frame sequence according to the ambient light information obtained in real time and the pre-stored color characteristics of the display device to obtain an output video stream.

[0007] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0008] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0009] In summary, the technical solution provided by the present invention realizes efficient video enhancement, improves image quality and processing speed by performing feature analysis and overlapping image block processing on the input video stream, and is suitable for resource-constrained embedded devices. The dual-branch Transformer model is used for multimodal information fusion, which effectively integrates visual and voice interaction information and improves the accuracy and robustness of user interaction intention recognition. The user interaction intention recognition and response strategy generation method based on joint analysis realizes a smarter and more natural human-computer interaction experience. Through local contrast enhancement and motion compensation technology, the detail expression and motion smoothness of the video are optimized in a targeted manner, and the visual quality is significantly improved. The introduction of scene-adaptive color space mapping and dynamic color temperature adjustment based on machine learning enables the output video stream to automatically adjust according to changes in ambient light, ensuring a consistent viewing experience under different lighting conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a schematic diagram of the steps of an image processing method of an intelligent interactive control unit of a TV box in one embodiment of the present invention; Figure 2 This is a structural block diagram of an image processing device of an intelligent interactive control unit of a TV box in one embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0011] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0013] Reference Figure 1 This embodiment provides an image processing method for a TV box intelligent interactive control unit, comprising the following steps: S1, perform feature analysis on the input video stream of the TV box to obtain a video feature data structure; The input video stream of the TV box is segmented at the frame level to obtain a video frame sequence. The continuous video signal is decomposed into independent frames so that each frame is analyzed and processed as an independent image. Each frame in the video frame sequence is analyzed in the spatial domain. Spatial domain analysis is a method for directly extracting features from an image in the spatial domain. By calculating the spatial domain of each frame of the image, a series of spatial feature parameters such as resolution, color space, brightness distribution, contrast and saturation are obtained. Among them, resolution reflects the ability to express details of an image, color space involves the representation of colors in an image, brightness distribution describes the lighting conditions of the image, and contrast and saturation directly affect the visual effect of the image. At the same time, adjacent frames in the video frame sequence are differentially operated to obtain an inter-frame difference map, which reveals the changes between video frames, especially the movement of objects in dynamic scenes. Based on the inter-frame difference map, the motion vector is calculated. The motion vector is an important feature for describing motion information in the video, which is used to reflect the motion trajectory and direction of objects in the scene, thereby obtaining motion feature parameters. A two-dimensional discrete Fourier transform is performed on the video frame sequence to convert the video frame from the spatial domain to the frequency domain and extract the frequency domain feature parameters. The two-dimensional discrete Fourier transform converts the image from the spatial domain to the frequency domain, which can effectively separate the low-frequency information and high-frequency information in the image, where the low-frequency information represents the overall structure and contour of the image, while the high-frequency information represents the details and edge features in the image. The spatial feature parameters, motion feature parameters and frequency domain feature parameters are combined to construct a multidimensional feature vector. The multidimensional feature vector is subjected to dimensionality reduction processing. The adaptive principal component analysis method is used to reduce the dimensionality of the multidimensional feature vector, extract the feature components that are most important for describing the video content, and obtain the reduced-dimensional feature vector. High-level semantic features are extracted from the reduced-dimensional feature vector. High-level semantic features are semantic-level descriptions of video content, which are used to express higher-level information in the video, such as scene type, object category, user behavior, etc. High-level semantic features, spatial feature parameters, motion feature parameters and frequency domain feature parameters are weighted and fused. The various features are weighted by the attention mechanism, and their weights are adaptively adjusted according to the importance of the features in different scenes to obtain more accurate feature fusion results. The video feature data structure is obtained through weighted fusion of the attention mechanism.

[0014] S2, dividing the input video stream into overlapping image blocks, independently processing and weighted reorganizing according to the video feature data structure to obtain an enhanced video stream; Specifically, according to the resolution parameter in the video feature data structure, each frame in the input video stream is grid-divided, and the entire image is divided into multiple small areas to obtain NxM basic image blocks. Each basic image block is overlapped and expanded by 50%, which effectively reduces the boundary effect and the gap problem during image splicing, and obtains (2N-1)x(2M-1) overlapping image blocks, and the size of each overlapping image block is twice the size of the original basic image block. The expansion makes the adjacent areas more related, which helps to process the image details more finely and ensure the visual consistency of the overlapping areas. According to the brightness distribution parameters in the video feature data structure, each overlapping image block is subjected to adaptive histogram equalization. Histogram equalization is a method for enhancing the overall contrast of an image. By redistributing the brightness in an adaptive manner, it can effectively improve the brightness and detail performance of the image, so that the originally too dark or too bright areas can present more details. This process is performed on each overlapping image block separately, improving the brightness balance of the video frame as a whole, and obtaining overlapping image blocks after brightness balance. Based on the frequency domain feature parameters in the video feature data structure, wavelet transform is performed on the overlapping image blocks after brightness equalization. Wavelet transform is an analysis method that decomposes an image into different scales and frequency components. The multi-scale features of the image are extracted through wavelet transform, so that different details in the image can be expressed in different frequency sub-bands. For high-frequency sub-bands, which usually contain edge information and detail features in the image, in order to remove noise and retain useful information, soft threshold denoising is performed on the high-frequency sub-bands. By setting a suitable threshold, high-frequency noise is removed to obtain denoised overlapping image blocks. Based on the motion feature parameters in the video feature data structure, motion compensation is performed on the denoised overlapping image blocks. The goal of motion compensation is to reduce image blur caused by object motion. The motion vector is calculated by the block matching algorithm, the corresponding relationship between adjacent frames is found, and the sub-pixel interpolation method is used to accurately perform motion compensation to obtain motion compensated overlapping image blocks. In the overlapping image blocks after motion compensation, the local contrast enhancement algorithm is applied based on the contrast parameters in the video feature data structure to optimize the visual performance of the image. Local contrast enhancement uses an adaptive gamma correction method to adjust the intensity of pixels, so that the dark details in the image are enhanced and the bright parts are not overexposed, resulting in contrast-enhanced overlapping image blocks. Adaptive gamma correction can flexibly adjust the contrast according to the local characteristics of the image, making the image more visually layered and the overall effect more vivid. Weighted overlapping image blocks after contrast enhancement are weighted to ensure seamless fusion between different image blocks in the overlapping area. The weighted overlapping image blocks are weighted averaged at the pixel level in the overlapping area according to the original position to reconstruct a complete video frame. The weighted average method allows the information from different image blocks to transition smoothly in the overlapping area, thereby avoiding splicing marks and making the reconstructed image more natural and smooth visually.After the above processing, all reconstructed video frame sequences are combined to obtain the final enhanced video stream.

[0015] S3, based on the intelligent interaction control unit, collects the user's visual and voice interaction information, and inputs the visual and voice interaction information into the dual-branch Transformer model for multimodal information fusion to obtain a fused feature vector; It should be noted that the collected user visual information is preprocessed based on the intelligent interactive control unit. The preprocessing of the visual information includes steps such as image denoising, normalization, and feature extraction to obtain a visual input sequence. At the same time, the collected user voice information is preprocessed. Voice preprocessing involves operations such as removing background noise and feature extraction (such as MFCC or Mel spectrum) to obtain a voice input sequence. The visual input sequence is input into the visual branch Transformer encoder in the dual-branch Transformer model. The visual branch Transformer encoder contains 6 encoding layers, each of which consists of a multi-head self-attention sublayer and a feedforward neural network sublayer. In the multi-head self-attention sublayer, the visual input sequence obtains global dependencies through the multi-head attention mechanism, which can not only capture local features, but also integrate the overall information of the image into the feature representation. The feedforward neural network sublayer performs nonlinear mapping on the attention results to enhance the expression ability of the features, and finally obtains the visual feature representation through the step-by-step calculation of the 6 encoding layers. The visual feature representation not only contains static information such as the user's facial expression, but also can capture dynamic information such as the user's head movement. At the same time, the speech input sequence is input into the speech branch Transformer encoder in the dual-branch Transformer model. The structure of the speech branch Transformer encoder is the same as that of the visual branch, which contains 6 encoding layers, each of which consists of a multi-head self-attention sublayer and a feedforward neural network sublayer. The multi-head self-attention sublayer is used to globally model the speech signal and capture the time series correlation and contextual information of the speech, and the feedforward neural network sublayer enhances the nonlinear expression ability of the speech features. In this way, after the speech input sequence is processed by the 6 encoding layers, the speech feature representation is finally obtained, which contains key information such as the emotion, intonation, and speaking speed in the user's speech. Position encoding is performed on the visual feature representation and the speech feature representation, and the position information is injected into the feature representation. The purpose of position encoding is to make the Transformer model have sequence perception without explicitly using a recursive structure, so that the model can understand the relative relationship between features at different time steps in the subsequent processing process, and obtain enhanced visual feature representation and enhanced speech feature representation. The enhanced visual feature representation and enhanced speech feature representation are input into the cross-attention layer to realize information interaction between different modalities. The cross-attention layer can effectively capture the correlation between different modalities by calculating the attention weights between vision and speech, realize the mutual understanding and fusion of visual and speech information, and obtain preliminary fusion features. The preliminary fusion features are input into the fusion Transformer encoder in the dual-branch Transformer model. The fusion Transformer encoder contains 4 encoding layers, each of which consists of a multi-head self-attention sublayer, a cross-attention sublayer, and a feedforward neural network sublayer.In the multi-head self-attention sublayer, the fused features obtain the global correlation between the sequence elements through the self-attention mechanism, enhancing the internal consistency of the features. In the cross-attention sublayer, the information between different modalities interacts again, fusing the relevant information of vision and speech to obtain higher-level semantic associations. The feedforward neural network sublayer is used to enhance the nonlinear expression ability of the features, and finally obtains the deep fusion features through the step-by-step calculation of the four encoding layers. The deep fusion features are subjected to global average pooling operation to compress the original sequence features into a vector representation of fixed dimensions. Global average pooling compresses the time dimension information in the sequence features to obtain a fused feature vector.

[0016] S4, jointly analyze the fused feature vector and the enhanced video stream to obtain the user interaction intention and interaction response strategy; Specifically, the enhanced video stream is subjected to spatiotemporal feature extraction, and features containing temporal and spatial information are extracted from the video. At the same time, the temporal relationship between frames in the video and the spatial features of a single frame image are captured to obtain the spatiotemporal feature map of the video. The spatiotemporal feature map contains the dynamic changes of the video and the details of the static scene. The fused feature vector is dimensionally expanded to facilitate fusion with the spatiotemporal feature map. The fused feature vector is expanded by copying and splicing operations so that its spatiotemporal dimension is consistent with the spatiotemporal feature map of the video to obtain an extended fused feature map. The spatiotemporal feature map of the video and the extended fused feature map are feature fused, and a joint feature map is obtained by element-by-element addition. Based on the joint feature map, the spatial attention weight and the channel attention weight are calculated to obtain an attention-enhanced joint feature map. The spatial attention weight is used to emphasize the important spatial areas in the image, so that the model pays more attention to key scenes and objects; while the channel attention weight enhances the expression ability of specific features by allocating different weights between feature channels, and improves the ability to distinguish multimodal information. Through the combination of these two attention mechanisms, the joint feature map is effectively enhanced, so that important information is highlighted and strengthened. Temporal modeling is performed on the joint feature map of attention enhancement to obtain temporal coding features. The input spatiotemporal features are integrated in time series to capture the dynamic change characteristics in the video and interactive information. Through modeling, the evolution of user behavior over time can be better understood. In order to obtain the user's interaction intention, the temporal coding features are input into the multi-task learning head, which contains an intention classification branch and a response generation branch. The intention classification branch classifies the user's interaction intention through a fully connected layer and a softmax function to obtain the probability distribution of the user's interaction intention. The fully connected layer performs a linear transformation on the input features to extract feature representations of different categories, while the softmax function normalizes the feature representations of these categories and finally outputs a probability distribution, which represents the user's interaction intention and its corresponding probability. In this way, the system can describe the user's intention in the form of probability. Based on the probability distribution of the user's interaction intention, the user's interaction intention is determined by a set threshold judgment. When the probability value of a certain intention exceeds the preset threshold, the intention is determined as the actual interaction intention of the current user. After determining the user's interaction intention, the corresponding interaction response strategy template is selected according to the predefined intention-response mapping table. By searching the mapping table, the most appropriate response strategy template can be quickly found to make appropriate responses to users in different situations. The temporal coding features and interactive response strategy templates are input into the strategy refinement network to generate specific interactive response strategy parameters. Based on the temporal coding features and strategy templates, the strategy refinement network concretizes the high-level response strategy into executable parameters, so that the system responds to the user's request according to these parameters, and finally obtains the interactive response strategy, including the adjustment of video content, the change of interface, the playback of audio, etc.

[0017] S5, based on the user's interaction intention and interaction response strategy, performs local contrast enhancement and motion compensation on the enhanced video stream to obtain a high-quality video frame sequence; Among them, according to the user's interactive intention, the corresponding local contrast enhancement parameters and motion compensation parameters are extracted from the preset intention-processing parameter mapping table. The preset intention-processing parameter mapping table stores the optimal image processing parameters corresponding to different user intentions, including parameters related to contrast enhancement and motion compensation, to ensure the best visual effect in different user demand scenarios. The enhanced video stream is frame-level segmented to obtain a video frame sequence. Each frame is regarded as an independent image so that each frame can be further processed. Based on the extracted local contrast enhancement parameters, each frame image is adaptively divided into blocks to obtain an image block set. The image is divided into multiple regions so that different enhancement strategies can be adopted for different regions, and the contrast can be flexibly adjusted according to the scene characteristics to obtain better visual effects. Local statistical features are calculated for each image block in the image block set, and the statistical features include local brightness distribution, contrast, and variance of pixel values. Based on the local statistical features and the extracted local contrast enhancement parameters, an adaptive S-curve mapping function is constructed. The S-curve mapping function is a nonlinear mapping function that significantly improves the local contrast of the image by adjusting the pixel value distribution of the image, making the dark details of the image clearer. The adaptive S-curve mapping function is applied to the corresponding image block to obtain the image block after local contrast enhancement. Since there are overlapping areas between image blocks, in order to avoid visual discontinuity in these areas after enhancement, the overlapping areas of adjacent image blocks are smoothly transitioned. Through smoothing, the splicing traces between different image blocks are effectively eliminated to obtain a video frame sequence after contrast enhancement. Based on the motion compensation parameters, block matching motion estimation is performed on adjacent frames in the video frame sequence to obtain a motion vector field. The motion vector field is important information that describes the direction and speed of movement of objects in the video. The block matching algorithm finds the corresponding motion vector by comparing adjacent frames, so as to accurately capture the motion features in the video. Based on the obtained motion vector field, motion compensation interpolation is performed on the video frame sequence after contrast enhancement to generate a video sequence after frame rate enhancement. Motion compensation interpolation effectively increases the frame rate of the video by generating new intermediate frames between the original frames, making the object movement smoother and more natural and reducing motion blur. In order to improve the visual effect of the video, a time domain filter is applied to the video sequence after frame rate enhancement to obtain a smooth high frame rate video sequence. The temporal filter smoothes the video sequence after the frame rate is increased, reducing the artifacts or noise introduced by interpolation, making the video transition in the temporal dimension smoother and more natural. According to the video enhancement level in the interactive response strategy, the global brightness and saturation of the smoothed high frame rate video sequence are adjusted. The video enhancement level in the interactive response strategy is set according to the personalized needs of the user. According to different user preferences, by adjusting the brightness and saturation of the video frame, a high-quality video frame sequence that meets the user's needs is obtained.

[0018] S6, performing scene adaptive color space mapping and dynamic color temperature adjustment on the high-quality video frame sequence according to the ambient light information obtained in real time and the pre-stored color characteristics of the display device to obtain an output video stream.

[0019] Among them, the color space conversion is performed on the high-quality video frame sequence, and the original RGB color space is converted into the HSV color space to obtain the hue (H), saturation (S) and brightness (V) components. The RGB color space is mainly used for display devices, while the HSV color space is more in line with human perception of color and has more advantages in processing tasks such as color enhancement and color temperature adjustment. Based on the ambient light sensor, the ambient light information is collected in real time. The ambient light information includes two aspects: light intensity and color temperature. The light intensity reflects the brightness of the current environment, while the color temperature describes the hue of the light, usually in Kelvin (K). Through these sensor data, the current environment is classified, for example, the environment is divided into bright, moderate or dim scenes. Different states of ambient light will directly affect the human eye's perception of the display picture. According to the type of environmental scene and the pre-stored color characteristics of the display device, the corresponding color enhancement coefficients α, β and γ are selected from the preset scene-color mapping table. α and β are used for saturation adjustment, while γ is used for brightness adjustment. The scene-color mapping table is pre-set based on the visual perception of the human eye in different scenes and the color characteristics of the display device, and is used to select the optimal color enhancement parameters for each scene. For example, in bright scenes, a larger saturation enhancement coefficient α and a higher brightness adjustment parameter γ are selected; in dim scenes, a lower γ value is selected to prevent overexposure of the picture. The brightness V component in the HSV color space is nonlinearly adjusted to adapt to the current ambient light changes. Use the Gamma correction formula ,in is the original brightness component, It is a brightness adjustment parameter selected from the mapping table according to the current environment. Through Gamma correction, nonlinear enhancement of brightness is achieved to control the overall brightness level of the picture so that the picture can maintain good visibility in different light environments. For the saturation S component, linear adjustment is performed based on the color enhancement coefficients α and β. The adjusted saturation The calculation formula is ,in is the original saturation, is the magnification factor used to adjust the saturation, It is the offset of saturation. Through linear adjustment, the saturation of the picture can be flexibly controlled to make the colors more vivid or softer. According to the color temperature information of the environment, the offset of the hue is calculated. , and adjust the original hue H component. The adjusted hue component By formula Calculated, where is the original hue component, The offset is calculated based on the ambient color temperature. The higher the color temperature, the bluer the light appears, and the lower the color temperature, the redder it appears. By properly offsetting the hue, the picture can maintain a more natural color balance at different color temperatures. , Saturation component and lightness component Combined together, and inversely transformed from HSV to RGB color space to obtain color-enhanced video frames. After inversely transformed back to RGB color space, the video frames are correctly decoded and presented by the display device. In order to ensure that the color depth of the video frame matches the supported bit depth of the display device, the Floyd-Steinberg dithering algorithm is applied to the color-enhanced video frame. The function of the dithering algorithm is to reduce the color banding effect caused by quantization error when reducing the color depth, so that the color looks smoother and more natural. For example, if the display device can only support 8-bit color depth, and the original video frame is 10-bit color depth, the dithering algorithm can reduce the color depth to the bit depth supported by the display device without significantly reducing the visual quality. The output video frame sequence is encoded into a video stream in H.264 format. H.264 is a standard applied to video compression that provides high video quality at a lower bit rate. During the encoding process, the bit rate and key frame interval of the network transmission are set to ensure the transmission quality of the video under different network conditions. For example, in a network environment with limited bandwidth, a lower bit rate and a larger key frame interval are selected to reduce the amount of data transmitted; when bandwidth is sufficient, a higher bit rate and a smaller key frame interval are selected to obtain higher video quality and a better user experience.

[0020] In one example, feature analysis is performed on the input video stream of the TV box to obtain a video feature data structure, including: Perform frame-level segmentation on the input video stream of the TV box to obtain a video frame sequence, and perform spatial domain analysis on each frame in the video frame sequence to obtain spatial feature parameters including resolution, color space, brightness distribution, contrast, and saturation; Performing differential operation on adjacent frames in a video frame sequence to obtain an inter-frame difference map, and calculating motion vectors based on the inter-frame difference map to obtain motion feature parameters; Perform a two-dimensional discrete Fourier transform on the video frame sequence to obtain frequency domain feature parameters, and combine the spatial feature parameters, motion feature parameters and frequency domain feature parameters to construct a multidimensional feature vector; Performing adaptive principal component analysis on the multi-dimensional feature vector to obtain the feature vector after dimension reduction, and extracting the high-level semantic features corresponding to the feature vector after dimension reduction; The high-level semantic features, spatial feature parameters, motion feature parameters and frequency domain feature parameters are weightedly fused using the attention mechanism to obtain the video feature data structure.

[0021] In this example, the input video stream of the TV box is segmented at the frame level to obtain a video frame sequence. The video stream is split into separate frames so that each frame can be analyzed and processed independently. Each frame in the video frame sequence is analyzed in the spatial domain to obtain spatial feature parameters including resolution, color space, brightness distribution, contrast, and saturation. Resolution describes the number of pixels in an image; color space refers to the color representation method in an image, and the common ones are RGB and HSV color spaces; brightness distribution represents the distribution of brightness values ​​in an image, which is an important parameter for measuring the overall light of the picture; contrast reflects the degree of difference between the bright and dark parts of the image; and saturation represents the purity or intensity of the color. By calculating and extracting these parameters, the spatial feature description of each frame is obtained. Adjacent frames in the video frame sequence are differentially operated to obtain an inter-frame difference map. The differential operation can capture the movement of objects in the video. By gradually subtracting the pixels of adjacent frames, the area of ​​change between frames, namely the inter-frame difference map, is obtained. The inter-frame difference map can reflect the position and range of the object's movement in the video, and the motion vector is calculated based on these difference maps. The motion vector is a vector that represents the direction and speed of the object's movement in the video. It is obtained through methods such as block matching algorithm or optical flow method. The motion feature parameters describe the dynamic information in the video. The video frame sequence is analyzed in the frequency domain to extract the frequency features. By performing a two-dimensional discrete Fourier transform (2D-DFT) on the video frame, the video frame is converted from the spatial domain to the frequency domain to obtain the frequency domain feature parameters. The spatial information in the image is decomposed into different frequency components, where the low-frequency part represents the overall structure of the image and the high-frequency part represents the details and edge information of the image. The formula for the two-dimensional discrete Fourier transform is: ; in, represents the amplitude in the frequency domain, represents the pixel value in the spatial domain, and are the width and height of the image, respectively. and is the frequency index. Through Fourier transform, the contribution of different frequency components in the image is analyzed, and the frequency domain feature parameters are extracted to describe the texture and edge features of the image. The spatial feature parameters, motion feature parameters and frequency domain feature parameters are combined to construct a multidimensional feature vector. The multidimensional feature vector is reduced in dimension. The adaptive principal component analysis method is used to reduce the dimension of the multidimensional feature vector. Principal component analysis is a statistical technique for data dimensionality reduction. By linearly transforming the data, the data in the original high-dimensional space is projected onto a low-dimensional space, thereby retaining the most important characteristic components in the data. The dimensionality reduction process of principal component analysis is expressed by the following formula: ; in, represents the feature vector after dimensionality reduction, represents the original multidimensional feature vector, is the principal component matrix. By adaptively selecting the principal components, the principal component analysis method significantly reduces the dimension of the feature vector while ensuring that the main features of the data are not lost, making subsequent calculations more efficient. Extract the high-level semantic features corresponding to the feature vector after dimensionality reduction. High-level semantic features are a higher-level understanding of the video content, such as scene categories, object categories, and user interaction behaviors. High-level semantic features are obtained by clustering analysis of the feature vector after dimensionality reduction or feature extraction of deep neural networks. The high-level semantic features, spatial feature parameters, motion feature parameters, and frequency domain feature parameters are weightedly fused using the attention mechanism. The attention mechanism is a mechanism used to select important information in deep learning. By assigning different weights to each part of the features, the attention mechanism can adaptively adjust the importance of various features according to the needs of the task. The calculation of the attention mechanism is expressed by the following formula: ; in, represents the weighted fusion feature, , and are the matrix representations of query, key, and value, respectively. is the dimension of the feature vector. By calculating the attention weight, the most important features for the task are effectively highlighted, and the video feature data structure is obtained.

[0022] In one example, the input video stream is divided into overlapping image blocks, independently processed, and weighted reorganized according to the video feature data structure to obtain an enhanced video stream, including: According to the resolution parameter in the video feature data structure, each frame in the input video stream is grid-divided to obtain NxM basic image blocks, and the NxM basic image blocks are expanded by 50% overlap to obtain (2N-1)x(2M-1) overlapping image blocks, and the size of each overlapping image block is twice that of the original basic image block; According to the brightness distribution parameters in the video feature data structure, each overlapping image block is subjected to adaptive histogram equalization to obtain the overlapping image blocks after brightness equalization; Based on the frequency domain feature parameters in the video feature data structure, wavelet transform is performed on the overlapping image blocks after brightness equalization to extract multi-scale features, and soft threshold denoising is performed on the high frequency sub-band to obtain the denoised overlapping image blocks; According to the motion feature parameters in the video feature data structure, motion compensation is performed on the denoised overlapping image blocks, motion vectors are calculated by a block matching algorithm, and sub-pixel interpolation is performed to obtain the motion compensated overlapping image blocks; Based on the contrast parameter in the video feature data structure, a local contrast enhancement algorithm is applied to the overlapping image blocks after motion compensation, and the pixel intensity is adjusted through adaptive gamma correction to obtain the overlapping image blocks after contrast enhancement; The contrast-enhanced overlapping image blocks are weighted to obtain weighted overlapping image blocks, and the weighted overlapping image blocks are weighted averaged at the pixel level in the overlapping area according to the original position to reconstruct the complete video frame, and the reconstructed video frame sequence is combined to obtain the enhanced video stream.

[0023] In this example, each frame in the input video stream is gridded according to the resolution parameter in the video feature data structure. The grid is divided into In order to reduce the blocking effect and achieve better information fusion between image blocks, each basic image block is extended by 50% overlap, and the number of overlapped image blocks is , the size of each overlapping image block is twice that of the original basic image block. The overlapping extension method ensures that when performing image enhancement, no obvious visual boundaries will be generated at the boundaries of the image blocks, thereby improving the visual quality of the overall video. The overlapping image blocks are equalized in brightness. Based on the brightness distribution parameters in the video feature data structure, each overlapping image block is subjected to adaptive histogram equalization. Histogram equalization is an algorithm used to improve the brightness distribution of an image. By redistributing the grayscale of the image, the overall brightness of the image is made more uniform and natural. According to the local brightness histogram of each image block, an adaptive method is applied to map the pixel values ​​so that the lower brightness parts of the image block are enhanced and the higher brightness parts are suppressed, thereby obtaining the overlapped image blocks after brightness equalization. Multi-scale features are extracted from the image blocks. Based on the frequency domain feature parameters in the video feature data structure, the overlapped image blocks after brightness equalization are subjected to wavelet transform. Wavelet transform is an analysis method that decomposes an image into components of different scales and frequencies. It can simultaneously retain information in the time domain and frequency domain, and better capture edge and detail features in the image through multi-scale analysis. The formula for wavelet transform is: ; in, represents the wavelet transform coefficients, is the brightness value of the input image, is the wavelet basis function. After completing the wavelet decomposition, the image is decomposed into different frequency subbands, where the high-frequency subband contains the edge information and noise of the image. In order to eliminate the high-frequency noise in the image, the high-frequency subband is subjected to soft threshold denoising. Soft threshold denoising reduces noise by attenuating coefficients less than the threshold while retaining the main edge information of the image to obtain denoised overlapping image blocks. According to the motion feature parameters in the video feature data structure, motion compensation is performed on the denoised overlapping image blocks. Motion compensation reduces the blurring caused by object motion by estimating the motion between adjacent frames. In this process, adjacent frames are matched by a block matching algorithm to calculate motion vectors. Motion vector It is expressed as: ; in, Represents the motion vector of the image block at the current position, is the motion offset, and are the width and height of the image block, respectively. and Represent the pixel values ​​of the current frame and the previous frame respectively. By finding the offset that minimizes the difference between frames, the motion vector of the image block is obtained, and then the motion is compensated using sub-pixel interpolation technology to obtain the overlapping image blocks after motion compensation. This process can significantly reduce the blur caused by object motion, making the dynamic scenes in the video appear smoother and more natural. After completing the motion compensation, the local contrast enhancement algorithm is applied to the overlapping image blocks after motion compensation according to the contrast parameters in the video feature data structure. Local contrast enhancement adjusts the pixel intensity through adaptive gamma correction to make the details of the image more prominent. The adjusted pixel value The calculation is done by the following formula: ; in, is the original pixel value, is an adaptive contrast enhancement parameter. By nonlinearly adjusting pixels, the details of dark and bright areas are enhanced, making the contrast of the image more obvious, thereby improving the visual effect. Weighted processing is performed on the overlapping image blocks after contrast enhancement to fuse the information between the image blocks to obtain weighted overlapping image blocks. In order to ensure the visual consistency between overlapping image blocks during reconstruction, the weighted overlapping image blocks are weighted averaged at the pixel level in the overlapping area according to the original position. Through pixel-level weighted averaging, the splicing marks caused by independent processing of adjacent image blocks in the overlapping area are eliminated to ensure that the reconstructed video frames are visually consistent and smooth. All reconstructed video frame sequences are combined to obtain the final enhanced video stream.

[0024] In one example, the user's visual and voice interaction information is collected based on the intelligent interaction control unit, and the visual and voice interaction information is input into a dual-branch Transformer model for multimodal information fusion to obtain a fused feature vector, including: Preprocessing the collected user visual information based on the intelligent interactive control unit to obtain a visual input sequence, and preprocessing the collected user voice information to obtain a voice input sequence; The visual input sequence is input into the visual branch Transformer encoder in the dual-branch Transformer model. The visual branch Transformer encoder contains 6 encoding layers, each of which includes a multi-head self-attention sublayer and a feedforward neural network sublayer, and the visual feature representation is obtained by calculation; The speech input sequence is input into the speech branch Transformer encoder in the dual-branch Transformer model. The speech branch Transformer encoder contains 6 encoding layers. The structure of each encoding layer is the same as that of the visual branch. The speech feature representation is obtained by calculation. Position encoding is performed on the visual feature representation and the speech feature representation, and the position information is injected into the feature representation to obtain enhanced visual feature representation and enhanced speech feature representation; The enhanced visual feature representation and the enhanced speech feature representation are input into the cross attention layer, and the interaction of visual and speech information is realized by calculating the attention weights between the modalities to obtain the preliminary fusion features; The preliminary fusion features are input into the fusion Transformer encoder in the dual-branch Transformer model. The fusion Transformer encoder contains 4 encoding layers, each of which includes a multi-head self-attention sublayer, a cross-attention sublayer, and a feedforward neural network sublayer. The deep fusion features are obtained by calculation. A global average pooling operation is performed on the deep fusion features to compress the sequence features into a vector representation of a fixed dimension to obtain a fused feature vector.

[0025] In this example, the collected visual and voice information are preprocessed to obtain a visual input sequence and a voice input sequence, respectively. In the preprocessing of visual information, the user's visual data such as facial images and posture information are standardized through operations such as image denoising, scale normalization, and image enhancement to obtain a structured visual input sequence. For voice information, the preprocessing steps include removing background noise, normalizing volume, extracting voice features (such as Mel frequency cepstral coefficients, MFCC), etc., to obtain a processed voice input sequence. The visual input sequence is input into the visual branch Transformer encoder in the dual-branch Transformer model. The visual branch Transformer encoder contains 6 encoding layers, each of which consists of a multi-head self-attention sublayer and a feedforward neural network sublayer. The multi-head self-attention sublayer is used to capture the relationship between different parts of the visual input sequence. The input data is weighted through the self-attention mechanism, so that the model can focus on important information in the sequence and ignore irrelevant content. The calculation of multi-head self-attention is expressed as: ; in, , and Matrices representing query, key, and value, respectively. is the dimension of the key. Through this mechanism, the visual branch encoder can extract important visual features in each frame of the image and obtain a semantically rich visual feature representation. The feedforward neural network sublayer performs nonlinear mapping on the output of the self-attention sublayer to enhance the expressiveness of the features. These steps are calculated layer by layer through 6 encoding layers to obtain a visual feature representation that contains rich semantics of the user's visual information. At the same time, the speech input sequence is input into the speech branch Transformer encoder in the dual-branch Transformer model. The structure of the speech branch Transformer encoder is the same as that of the visual branch, and it also contains 6 encoding layers. Each encoding layer consists of a multi-head self-attention sublayer and a feedforward neural network sublayer. Multi-head self-attention is used to model the global dependency of speech features to capture the correlation between time steps in the speech sequence. The feedforward neural network sublayer performs nonlinear transformation on the speech features to enhance their semantic expression ability. After layer-by-layer processing of the 6 encoding layers, a speech feature representation is obtained, which contains various information in the user's speech, such as speech content, emotion, tone, etc. Position encoding is performed on the visual and speech feature representations, and the position information is injected into the feature representation. The purpose of position encoding is to enable the Transformer model to understand the relative relationship of each position in the sequence without an explicit sequential structure. Position encoding is calculated using the following formula: ; Among them, pos represents the position, represents the dimension of the feature, represents the total dimension of the feature representation. Through position encoding, the data of each time step in the visual feature representation and the speech feature representation contains position information, so that the model can better understand the structure and context of the sequence. The enhanced visual feature representation and the enhanced speech feature representation are input into the cross-attention layer, and the information interaction between the modalities is realized by calculating the attention weights between the visual and speech modalities. In the cross-attention layer, the visual features and the speech features pay attention to each other, so as to capture the potential correlation between vision and speech. For example, when the user speaks, the emotions in the speech correspond to the user's facial expressions through cross-attention, so as to enhance the system's understanding of the user's current state. Through the calculation of the cross-attention layer, the preliminary fusion features are obtained. The preliminary fusion features are input into the fusion transformer encoder in the dual-branch transformer model to realize the deep fusion of multimodal information. The fusion transformer encoder contains 4 encoding layers, each of which consists of a multi-head self-attention sublayer, a cross-attention sublayer, and a feedforward neural network sublayer. The multi-head self-attention sublayer is used to capture the dependencies between different time steps in the fusion features, so that the information can be effectively integrated in time sequence; the cross-attention sublayer strengthens the fusion between vision and speech, ensuring that the information between different modalities can understand and enhance each other. The feedforward neural network sublayer performs nonlinear mapping on the fusion features to enhance the expressive power of the features. After layer-by-layer processing of the four encoding layers, the deep fusion features are obtained. A global average pooling operation is performed on the deep fusion features. By averaging the time dimension of the feature sequence, a fixed-size vector representation is obtained. Assume that the deep fusion feature is , the global average pooling is expressed as: ; in, is the length of the feature sequence, It is The feature representation of time steps is is the fused feature vector after global average pooling. Through the global average pooling operation, the deep fusion features are compressed into a fixed-size vector that contains a comprehensive description of the visual and speech modality information.

[0026] In one example, the fused feature vector and the enhanced video stream are jointly analyzed to obtain the user interaction intention and interaction response strategy, including: Extracting spatiotemporal features from the enhanced video stream to obtain a spatiotemporal feature map of the video, and expanding the dimension of the fused feature vector. By copying and splicing operations, the fused feature vector is expanded to the same spatiotemporal dimension as the spatiotemporal feature map of the video to obtain an expanded fused feature map. The video spatiotemporal feature map and the extended fusion feature map are fused, and a joint feature map is obtained by element-by-element addition. Based on the joint feature map, the spatial attention weight and channel attention weight are calculated to obtain an attention-enhanced joint feature map. Perform temporal modeling on the attention-enhanced joint feature map to obtain temporal coding features, and input the temporal coding features into the multi-task learning head, including the intent classification branch and the response generation branch. The probability distribution of user interaction intention is obtained through the fully connected layer and the softmax function. Based on the probability distribution of user interaction intention, determine the user interaction intention through threshold judgment, and select the corresponding interaction response strategy template according to the predefined intention-response mapping table; The temporal coding features and the interactive response strategy template are input into the strategy refinement network to generate specific interactive response strategy parameters and obtain the interactive response strategy.

[0027] In this example, spatiotemporal feature extraction is performed on the video, and the temporal and spatial information in the video is captured at the same time, so that the system can understand the dynamic changes and static scenes in the video. Spatiotemporal feature extraction is achieved by combining convolutional neural network (CNN) and long short-term memory network (LSTM). CNN is responsible for extracting spatial features, while LSTM is responsible for modeling temporal dependencies to obtain the spatiotemporal feature map of the video. The spatiotemporal feature map is a feature that represents the changes of objects, actions, and scenes in the video. It contains spatial information about each frame of the image and temporal change information between frames. The fused feature vector is dimensionally expanded, and the fused feature vector is expanded to the same spatiotemporal dimension as the spatiotemporal feature map of the video through copying and splicing operations. In the feature fusion stage, the spatiotemporal feature map of the video and the expanded fused feature map are element-by-element added to obtain a joint feature map. The element-by-element addition operation adds the feature values ​​of the corresponding positions to achieve a comprehensive representation of the two features, so that the joint feature map contains both the spatiotemporal feature information of the video and the multimodal interaction information of the user. Based on the joint feature map, the spatial attention weight and channel attention weight are calculated to obtain the joint feature map with enhanced attention. The spatial attention weight is used to highlight the important spatial regions in the video, such as the part the user is looking at or the location of a significant object; while the channel attention weight is used to enhance or suppress specific feature dimensions by assigning different channel weights. The calculation of spatial attention is expressed as: ; in, Indicates location The attention weight at Indicates that the joint feature map is at position Features, is a mapping function used to calculate the attention weight, such as a simple linear transformation. Through the attention mechanism, the system pays more attention to important areas or features related to user interaction and obtains more accurate feature representation. The attention-enhanced joint feature map is temporally modeled to capture the changes in the feature sequence in the time dimension and obtain the temporal encoding features. Temporal modeling uses a recurrent neural network (RNN) or Transformer structure to ensure that the system can understand the dynamic changes in the video. Through temporal modeling, features at different time steps are linked to obtain a comprehensive understanding of the entire interaction process. The temporal encoding features contain the temporal change information in the video, as well as the behaviors and intentions shown by the user during the entire interaction process. The temporal encoding features are input into the multi-task learning head. The multi-task learning head includes an intent classification branch and a response generation branch, where the intent classification branch is used to predict the user's interaction intention, and the response generation branch is used to generate the corresponding interaction response strategy. Intent classification is implemented through a fully connected layer and a softmax function. The fully connected layer is used to map the temporal encoding features to a predefined intent category space, and the softmax function normalizes the scores of each category to obtain the probability distribution of the user's interaction intention. The calculation formula for the intent probability distribution is: ; in, Indicates The probability of an intention, is the output score of the fully connected layer, is the total number of intent categories. Through the softmax function, the probability of each intent category is obtained to determine the most likely interaction intention of the user. Based on the probability distribution of the user's interaction intention, the final user interaction intention is judged by the set threshold. For example, when the probability of a certain intention exceeds the preset threshold, the intention is determined as the user's current intention. After determining the user's interaction intention, the corresponding interaction response strategy template is selected according to the predefined intention-response mapping table. The intention-response mapping table is a mapping relationship that maps the user's interaction intention to a specific response strategy to ensure that different user intentions trigger appropriate system responses. After selecting the corresponding interaction response strategy template, the temporal coding features and the interaction response strategy template are input into the strategy refinement network to generate specific interaction response strategy parameters. The strategy refinement network is a neural network structure that can generate detailed response parameters that meet the user's current needs based on the input of the temporal coding features and the response strategy template. Through the strategy refinement network, the high-level response intention is concretized into an executable response strategy, such as adjustment of the playback content, switching of the interface, selection of audio feedback, etc., and finally the interaction response strategy is obtained.

[0028] In one example, based on the user interaction intention and the interaction response strategy, local contrast enhancement and motion compensation are performed on the enhanced video stream to obtain a high-quality video frame sequence, including: According to the user interaction intention, the corresponding local contrast enhancement parameters and motion compensation parameters are extracted from a preset intention-processing parameter mapping table; Perform frame-level segmentation on the enhanced video stream to obtain a video frame sequence, and adaptively divide each frame image into blocks based on a local contrast enhancement parameter to obtain an image block set; Calculating local statistical features for each image block in the image block set, and constructing an adaptive S-curve mapping function based on the local statistical features and local contrast enhancement parameters; The adaptive S-curve mapping function is applied to the corresponding image block to obtain the image block after local contrast enhancement, and the overlapping area of ​​adjacent image blocks is smoothly transitioned to obtain the video frame sequence after contrast enhancement; Based on the motion compensation parameters, block matching motion estimation is performed on adjacent frames in the contrast-enhanced video frame sequence to obtain a motion vector field, and motion compensation interpolation is performed on the contrast-enhanced video frame sequence according to the motion vector field to generate a video sequence with an increased frame rate; A time domain filter is applied to the video sequence after frame rate enhancement to obtain a smooth high frame rate video sequence, and the global brightness and saturation of the smooth high frame rate video sequence are adjusted according to the video enhancement level in the interactive response strategy to obtain a high-quality video frame sequence.

[0029] In this example, according to the user's interaction intention, the corresponding local contrast enhancement parameters and motion compensation parameters are extracted from the preset intention-processing parameter mapping table. The preset mapping table predefines suitable image processing parameters according to the user's different interaction intentions to provide the user with the best viewing experience. The local contrast enhancement parameters are used to adjust the detail performance of the image, while the motion compensation parameters are used to improve the smoothness of the video and reduce motion blur. The enhanced video stream is frame-level segmented to obtain a video frame sequence. The continuous video signal is divided into separate frames so that each frame can be processed and analyzed independently. Based on the extracted local contrast enhancement parameters, each frame of the image is adaptively divided into blocks to obtain a set of image blocks. Assume that each frame of the image is divided into image blocks, which are used for subsequent local contrast enhancement processing. For each image block, its local statistical features are calculated, such as brightness mean, standard deviation, contrast, etc. Local statistical features are indicators used to describe the distribution and changes of pixel values ​​in an image block, which can reflect information such as brightness, texture and contrast of the image block. Based on the local statistical features and the extracted local contrast enhancement parameters, an adaptive S-curve mapping function is constructed. The S-curve mapping function is a nonlinear function used to adjust the pixel value distribution in an image block to improve local contrast and detail performance. The adaptive S-curve mapping function is expressed as: ; in, Represents the position in the image block The original pixel value, represents the enhanced pixel value, is the contrast enhancement parameter, which is used to control the slope of the S-shaped curve. Represents the brightness mean of the image block. Through nonlinear mapping, low-brightness and high-brightness pixel values ​​are compressed, while medium-brightness pixels are stretched, thereby improving the local contrast of the image and making the details more obvious. The adaptive S-curve mapping function is applied to the corresponding image block to obtain the image block with local contrast enhancement. In order to ensure smooth transition between image blocks without obvious splicing marks, the overlapping area of ​​adjacent image blocks is smoothed. Smooth transition is achieved by weighted averaging, that is, in the overlapping area, the pixel values ​​from different image blocks are weighted summed to eliminate the boundary effect and obtain the contrast-enhanced video frame sequence. Based on the motion compensation parameters, block matching motion estimation is performed on adjacent frames in the contrast-enhanced video frame sequence to obtain the motion vector field. The motion vector field is a feature map that describes the direction and speed of movement of objects in the video, which is calculated by the block matching algorithm. The block matching algorithm compares an image block in the current frame with image blocks at different positions in the previous frame, finds the most similar position, and determines the motion vector of the image block. The calculation of the motion vector is expressed as: ; in, Represents the motion vector of the image block at the current position, is the motion offset, and are the width and height of the image block, respectively. and Represent the pixel values ​​of the current frame and the previous frame respectively. By finding the offset that minimizes the difference between frames, the motion vector is obtained, and then the video frame is subjected to motion compensation interpolation to generate a video sequence with an increased frame rate. Motion compensation interpolation increases the frame rate of the video by generating new intermediate frames between adjacent frames, making the video motion smoother and more natural, and reducing the blur caused by the rapid movement of objects. A time domain filter is applied to the video sequence after the frame rate is increased to obtain a smooth high frame rate video sequence. The time domain filter smoothes the time changes in the video frame sequence to eliminate artifacts and noise introduced by interpolation or other processing steps, thereby ensuring that the video transition in the time dimension is more natural and smooth. The time domain filter adopts the form of weighted moving average, and its calculation formula is: ; in, represents the frame after time domain filtering, Indicates time The original frame at is the size of the filter window. By taking a weighted average of the current frame and several frames before and after it, the discontinuity between video frames is eliminated to obtain a smooth high frame rate video sequence. According to the video enhancement level in the interactive response strategy, the global brightness and saturation of the smooth high frame rate video sequence are adjusted to obtain the final high-quality video frame sequence. The video enhancement level is set according to the user's personalized needs, including different brightness and saturation adjustment strategies to adapt to different viewing environments and preferences. The adjustment of global brightness and saturation is achieved through the following formula: ; in, represents the original pixel value, Represents the adjusted pixel value, is the saturation adjustment factor, which controls the intensity of the color. It is the brightness offset that controls the overall brightness of the picture. Through linear adjustment, the brightness and color performance of the video can be flexibly adjusted to ensure that the final video output meets the viewing needs of users.

[0030] In one example, scene adaptive color space mapping and dynamic color temperature adjustment are performed on a high-quality video frame sequence according to ambient light information acquired in real time and pre-stored color characteristics of a display device to obtain an output video stream, including: Perform color space conversion on high-quality video frame sequences, convert the RGB color space into the HSV color space, and obtain the hue H, saturation S and lightness V components; Based on the ambient light sensor, the ambient light information is collected in real time, including the ambient light intensity and color temperature, and the current environment is classified into bright, moderate or dim scenes according to the preset ambient light threshold to obtain the ambient scene type; According to the environmental scene type and the pre-stored color characteristics of the display device, the corresponding color enhancement coefficients α, β and γ are selected from the preset scene-color mapping table; Perform nonlinear adjustment on the brightness V component and use the formula V'=V γ Perform Gamma correction to obtain the adjusted brightness component V'; Based on the color enhancement coefficients α and β, the saturation S component is linearly adjusted, and the adjusted saturation component S' is obtained using the formula S'=α*S+β; According to the ambient color temperature information, the hue offset ΔH is calculated, and the hue H component is adjusted. The adjusted hue component H' is obtained using the formula H'=H+ΔH; The adjusted hue component H', saturation component S' and lightness component V' are combined, and an inverse transformation from HSV to RGB color space is performed to obtain a color-enhanced video frame; Apply the Floyd-Steinberg dithering algorithm to the color-enhanced video frames, adjust the color depth to the bit depth supported by the display device, obtain an output video frame sequence, encode the output video frame sequence into a video stream in H.264 format, set the bit rate and key frame interval of network transmission, and obtain the output video stream.

[0031] In this example, the color space conversion is performed on the high-quality video frame sequence, and the RGB color space is converted to the HSV color space. The RGB color space representation is more direct for machines and display devices, but the HSV color space is more in line with human perception of color and is converted to HSV when color adjustment is performed. The HSV color space includes three components: hue (H), saturation (S), and brightness (V), where hue represents the type of color, such as red, green, blue, etc., saturation represents the purity of the color, and brightness represents the brightness of the color or the intensity of light. Based on the ambient light information collected in real time by the ambient light sensor, including the ambient light intensity and color temperature, the current environment is classified into bright, moderate or dim scenes according to the preset ambient light threshold, and the ambient scene type is obtained. For example, when the ambient light sensor detects a higher light intensity, the current environment is classified as a bright scene, and when the light intensity is low, it is classified as a dim scene. According to the different light conditions of the environment, the video content on the display device is intelligently adjusted to ensure that the best viewing experience can be provided to users under various light conditions. According to the environmental scene type and the pre-stored display device color characteristics, select the corresponding color enhancement coefficient from the preset scene-color mapping table , β and γ. These color enhancement coefficients are used to adjust the various components of the HSV color space to adapt to the current ambient light conditions. For bright scenes, a larger brightness enhancement coefficient is selected to ensure that the image is still clear under strong light, while in dim scenes, an appropriate saturation enhancement coefficient is selected to improve the color expression of the picture, making the picture appear more vivid and bright. The brightness V component is adjusted nonlinearly and corrected using the Gamma correction formula. The formula for Gamma correction is: ; in, represents the original brightness component, represents the adjusted brightness component, is the gamma correction coefficient selected from the scene-color map. Gamma correction adapts to different display environments by performing nonlinear transformations on brightness. For example, when watching a video in a dim environment, choose a lower The value enhances the performance of the bright part, making the image clearer in a dark environment; in a strong light environment, choose a higher The value prevents the image from being too bright and preserves the details. After completing the brightness adjustment, the saturation S component is linearly adjusted using the following formula: ; in, represents the original saturation component, represents the adjusted saturation component, and The saturation enhancement factor and offset are selected from the scene-color map. Linear adjustment makes the colors of the picture richer or softer by changing the saturation value. For example, in a dim environment, increasing The value is used to increase the saturation and make the color more vivid, thus compensating for the visual color loss caused by insufficient ambient light. In a brighter environment, the saturation is reduced to prevent the color from being too glaring and ensure visual comfort. The hue offset is calculated based on the color temperature data in the ambient light information. , and adjust the hue H component to obtain the adjusted hue component. The hue adjustment formula is: ; in, represents the original hue component, Represents the adjusted hue component, Indicates the hue shift, which is calculated based on the color temperature information of the environment. For example, when the color temperature of the environment is high, the light is cooler. The value of to make the image hue warmer, thereby neutralizing the visual discomfort caused by cold light; when the ambient color temperature is low, the light is warmer, then increase The value of makes the image hue cooler to keep the color balanced and natural. , Saturation component and lightness component Recombined, and the color space is inversely transformed from HSV to RGB to obtain a color-enhanced video frame. The color performance of the video is adaptively adjusted according to the current ambient light conditions, so that the output video content is more in line with human visual perception and provides a better viewing experience. The Floyd-Steinberg dithering algorithm is applied to the color-enhanced video frame to adjust the color depth to the bit depth supported by the display device. Floyd-Steinberg dithering is an image processing algorithm used to reduce the visual artifacts introduced by quantization by propagating errors between pixels when reducing the color depth, making the image visually smoother and more natural. For example, if the display device can only support 8-bit color depth, and the original video frame is 10-bit color depth, the dithering algorithm effectively converts the 10-bit color depth video frame to 8-bit color depth while keeping the visual quality of the image unnoticeably affected. After the color depth adjustment is completed, the output video frame sequence is encoded into a video stream in H.264 format. H.264 is a coding standard applied to video compression that can achieve efficient compression while ensuring high image quality. During the encoding process, the bit rate and key frame interval of network transmission are set to ensure smooth transmission of video content under different network conditions. For example, under network conditions with limited bandwidth, a lower bit rate and a longer key frame interval are selected to reduce the burden of data transmission; while under conditions with sufficient bandwidth, a higher bit rate and a shorter key frame interval are selected to provide higher image quality and a better user experience.

[0032] Reference Figure 2 This embodiment provides an image processing device for an intelligent interactive control unit of a TV box, including: Feature analysis module 1, used to perform feature analysis on the input video stream of the TV box to obtain a video feature data structure; Processing module 2, used for dividing the input video stream into overlapping image blocks, independently processing and weighted reorganizing according to the video feature data structure to obtain an enhanced video stream; Fusion module 3, used for collecting the user's visual and voice interaction information based on the intelligent interaction control unit, and inputting the visual and voice interaction information into the dual-branch Transformer model for multimodal information fusion to obtain a fusion feature vector; A joint analysis module 4 is used to jointly analyze the fused feature vector and the enhanced video stream to obtain the user interaction intention and interaction response strategy; The compensation module 5 is used to perform local contrast enhancement and motion compensation on the enhanced video stream based on the user interaction intention and the interaction response strategy to obtain a high-quality video frame sequence; The adjustment module 6 is used to perform scene adaptive color space mapping and dynamic color temperature adjustment on the high-quality video frame sequence according to the ambient light information obtained in real time and the pre-stored color characteristics of the display device to obtain an output video stream.

[0033] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the above method embodiment, which will not be repeated here.

[0034] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0035] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0036] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0037] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0038] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0039] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for image processing of a TV box intelligent interactive control unit, characterized in that: The following steps are involved: Perform feature analysis on the input video stream of the TV box to obtain the video feature data structure; According to the video feature data structure, the input video stream is divided into overlapping image blocks, independently processed and weightedly reorganized to obtain an enhanced video stream; Based on the intelligent interaction control unit, the user's visual and voice interaction information is collected, and the visual and voice interaction information is input into the dual-branch Transformer model for multimodal information fusion to obtain a fused feature vector; Performing joint analysis on the fused feature vector and the enhanced video stream to obtain a user interaction intention and an interaction response strategy; Based on the user interaction intention and the interaction response strategy, performing local contrast enhancement and motion compensation on the enhanced video stream to obtain a high-quality video frame sequence; The high-quality video frame sequence is subjected to scene-adaptive color space mapping and dynamic color temperature adjustment according to the ambient light information acquired in real time and the pre-stored color characteristics of the display device to obtain an output video stream.

2. The image processing method of the TV box intelligent interactive control unit according to claim 1, characterized in that: The feature analysis of the input video stream of the TV box is performed to obtain a video feature data structure, including: Perform frame-level segmentation on the input video stream of the TV box to obtain a video frame sequence, and perform spatial domain analysis on each frame in the video frame sequence to obtain spatial feature parameters including resolution, color space, brightness distribution, contrast, and saturation; Performing a differential operation on adjacent frames in the video frame sequence to obtain an inter-frame difference map, and calculating a motion vector based on the inter-frame difference map to obtain a motion feature parameter; Performing a two-dimensional discrete Fourier transform on the video frame sequence to obtain frequency domain feature parameters, and combining the spatial feature parameters, the motion feature parameters and the frequency domain feature parameters to construct a multidimensional feature vector; Performing an adaptive principal component analysis on the multidimensional feature vector to obtain a feature vector after dimension reduction, and extracting high-level semantic features corresponding to the feature vector after dimension reduction; The high-level semantic features, the spatial feature parameters, the motion feature parameters and the frequency domain feature parameters are weightedly fused using an attention mechanism to obtain a video feature data structure.

3. The image processing method of the TV box intelligent interactive control unit according to claim 2, characterized in that: The step of dividing the input video stream into overlapping image blocks, independently processing and weighted reorganizing the input video stream according to the video feature data structure to obtain an enhanced video stream includes: According to the resolution parameter in the video feature data structure, each frame in the input video stream is grid-divided to obtain NxM basic image blocks, and the NxM basic image blocks are expanded by 50% overlap to obtain (2N-1)x(2M-1) overlapping image blocks, where the size of each overlapping image block is twice that of the original basic image block; According to the brightness distribution parameters in the video feature data structure, each overlapping image block is subjected to adaptive histogram equalization to obtain overlapping image blocks after brightness equalization; Based on the frequency domain feature parameters in the video feature data structure, wavelet transform is performed on the overlapping image blocks after brightness equalization to extract multi-scale features, and soft threshold denoising is performed on the high frequency sub-band to obtain denoised overlapping image blocks; According to the motion feature parameters in the video feature data structure, motion compensation is performed on the denoised overlapping image blocks, motion vectors are calculated by a block matching algorithm, and sub-pixel interpolation is performed to obtain motion-compensated overlapping image blocks; Based on the contrast parameter in the video feature data structure, applying a local contrast enhancement algorithm to the overlapping image blocks after motion compensation, adjusting pixel intensity through adaptive gamma correction, and obtaining overlapping image blocks after contrast enhancement; The contrast-enhanced overlapping image blocks are weighted to obtain weighted overlapping image blocks, and pixel-level weighted averaging of the overlapping areas of the weighted overlapping image blocks is performed according to the original positions to reconstruct a complete video frame, and the reconstructed video frame sequences are combined to obtain an enhanced video stream.

4. The image processing method of the TV box intelligent interactive control unit according to claim 3 is characterized in that: The intelligent interactive control unit collects the user's visual and voice interaction information, and inputs the visual and voice interaction information into a dual-branch Transformer model for multimodal information fusion to obtain a fused feature vector, including: Based on the intelligent interactive control unit, the collected user visual information is preprocessed to obtain a visual input sequence, and the collected user voice information is preprocessed to obtain a voice input sequence; Inputting the visual input sequence into a visual branch Transformer encoder in a dual-branch Transformer model, wherein the visual branch Transformer encoder comprises 6 encoding layers, each encoding layer comprises a multi-head self-attention sublayer and a feedforward neural network sublayer, and obtaining a visual feature representation by calculation; Inputting the speech input sequence into a speech branch Transformer encoder in a dual-branch Transformer model, wherein the speech branch Transformer encoder comprises 6 encoding layers, each encoding layer having the same structure as the visual branch, and obtaining speech feature representation by calculation; Performing position encoding on the visual feature representation and the speech feature representation, injecting position information into the feature representation, and obtaining an enhanced visual feature representation and an enhanced speech feature representation; Inputting the enhanced visual feature representation and the enhanced speech feature representation into a cross attention layer, realizing the interaction of visual and speech information by calculating the attention weights between the modalities, and obtaining preliminary fusion features; Inputting the preliminary fusion features into a fusion Transformer encoder in a dual-branch Transformer model, wherein the fusion Transformer encoder comprises four encoding layers, each encoding layer comprising a multi-head self-attention sublayer, a cross-attention sublayer, and a feedforward neural network sublayer, and obtaining deep fusion features through calculation; A global average pooling operation is performed on the deep fusion features to compress the sequence features into a vector representation of a fixed dimension to obtain a fusion feature vector.

5. The image processing method of the TV box intelligent interactive control unit according to claim 4 is characterized in that: The jointly analyzing the fused feature vector and the enhanced video stream to obtain the user interaction intention and the interaction response strategy includes: Extracting spatiotemporal features from the enhanced video stream to obtain a video spatiotemporal feature map, and performing dimension expansion on the fused feature vector, expanding the fused feature vector to the same spatiotemporal dimension as the video spatiotemporal feature map through copying and splicing operations to obtain an expanded fused feature map; Performing feature fusion on the video spatiotemporal feature map and the extended fusion feature map, obtaining a joint feature map by element-by-element addition operation, and calculating spatial attention weights and channel attention weights based on the joint feature map to obtain an attention-enhanced joint feature map; Performing temporal modeling on the attention-enhanced joint feature map to obtain temporal coding features, and inputting the temporal coding features into a multi-task learning head, including an intent classification branch and a response generation branch, and obtaining a probability distribution of user interaction intentions through a fully connected layer and a softmax function; Based on the probability distribution of user interaction intention, determine the user interaction intention through threshold judgment, and select a corresponding interaction response strategy template according to a predefined intention-response mapping table; The temporal coding feature and the interactive response strategy template are input into a strategy refinement network to generate specific interactive response strategy parameters to obtain an interactive response strategy.

6. The image processing method of the TV box intelligent interactive control unit according to claim 5, characterized in that: The method of performing local contrast enhancement and motion compensation on the enhanced video stream based on the user interaction intention and the interaction response strategy to obtain a high-quality video frame sequence includes: According to the user interaction intention, extracting corresponding local contrast enhancement parameters and motion compensation parameters from a preset intention-processing parameter mapping table; Performing frame-level segmentation on the enhanced video stream to obtain a video frame sequence, and adaptively dividing each frame of the image into blocks based on the local contrast enhancement parameter to obtain an image block set; Calculating local statistical features for each image block in the image block set, and constructing an adaptive S-curve mapping function according to the local statistical features and the local contrast enhancement parameters; Applying the adaptive S-curve mapping function to the corresponding image block to obtain an image block with local contrast enhancement, and performing smooth transition on the overlapping areas of adjacent image blocks to obtain a video frame sequence with contrast enhancement; Based on the motion compensation parameters, block matching motion estimation is performed on adjacent frames in the contrast enhanced video frame sequence to obtain a motion vector field, and motion compensation interpolation is performed on the contrast enhanced video frame sequence according to the motion vector field to generate a video sequence with an increased frame rate; A time domain filter is applied to the video sequence after the frame rate is increased to obtain a smooth high frame rate video sequence, and global brightness and saturation are adjusted for the smooth high frame rate video sequence according to the video enhancement level in the interactive response strategy to obtain a high-quality video frame sequence.

7. The image processing method of the TV box intelligent interactive control unit according to claim 6, characterized in that: The method of performing scene adaptive color space mapping and dynamic color temperature adjustment on the high-quality video frame sequence according to the ambient light information acquired in real time and the pre-stored color characteristics of the display device to obtain an output video stream includes: Performing color space conversion on the high-quality video frame sequence, converting the RGB color space into the HSV color space, and obtaining hue H, saturation S, and lightness V components; Based on the ambient light sensor, the ambient light information is collected in real time, including the ambient light intensity and color temperature, and the current environment is classified into bright, moderate or dim scenes according to the preset ambient light threshold to obtain the ambient scene type; According to the environmental scene type and the pre-stored color characteristics of the display device, the corresponding color enhancement coefficients α, β and γ are selected from a preset scene-color mapping table; The lightness V component is nonlinearly adjusted and the formula V'=V is used. γ Perform Gamma correction to obtain the adjusted brightness component V'; Based on the color enhancement coefficients α and β, the saturation S component is linearly adjusted, and the adjusted saturation component S' is obtained using the formula S'=α*S+β; Calculate the hue offset ΔH according to the ambient color temperature information, and adjust the hue H component, using the formula H'=H+ΔH to obtain the adjusted hue component H'; Combining the adjusted hue component H', saturation component S' and lightness component V', and performing an inverse transformation from HSV to RGB color space to obtain a color-enhanced video frame; Applying the Floyd-Steinberg dithering algorithm to the color-enhanced video frame, adjusting the color depth to the bit depth supported by the display device, obtaining an output video frame sequence, encoding the output video frame sequence into a video stream in H.264 format, setting the bit rate and key frame interval of network transmission, and obtaining an output video stream.

8. An image processing device for an intelligent interactive control unit of a TV box, characterized in that: For implementing the steps of the method according to any one of claims 1 to 7, the device comprises: The feature analysis module is used to perform feature analysis on the input video stream of the TV box to obtain a video feature data structure; A processing module, used for dividing the input video stream into overlapping image blocks, independently processing and weighted reorganizing according to the video feature data structure to obtain an enhanced video stream; A fusion module is used to collect the user's visual and voice interaction information based on the intelligent interaction control unit, and input the visual and voice interaction information into a dual-branch Transformer model for multimodal information fusion to obtain a fusion feature vector; A joint analysis module, used for jointly analyzing the fused feature vector and the enhanced video stream to obtain a user interaction intention and an interaction response strategy; A compensation module, configured to perform local contrast enhancement and motion compensation on the enhanced video stream based on the user interaction intention and the interaction response strategy to obtain a high-quality video frame sequence; The adjustment module is used to perform scene-adaptive color space mapping and dynamic color temperature adjustment on the high-quality video frame sequence according to the ambient light information obtained in real time and the pre-stored color characteristics of the display device to obtain an output video stream.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images