Image processing method and device for smart interactive control unit of television box

Abstract

The application relates to a television box intelligent interaction control unit image processing method and equipment. The method comprises the following steps: performing feature analysis on the input video stream of the television box to obtain an enhanced video stream; based on the intelligent interaction control unit, visual and voice interaction information of a user is collected and input into a double-branch Transformer model for multi-modal information fusion to obtain a fusion feature vector; joint analysis is performed to obtain a user interaction intention and an interaction response strategy; local contrast enhancement and motion compensation are performed to obtain a high-quality video frame sequence; and the high-quality video frame sequence is subjected to scene adaptive color space mapping and dynamic color temperature adjustment according to real-time acquired ambient light information and pre-stored display device color characteristics to obtain an output video stream. The implementation of the application can fully utilize limited computing resources, realize efficient video processing and natural human-computer interaction, and dynamically optimize display effects according to environmental changes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a television box intelligent interaction control unit image processing method and device. BACKGROUND

[0002] With the rapid development of smart home technology, television boxes as important equipment of home entertainment centers, their intelligent interaction and image processing capabilities are increasingly attracting the attention of users. Traditional television boxes have obvious shortcomings in processing high dynamic range videos, user interaction in complex scenes, and adapting to different environmental light conditions. Existing technologies often use single modal interaction methods, which are difficult to accurately understand the complex intentions of users, resulting in poor interaction experience. At the same time, the image processing algorithm has high computational complexity when processing high-resolution, high-frame-rate videos, making it difficult to run in real time on resource-constrained embedded devices.

[0003] In addition, the current image enhancement technology of television boxes mostly uses global processing methods, which are difficult to adaptively optimize the image content of different regions, resulting in unsatisfactory processing results. In different light environments, the consistency of display effect is also a problem to be solved. Traditional methods cannot dynamically adjust display parameters according to real-time environmental light changes, causing image overexposure or underexposure in some scenes, affecting user viewing experience. SUMMARY

[0004] The main purpose of the present application is to provide a television box intelligent interaction control unit image processing method and device, which can fully utilize limited computing resources, realize efficient video processing and natural human-computer interaction, and dynamically optimize display effect according to environmental changes.

[0005] To achieve the above purpose, the present application provides a television box intelligent interaction control unit image processing method, comprising the following steps:

[0006] Performing feature analysis on the input video stream of the television box to obtain a video feature data structure;

[0007] According to the video feature data structure, performing overlapping image block division, independent processing and weighted reorganization on the input video stream to obtain an enhanced video stream;

[0008] Based on the intelligent interaction control unit, collecting visual and voice interaction information of the user, and inputting the visual and voice interaction information into a double-branch Transformer model for multi-modal information fusion to obtain a fusion feature vector;

[0009] Performing joint analysis on the fusion feature vector and the enhanced video stream to obtain a user interaction intention and an interaction response strategy;

[0010] based on the user interaction intention and the interaction response strategy, the enhanced video stream is subjected to local contrast enhancement and motion compensation to obtain a high-quality video frame sequence;

[0011] According to the real-time acquired ambient light information and the pre-stored display device color characteristics, scene adaptive color space mapping and dynamic color temperature adjustment are performed on the high-quality video frame sequence to obtain an output video stream.

[0012] The application also provides a television box intelligent interaction control unit image processing device, comprising:

[0013] a feature analysis module for performing feature analysis on the input video stream of the television box to obtain a video feature data structure;

[0014] a processing module for performing overlapping image block division, independent processing and weighted reorganization on the input video stream according to the video feature data structure to obtain an enhanced video stream;

[0015] a fusion module for collecting visual and voice interaction information of a user based on an intelligent interaction control unit, and inputting the visual and voice interaction information into a double-branch Transformer model for multi-modal information fusion to obtain a fusion feature vector;

[0016] a joint analysis module for performing joint analysis on the fusion feature vector and the enhanced video stream to obtain a user interaction intention and an interaction response strategy;

[0017] a compensation module for performing local contrast enhancement and motion compensation on the enhanced video stream based on the user interaction intention and the interaction response strategy to obtain a high-quality video frame sequence;

[0018] an adjustment module for performing scene adaptive color space mapping and dynamic color temperature adjustment on the high-quality video frame sequence according to real-time acquired ambient light information and pre-stored display device color characteristics to obtain an output video stream.

[0019] The application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to realize the steps of the method according to any one of the above.

[0020] The application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to realize the steps of the method according to any one of the above.

[0021] In summary, the technical scheme provided by the present application realizes efficient video enhancement by performing feature analysis and overlapping image block processing on the input video stream, improves image quality and processing speed, and is suitable for resource-constrained embedded devices. The dual-branch Transformer model is used for multi-modal information fusion, effectively integrating visual and voice interaction information, improving the accuracy and robustness of user interaction intent recognition. The user interaction intent recognition and response strategy generation method based on joint analysis realizes a more intelligent and natural human-computer interaction experience. Through local contrast enhancement and motion compensation technology, the details of the video and the smoothness of the motion are optimized, significantly improving the visual quality. The introduction of scene adaptive color space mapping and dynamic color temperature adjustment based on machine learning enables the output video stream to automatically adjust according to changes in environmental light, ensuring consistent viewing experience under different lighting conditions. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a schematic diagram of the image processing method steps of the television box intelligent interaction control unit in an embodiment of the present application;

[0023] Figure 2 is a structural block diagram of the image processing device of the television box intelligent interaction control unit in an embodiment of the present application;

[0024] Figure 3 is a structural schematic block diagram of a computer device according to an embodiment of the present application.

[0025] The implementation of the object of the present application, functional characteristics and advantages will be further described with reference to the accompanying drawings. DETAILED DESCRIPTION

[0026] In order to make the object, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0027] With reference to Figure 1 The present embodiment provides an image processing method for a television box intelligent interaction control unit, comprising the following steps:

[0028] S1, performing feature analysis on the input video stream of the television box to obtain a video feature data structure;

[0029] The input video stream of the television box is frame-level segmented to obtain a video frame sequence. The continuous video signal is decomposed into independent frames, so that each frame is analyzed and processed as an independent image. Spatial domain analysis is performed on each frame in the video frame sequence. Spatial domain analysis is a method of directly extracting features from an image in the spatial domain. By calculating the spatial domain of each frame image, a series of spatial feature parameters such as resolution, color space, brightness distribution, contrast, and saturation are obtained. Among them, the resolution reflects the detail expression ability of the image, the color space relates to the representation of colors in the image, the brightness distribution describes the lighting condition of the image, and the contrast and saturation directly affect the visual effect of the image. At the same time, difference operation is performed on adjacent frames in the video frame sequence to obtain an inter-frame difference map, which reveals the changes between video frames, especially the motion of objects in dynamic scenes. Based on the inter-frame difference map, a motion vector is calculated. The motion vector is an important feature for describing motion information in a video, which is used to reflect the motion trajectory and direction of objects in a scene, thereby obtaining motion feature parameters. Two-dimensional discrete Fourier transform is performed on the video frame sequence to convert the video frame from the spatial domain to the frequency domain, and frequency domain feature parameters are extracted. Two-dimensional discrete Fourier transform converts an image from the spatial domain to the frequency domain, which can effectively separate low-frequency information and high-frequency information in the image. The low-frequency information represents the overall structure and contour of the image, while the high-frequency information represents the details and edge features of the image. The spatial feature parameters, motion feature parameters, and frequency domain feature parameters are combined to construct a multi-dimensional feature vector. The multi-dimensional feature vector is processed by dimension reduction. The adaptive principal component analysis method is used to reduce the dimension of the multi-dimensional feature vector, extract the most important feature components for describing the video content, and obtain the reduced feature vector. High-level semantic features are extracted from the reduced feature vector. High-level semantic features are semantic-level descriptions of video content, which are used to express higher-level information in the video, such as scene type, object category, and user behavior. The high-level semantic features, spatial feature parameters, motion feature parameters, and frequency domain feature parameters are weighted and fused. The attention mechanism is used to weight the features, and the weights are adaptively adjusted according to the importance of the features in different scenes to obtain more accurate feature fusion results. Through the weighted fusion of the attention mechanism, a video feature data structure is obtained.

[0030] S2, performing overlapping image block division, independent processing, and weighted reorganization on the input video stream according to the video feature data structure to obtain an enhanced video stream;

[0031] Specifically, according to the resolution parameter in the video feature data structure, each frame in the input video stream is grid divided, and the whole image is divided into a plurality of small regions, obtaining NxM basic image blocks. Each basic image block is expanded by 50% overlap, effectively reducing the boundary effect and the gap problem during image stitching, obtaining (2N-1)x(2M-1) overlapping image blocks, and the size of each overlapping image block is 2 times the size of the original basic image block. The expansion makes the adjacent regions have more relevance, which helps to process the image details more finely and ensures the visual consistency of the overlapping regions. According to the brightness distribution parameter in the video feature data structure, each overlapping image block is adaptively histogram equalized. Histogram equalization is a method for enhancing the overall contrast of the image, which can effectively improve the brightness and detail performance of the image by adaptively redistributing the brightness, so that the originally dark or bright regions can present more details. This process is performed on each overlapping image block respectively, which improves the brightness balance of the video frame as a whole, obtaining the brightness equalized overlapping image block. Based on the frequency domain feature parameter in the video feature data structure, the brightness equalized overlapping image block is wavelet transformed. Wavelet transform is an analysis method that decomposes an image into different scale and frequency components. Through wavelet transform, the multi-scale features of the image are extracted, so that different details in the image can be represented in different frequency subbands. For high-frequency subbands, which usually contain edge information and detail features in the image, in order to remove noise and retain useful information, soft threshold denoising is performed on the high-frequency subbands by setting a suitable threshold to remove high-frequency noise, obtaining the denoised overlapping image block. Based on the motion feature parameter in the video feature data structure, the denoised overlapping image block is motion compensated. The goal of motion compensation is to reduce the image blur caused by object motion. Through block matching algorithm to calculate the motion vector, find the corresponding relationship between adjacent frames, and use sub-pixel level interpolation method to accurately perform motion compensation, obtain the motion compensated overlapping image block. In the motion compensated overlapping image block, based on the contrast parameter in the video feature data structure, a local contrast enhancement algorithm is applied to optimize the visual performance of the image. Local contrast enhancement adjusts the intensity of pixels through adaptive gamma correction, so that the dark details in the image can be enhanced, and the bright part will not be overexposed, obtaining the contrast enhanced overlapping image block. Adaptive gamma correction can flexibly adjust the contrast according to the local characteristics of the image, so that the image has more levels in vision, and the overall effect is more lively. The contrast enhanced overlapping image block is weighted to ensure seamless fusion between different image blocks in the overlapping region. The weighted overlapping image block is pixel-level weighted averaged according to the original position to reconstruct the complete video frame. The weighted averaging method makes the information from different image blocks in the overlapping region transition smoothly, thereby avoiding the appearance of stitching marks, making the reconstructed image more natural and smooth in vision.After the above processing, all the reconstructed video frame sequences are combined to obtain a final enhanced video stream.

[0032] S3, based on the intelligent interaction control unit, collecting visual and voice interaction information of the user, and inputting the visual and voice interaction information into a double-branch Transformer model for multi-modal information fusion to obtain a fusion feature vector;

[0033] It should be noted that the visual information collected by the intelligent interaction control unit is preprocessed, and the preprocessing of the visual information includes image denoising, normalization, feature extraction and the like to obtain a visual input sequence. Meanwhile, the collected user speech information is preprocessed, and the speech preprocessing involves removing background noise, feature extraction (such as MFCC or Mel spectrum), and the like to obtain a speech input sequence. The visual input sequence is input into a visual branch Transformer encoder in the dual-branch Transformer model. The visual branch Transformer encoder includes 6 encoding layers, and each encoding layer is composed of a multi-head self-attention sublayer and a feedforward neural network sublayer. In the multi-head self-attention sublayer, the visual input sequence obtains global dependency through the multi-head attention mechanism, which can not only capture local features but also integrate the overall information of the image into the feature representation. The feedforward neural network sublayer performs nonlinear mapping on the attention result to enhance the expression ability of the features, and finally the visual feature representation is obtained through the step-by-step calculation of the 6 encoding layers. The visual feature representation not only contains the static information such as the facial expression of the user, but also can capture the dynamic information such as the head movement of the user. At the same time, the speech input sequence is input into a speech branch Transformer encoder in the dual-branch Transformer model. The structure of the speech branch Transformer encoder is the same as that of the visual branch, including 6 encoding layers, and each encoding layer is composed of a multi-head self-attention sublayer and a feedforward neural network sublayer. The multi-head self-attention sublayer is used for global modeling of the speech signal to capture the time sequence correlation and context information of the speech, and the feedforward neural network sublayer enhances the nonlinear expression ability of the speech features. In this way, the speech input sequence finally obtains the speech feature representation after being processed by the 6 encoding layers, which contains key information such as emotion, tone, and speed in the user's speech. The visual feature representation and the speech feature representation are positionally encoded to inject position information into the feature representation. The purpose of position encoding is to enable the Transformer model to have sequence perception ability without using a recursive structure, so that the model can understand the relative relationship between features at different time steps in the subsequent processing process to obtain enhanced visual feature representation and enhanced speech feature representation. The enhanced visual feature representation and the enhanced speech feature representation are input into a cross-attention layer to realize information interaction between different modalities. The cross-attention layer can effectively capture the correlation between different modalities by calculating the attention weight between vision and speech, realize mutual understanding and fusion of visual and speech information, and obtain preliminary fusion features. The preliminary fusion features are input into a fusion Transformer encoder in the dual-branch Transformer model. The fusion Transformer encoder includes 4 encoding layers, and each encoding layer is composed of a multi-head self-attention sublayer, a cross-attention sublayer and a feedforward neural network sublayer.In the multi-head self-attention sublayer, the fusion features obtain the global correlation between each sequence element through the self-attention mechanism, and enhance the internal consistency of the features. In the cross-attention sublayer, the information between different modalities is interacted again, and the relevant information of vision and voice is fused to obtain higher-level semantic correlation. The feedforward neural network sublayer is used to strengthen the nonlinear expression ability of the features. Finally, through the step-by-step calculation of the four encoding layers, the deep fusion features are obtained. The global average pooling operation is performed on the deep fusion features, and the original sequence features are compressed into a fixed-dimensional vector representation. The global average pooling compresses the time dimension information in the sequence features to obtain the fusion feature vector.

[0034] S4, jointly analyzing the fusion feature vector and the enhanced video stream to obtain a user interaction intention and an interaction response strategy;

[0035] Specifically, spatio-temporal feature extraction is performed on the enhanced video stream to extract features containing time and space information from the video, while capturing the temporal relationship between frames in the video and the spatial features of single-frame images, obtaining a spatio-temporal feature map of the video. The spatio-temporal feature map contains dynamic changes and static scene details of the video. The fusion feature vector is dimensionally expanded to facilitate fusion with the spatio-temporal feature map. The fusion feature vector is expanded through copying and splicing operations to make its spatio-temporal dimensions consistent with the video spatio-temporal feature map, obtaining an expanded fusion feature map. Feature fusion is performed on the video spatio-temporal feature map and the expanded fusion feature map to obtain a joint feature map through element-by-element addition operations. Based on the joint feature map, spatial attention weights and channel attention weights are calculated to obtain an attention-enhanced joint feature map. Spatial attention weights are used to emphasize important spatial regions in the image, making the model pay more attention to key scenes and objects; while channel attention weights enhance the expression ability of specific features by assigning different weights between feature channels, improving the ability to distinguish multi-modal information. Through the combination of these two attention mechanisms, the joint feature map is effectively enhanced, making important information stand out and be strengthened. Temporal modeling is performed on the attention-enhanced joint feature map to obtain a temporal encoding feature. The input spatio-temporal features are integrated in the time sequence to capture the dynamic change characteristics in the video and interactive information, and through modeling, the system can better understand the evolution process of user behavior over time. In order to obtain the user's interactive intent, the temporal encoding feature is input into the multi-task learning head, which includes an intent classification branch and a response generation branch. The intent classification branch classifies the user's interactive intent through a fully connected layer and a softmax function to obtain the probability distribution of the user's interactive intent. The fully connected layer performs linear transformation on the input feature to extract different class feature representations, and the softmax function normalizes these class feature representations to finally output a probability distribution representing the user's interactive intent and its corresponding probability. In this way, the system can describe the user's intent in the form of probability. Based on the probability distribution of the user's interactive intent, a threshold setting is used to determine the user's interactive intent. When the probability value of an intent exceeds the pre-set threshold, that intent is determined as the actual interactive intent of the current user. After determining the user's interactive intent, the corresponding interactive response strategy template is selected according to the pre-defined intent-response mapping table. By looking up the mapping table, the most suitable response strategy template is quickly found to facilitate appropriate responses to the user in different situations. The temporal encoding feature and the interactive response strategy template are input into the strategy refinement network to generate specific interactive response strategy parameters. The strategy refinement network refines the high-level response strategy into executable parameters based on the temporal encoding feature and the strategy template, so that the system responds to the user's request according to these parameters, and finally obtains the interactive response strategy, including adjustment of video content, changes in interface, playback of audio, etc.

[0036] S5, based on the user interaction intention and the interaction response strategy, performing local contrast enhancement and motion compensation on the enhanced video stream to obtain a high-quality video frame sequence;

[0037] The corresponding local contrast enhancement parameter and motion compensation parameter are extracted from a preset intent-processing parameter mapping table according to the user's interaction intent. The preset intent-processing parameter mapping table stores the best image processing parameters corresponding to different user intents, including parameters related to contrast enhancement and motion compensation, to ensure the best visual effect in different user demand scenarios. The enhanced video stream is frame-level segmented to obtain a video frame sequence. Each frame is regarded as an independent image for further processing of each frame. Based on the extracted local contrast enhancement parameter, each image is adaptively divided into blocks to obtain an image block set. The image is divided into multiple regions to facilitate the use of different enhancement strategies for different regions and the flexible adjustment of contrast according to the scene characteristics to obtain better visual effects. The local statistical features of each image block in the image block set are calculated, including local brightness distribution, contrast, and pixel value variance. Based on the local statistical features and the extracted local contrast enhancement parameter, an adaptive S-shaped curve mapping function is constructed. The S-shaped curve mapping function is a nonlinear mapping function that significantly improves the local contrast of the image by adjusting the pixel value distribution of the image, making the dark details of the image clearer. The adaptive S-shaped curve mapping function is applied to the corresponding image block to obtain a local contrast-enhanced image block. Since there are overlapping regions between image blocks, in order to avoid visual discontinuity in these regions after enhancement, the overlapping regions of adjacent image blocks are smoothly transitioned. Through smoothing processing, the stitching marks between different image blocks are effectively eliminated to obtain a contrast-enhanced video frame sequence. Based on the motion compensation parameter, block matching motion estimation is performed on adjacent frames in the video frame sequence to obtain a motion vector field. The motion vector field is important information describing the direction and speed of object motion in the video. The block matching algorithm compares adjacent frames to find the corresponding motion vector, accurately capturing the motion characteristics in the video. Based on the obtained motion vector field, motion compensation interpolation is performed on the contrast-enhanced video frame sequence to generate a frame rate enhanced video sequence. Motion compensation interpolation generates new intermediate frames between the original frames, effectively improving the frame rate of the video, making the object motion smoother and more natural, and reducing motion blur. In order to improve the visual effect of the video, a time domain filter is applied to the frame rate enhanced video sequence to obtain a smooth high frame rate video sequence. The time domain filter smoothes the frame rate enhanced video sequence to reduce artifacts or noise introduced by interpolation, making the transition in the time dimension smoother and more natural. According to the video enhancement level in the interaction response strategy, the global brightness and saturation of the smooth high frame rate video sequence are adjusted. The video enhancement level in the interaction response strategy is set according to the user's individual needs. According to different user preferences, the brightness and saturation of the video frames are adjusted to obtain a high-quality video frame sequence that meets the user's needs.

[0038] S6, scene adaptive color space mapping and dynamic color temperature adjustment are performed on the high-quality video frame sequence according to the real-time acquired ambient light information and the pre-stored display device color characteristics, to obtain an output video stream.

[0039] wherein, the high-quality video frame sequence is color space converted from original RGB color space to HSV color space, to obtain hue (H), saturation (S) and lightness (V) components. RGB color space is mainly used for display devices, while HSV color space is more consistent with human color perception, and has more advantages in tasks such as color enhancement and color temperature adjustment. Ambient light information is collected in real time based on ambient light sensors, including light intensity and color temperature. Light intensity reflects the brightness of the current environment, while color temperature describes the color tone of light, usually in units of Kelvin (K). Through these sensor data, the current environment is classified, for example, the environment is divided into bright, moderate or dim scenes. Different states of ambient light will directly affect the perception of the display picture by the human eye. According to the environmental scene type and the pre-stored display device color characteristics, the corresponding color enhancement coefficients a, b and g are selected from the preset scene-color mapping table. a and b are used for saturation adjustment, while g is used for lightness adjustment. The scene-color mapping table is pre-set based on the visual perception of the human eye in different scenes and the color characteristics of the display device, and is used to select the optimal color enhancement parameters for each scene. For example, in a bright scene, a larger saturation enhancement coefficient a and a higher lightness adjustment parameter g are selected; while in a dim scene, a lower g value is selected to prevent the picture from being overexposed. The lightness V component in the HSV color space is adjusted non-linearly to adapt to the current ambient light changes. The Gamma correction formula wherein is the original lightness component, is the lightness adjustment parameter selected from the mapping table according to the current environment. Through Gamma correction, non-linear enhancement of lightness is realized, and the overall brightness level of the picture is controlled, so that the picture can maintain good visibility in different light environments. For the saturation S component, linear adjustment is performed based on the color enhancement coefficients a and b. The calculation formula of the adjusted saturation is wherein is the original saturation, is the amplification coefficient for adjusting the saturation, is the saturation offset. Through linear adjustment, the saturation of the picture is flexibly controlled, so that the color is more vivid or soft. According to the color temperature information of the environment, the offset of the hue is calculated, and the original hue H component is adjusted. The adjusted hue component is calculated by the formula wherein is the original hue component, is the offset calculated based on the ambient color temperature. The higher the color temperature, the more blue the light appears, and the lower the more red. By appropriately shifting the hue, the picture maintains a more natural color balance at different color temperatures. The adjusted hue component , saturation component and lightness component are combined together and subjected to a color space inverse transformation from HSV to RGB to obtain the color-enhanced video frame. After being transformed back to the RGB color space, the video frame is correctly decoded and presented by the display device. To ensure that the color depth of the video frame matches the supported bit depth of the display device, a Floyd-Steinberg dithering algorithm is applied to the color-enhanced video frame. The role of the dithering algorithm is to reduce the color banding effect caused by quantization error when reducing the color depth, so that the color looks more smooth and natural. For example, if the display device can only support 8-bit color depth, and the video frame is originally 10-bit color depth, the dithering algorithm reduces the color depth to the bit depth supported by the display device without significantly reducing the visual quality. The output video frame sequence is encoded into a video stream in H.264 format. H.264 is a standard applied to video compression, which provides higher video quality at a smaller code rate. During the encoding process, the bit rate and key frame interval for network transmission are set to ensure the transmission quality of the video under different network conditions. For example, in a network environment with limited bandwidth, a lower bit rate and a larger key frame interval are selected to reduce the amount of data transmitted; while in a bandwidth sufficient case, a higher bit rate and a smaller key frame interval are selected to obtain higher video quality and better user experience.

[0040] In one example, the input video stream of the TV box is subjected to feature analysis to obtain a video feature data structure, including:

[0041] The input video stream of the TV box is subjected to frame-level segmentation to obtain a video frame sequence, and each frame in the video frame sequence is subjected to spatial domain analysis to obtain spatial feature parameters including resolution, color space, brightness distribution, contrast and saturation;

[0042] Difference operation is performed on adjacent frames in the video frame sequence to obtain an inter-frame difference map, and motion vector is calculated based on the inter-frame difference map to obtain motion feature parameters;

[0043] Two-dimensional discrete Fourier transform is performed on the video frame sequence to obtain frequency domain feature parameters, and the spatial feature parameters, motion feature parameters and frequency domain feature parameters are combined to construct a multi-dimensional feature vector;

[0044] Adaptive principal component analysis is performed on the multi-dimensional feature vector to obtain a reduced dimension feature vector, and high-level semantic features corresponding to the reduced dimension feature vector are extracted;

[0045] The high-level semantic features, spatial feature parameters, motion feature parameters, and frequency domain feature parameters are weighted and fused by an attention mechanism to obtain a video feature data structure.

[0046] In this example, the input video stream of the TV box is frame-level segmented to obtain a video frame sequence. The video stream is split into individual frames, so that each frame is independently analyzed and processed. Spatial domain analysis is performed on each frame in the video frame sequence to obtain spatial feature parameters including resolution, color space, brightness distribution, contrast, and saturation. Resolution describes the number of pixels in the image; color space refers to the color representation in the image, commonly RGB and HSV color spaces; brightness distribution represents the distribution of brightness values in the image, which is an important parameter for measuring the overall light of the picture; contrast reflects the difference between light and dark parts of the image; and saturation represents the purity or intensity of color. By calculating and extracting these parameters, the spatial feature description of each frame is obtained. Difference operation is performed on adjacent frames in the video frame sequence to obtain inter-frame difference maps. Difference operation can capture the motion of objects in the video, and by comparing the pixels of adjacent frames, the inter-frame change area, i.e. the inter-frame difference map, is obtained. The inter-frame difference map can reflect the position and range of the motion of the object in the video, and the motion vector is calculated based on the difference map. The motion vector is a vector representing the direction and speed of the object in the video, which is obtained by block matching algorithm or optical flow method. The motion feature parameter is a description of the dynamic information in the video. Frequency domain analysis is performed on the video frame sequence to extract frequency features. By performing two-dimensional discrete Fourier transform (2D-DFT) on the video frame, the video frame is converted from the spatial domain to the frequency domain, and the frequency domain feature parameter is obtained. The spatial information in the image is decomposed into different frequency components, where the low-frequency part represents the overall structure of the image, and the high-frequency part represents the details and edge information of the image. The formula of two-dimensional discrete Fourier transform is:

[0047] ;

[0048] wherein, represents the amplitude in the frequency domain, represents the pixel value in the spatial domain, and are the width and height of the image, respectively, and is the frequency index. By Fourier transform, the contribution of different frequency components in the image is analyzed, and the frequency domain feature parameters are extracted to describe the texture and edge features of the image. The spatial feature parameters, motion feature parameters and frequency domain feature parameters are combined to construct a multi-dimensional feature vector. The multi-dimensional feature vector is processed by dimension reduction. The adaptive principal component analysis method is used to reduce the dimension of the multi-dimensional feature vector. Principal component analysis is a statistical technique for data dimensionality reduction, which projects the original high-dimensional data onto a low-dimensional space by linear transformation, thereby retaining the most important feature components in the data. The dimensionality reduction process of principal component analysis is represented by the following formula:

[0049] ;

[0050] wherein, represents the reduced feature vector, represents the original multi-dimensional feature vector, is the principal component matrix. By adaptively selecting the principal components, the principal component analysis method significantly reduces the dimension of the feature vector while ensuring that the main features of the data are not lost, making subsequent calculations more efficient. The high-level semantic features corresponding to the reduced feature vector are extracted. High-level semantic features are a higher-level understanding of video content, such as scene categories, object categories, and user interaction behaviors, etc. By clustering analysis or deep neural network feature extraction on the reduced feature vector, high-level semantic features are obtained. The high-level semantic features, spatial feature parameters, motion feature parameters and frequency domain feature parameters are weighted and fused by attention mechanism. Attention mechanism is a mechanism for selecting important information in deep learning, which assigns different weights to each part of the feature. Attention mechanism can adaptively adjust the importance of each type of feature according to the needs of the task. The calculation of the attention mechanism is represented by the following formula:

[0051] ;

[0052] wherein, represents the weighted fused feature, , and are the matrix representations of the query, key and value respectively, is the dimension of the feature vector. By calculating the attention weight, the most important features for the task are effectively highlighted, and the video feature data structure is obtained.

[0053] In one example, according to the video feature data structure, the input video stream is divided into overlapping image blocks, processed independently and weighted and reorganized to obtain an enhanced video stream, including:

[0054] According to the resolution parameter in the video feature data structure, each frame in the input video stream is grid-divided to obtain NxM basic image blocks, and the NxM basic image blocks are expanded by 50% overlap to obtain (2N-1)x(2M-1) overlapping image blocks, each of which has a size of 2 times that of the original basic image block;

[0055] According to the brightness distribution parameter in the video feature data structure, each overlapping image block is adaptively histogram equalized to obtain a brightness-equalized overlapping image block;

[0056] Based on the frequency domain feature parameter in the video feature data structure, the brightness-equalized overlapping image block is wavelet transformed to extract multi-scale features, and the high-frequency sub-band is soft threshold denoised to obtain a denoised overlapping image block;

[0057] According to the motion feature parameter in the video feature data structure, the denoised overlapping image block is motion compensated, the motion vector is calculated through a block matching algorithm, and sub-pixel level interpolation is performed to obtain a motion-compensated overlapping image block;

[0058] Based on the contrast parameter in the video feature data structure, a local contrast enhancement algorithm is applied to the motion-compensated overlapping image block, the pixel intensity is adjusted through adaptive gamma correction to obtain a contrast-enhanced overlapping image block;

[0059] The contrast-enhanced overlapping image block is weighted to obtain a weighted overlapping image block, and the weighted overlapping image block is pixel-level weighted averaged in the overlap region according to the original position to reconstruct a complete video frame. The reconstructed video frame sequence is combined to obtain an enhanced video stream.

[0060] In this example, according to the resolution parameter in the video feature data structure, each frame in the input video stream is grid-divided. For each frame image, the grid is divided according to to obtain basic image blocks. In order to reduce the block effect and achieve better information fusion between image blocks, each basic image block is expanded by 50% overlap to obtain , the size of each overlapping image block is twice the size of the original basic image block. The way of overlapping expansion ensures that there will be no obvious visual boundary at the image block boundary when image enhancement is performed, thereby improving the visual quality of the overall video. The overlapping image block is subjected to brightness equalization processing. Based on the brightness distribution parameters in the video feature data structure, adaptive histogram equalization is performed on each overlapping image block. Histogram equalization is an algorithm for improving the brightness distribution of an image, which redistributes the gray levels of the image so that the overall brightness of the image is more uniform and natural. According to the local brightness histogram of each image block, an adaptive method is applied to map the pixel values, so that the low brightness part of the image block is enhanced and the high brightness part is suppressed, and the brightness equalized overlapping image block is obtained. Multi-scale features are extracted from the image block. Based on the frequency domain feature parameters in the video feature data structure, wavelet transform is performed on the brightness equalized overlapping image block. Wavelet transform is an analysis method that decomposes an image into different scale and frequency components, which can retain the information in time and frequency domains at the same time, and better capture the edge and detail features in the image through multi-scale analysis. The formula of wavelet transform is:

[0061] ;

[0062] wherein, represents the wavelet transform coefficient, is the brightness value of the input image, is the wavelet basis function. After completing the wavelet decomposition, the image is decomposed into different frequency subbands, in which the high-frequency subband contains the edge information and noise of the image. In order to eliminate the high-frequency noise in the image, soft threshold denoising is performed on the high-frequency subband. Soft threshold denoising reduces noise by attenuating coefficients less than a threshold value, while retaining the main edge information of the image, and obtains the denoised overlapping image block. According to the motion feature parameters in the video feature data structure, motion compensation is performed on the denoised overlapping image block. Motion compensation reduces the blurring phenomenon caused by object motion by estimating the motion between adjacent frames. In this process, block matching algorithm is used to match adjacent frames to calculate the motion vector. is represented as:

[0063] ;

[0064] wherein, represents the motion vector of the image block at the current position, is the motion offset, and are the width and height of the image block, respectively, and respectively represent the pixel values of the current frame and the previous frame. By finding the offset that minimizes the inter-frame difference, the motion vector of the image block is obtained, and then the motion is compensated using sub-pixel level interpolation technology to obtain the motion compensated overlapping image block. This processing can significantly reduce the blurring phenomenon caused by object motion, making the dynamic scene in the video appear more smooth and natural. After completing the motion compensation, according to the contrast parameter in the video feature data structure, a local contrast enhancement algorithm is applied to the motion compensated overlapping image block. Local contrast enhancement adjusts the pixel intensity through adaptive gamma correction, making the details of the image more prominent. The adjusted pixel value is calculated by the following formula:

[0065] ;

[0066] wherein, is the original pixel value, is the adaptive contrast enhancement parameter. By adjusting the pixel in a nonlinear manner, the details of the dark and bright parts are enhanced, making the contrast of the image more obvious, thereby improving the visual effect. The overlapping image block after contrast enhancement processing is weighted to fuse the information between the image blocks, obtaining the weighted overlapping image block. In order to ensure the visual consistency between the overlapping image blocks when reconstructed, the weighted overlapping image block is pixel-level weighted average according to the original position of the overlapping region. Through pixel-level weighted average, the splicing traces generated by independent processing in the overlapping region of adjacent image blocks are eliminated, ensuring that the reconstructed video frame is visually consistent and smooth. All the reconstructed video frame sequences are combined to obtain the final enhanced video stream.

[0067] In one example, the intelligent interaction control unit collects visual and voice interaction information of the user, and inputs the visual and voice interaction information into a double-branch Transformer model for multi-modal information fusion to obtain a fusion feature vector, including:

[0068] The intelligent interaction control unit pre-processes the collected visual information of the user to obtain a visual input sequence, and pre-processes the collected voice information of the user to obtain a voice input sequence;

[0069] The visual input sequence is input into a visual branch Transformer encoder in the double-branch Transformer model, and the visual branch Transformer encoder includes 6 encoding layers, each encoding layer including a multi-head self-attention sublayer and a feedforward neural network sublayer, and a visual feature representation is obtained by calculation;

[0070] The speech input sequence is input into a speech branch Transformer encoder in the dual-branch Transformer model. The speech branch Transformer encoder includes 6 encoding layers, each of which has the same structure as the visual branch. The speech feature representation is obtained by calculation.

[0071] The visual feature representation and the speech feature representation are positionally encoded to inject position information into the feature representations, obtaining an enhanced visual feature representation and an enhanced speech feature representation.

[0072] The enhanced visual feature representation and the enhanced speech feature representation are input into a cross-attention layer to realize the interaction of visual and speech information by calculating the attention weights between modalities, obtaining a preliminary fusion feature.

[0073] The preliminary fusion feature is input into a fusion Transformer encoder in the dual-branch Transformer model. The fusion Transformer encoder includes 4 encoding layers, each of which includes a multi-head self-attention sublayer, a cross-attention sublayer, and a feedforward neural network sublayer. The deep fusion feature is obtained by calculation.

[0074] The global average pooling operation is performed on the deep fusion feature to compress the sequence feature into a fixed-dimensional vector representation, obtaining a fusion feature vector.

[0075] In this example, the collected visual and speech information is preprocessed to obtain a visual input sequence and a speech input sequence. In the preprocessing of visual information, the user's visual data such as facial images and posture information are standardized by image denoising, scale normalization, image enhancement, and other operations to obtain a structured visual input sequence. For speech information, the preprocessing steps include removing background noise, normalizing volume, extracting speech features (such as Mel frequency cepstral coefficients, MFCC), etc., to obtain a processed speech input sequence. The visual input sequence is input into a visual branch Transformer encoder in the dual-branch Transformer model. The visual branch Transformer encoder includes 6 encoding layers, each of which is composed of a multi-head self-attention sublayer and a feedforward neural network sublayer. The multi-head self-attention sublayer is used to capture the relationship between different parts of the visual input sequence. Through the self-attention mechanism, the input data is weighted calculated, so that the model can focus on important information in the sequence and ignore irrelevant content. The calculation of multi-head self-attention is represented as:

[0076] ;

[0077] wherein, , and matrixes representing Query, Key and Value respectively, is the dimension of Key. Through this mechanism, the visual branch encoder is able to extract important visual features in each frame of image, obtaining a semantic-rich visual feature representation. The feed-forward neural network sublayer then performs a nonlinear mapping on the output of the self-attention sublayer to enhance the expressiveness of the features. These steps are calculated layer by layer through 6 encoding layers, obtaining a visual feature representation containing rich semantics of user visual information. At the same time, the speech input sequence is input into the speech branch Transformer encoder in the dual-branch Transformer model. The structure of the speech branch Transformer encoder is the same as the visual branch, also containing 6 encoding layers, each of which is composed of a multi-head self-attention sublayer and a feed-forward neural network sublayer. Multi-head self-attention is used to model the global dependence of speech features to capture the relevance between time steps in the speech sequence. The feed-forward neural network sublayer then performs a nonlinear transformation on the speech features to enhance their semantic expressiveness. After layer-by-layer processing through 6 encoding layers, a speech feature representation is obtained, containing various information in the user's speech, such as speech content, emotion, tone, etc. The visual and speech feature representations are positionally encoded to inject position information into the feature representations. The purpose of position encoding is to enable the Transformer model to understand the relative relationship of each position in the sequence without explicit order structure. Position encoding is calculated by the following formula:

[0078] ;

[0079] where pos represents the position, is the dimension of the feature, Total dimension of the feature representation. Through positional encoding, the data at each time step in the visual feature representation and the speech feature representation contains position information, enabling the model to better understand the structure and context of the sequence. The enhanced visual feature representation and the enhanced speech feature representation are input into the cross-attention layer, and the information interaction between modalities is realized by calculating the attention weights between visual and speech modalities. In the cross-attention layer, the visual features and the speech features pay attention to each other, thereby capturing the potential association between visual and speech. For example, when the user is speaking, the emotion in the speech is corresponded to the user's facial expression through cross-attention, thereby enhancing the system's understanding of the user's current state. Through the calculation of the cross-attention layer, a preliminary fusion feature is obtained. The preliminary fusion feature is input into the fusion Transformer encoder in the dual-branch Transformer model to realize the deep fusion of multi-modal information. The fusion Transformer encoder contains 4 encoding layers, each of which is composed of a multi-head self-attention sublayer, a cross-attention sublayer, and a feedforward neural network sublayer. The multi-head self-attention sublayer is used to capture the dependency between different time steps in the fusion feature, so that the information can be effectively integrated in time; the cross-attention sublayer strengthens the fusion between visual and speech, ensuring that the information between different modalities can understand and enhance each other. The feedforward neural network sublayer performs nonlinear mapping on the fusion feature to enhance the expression ability of the feature. After layer-by-layer processing of the 4 encoding layers, a deep fusion feature is obtained. The deep fusion feature is subjected to a global average pooling operation. By averaging the time dimension of the feature sequence, a fixed-size vector representation is obtained. Assuming that the deep fusion feature is , the global average pooling is represented as:

[0080] ;

[0081] wherein, is the length of the feature sequence, is the feature representation at the th time step, is the fusion feature vector after global average pooling. Through the global average pooling operation, the deep fusion feature is compressed into a fixed-size vector, which contains a comprehensive description of the visual and speech modality information.

[0082] In one example, the fusion feature vector and the enhanced video stream are jointly analyzed to obtain the user interaction intention and the interaction response strategy, including:

[0083] The enhanced video stream is subjected to spatio-temporal feature extraction to obtain a video spatio-temporal feature map, and the fusion feature vector is subjected to dimension expansion by copying and splicing operations to expand the fusion feature vector to the same spatio-temporal dimension as the video spatio-temporal feature map, obtaining an expanded fusion feature map;

[0084] The video spatio-temporal feature map and the extended fusion feature map are fused, and a joint feature map is obtained through element-by-element addition operation, and based on the joint feature map, spatial attention weights and channel attention weights are calculated to obtain an attention-enhanced joint feature map;

[0085] The attention-enhanced joint feature map is modeled in time sequence to obtain a time sequence encoding feature, and the time sequence encoding feature is input into a multi-task learning head including an intent classification branch and a response generation branch, and a user interaction intent probability distribution is obtained through a full connection layer and a softmax function;

[0086] Based on the user interaction intent probability distribution, the user interaction intent is determined through threshold judgment, and the corresponding interaction response strategy template is selected according to the pre-defined intent-response mapping table;

[0087] The time sequence encoding feature and the interaction response strategy template are input into a strategy refinement network to generate specific interaction response strategy parameters, and an interaction response strategy is obtained.

[0088] In this example, spatio-temporal feature extraction is performed on the video, capturing both temporal and spatial information in the video, allowing the system to understand dynamic changes and static scenes in the video. Spatio-temporal feature extraction is achieved through the combination of convolutional neural networks (CNN) and long short-term memory networks (LSTM), with CNN responsible for extracting spatial features and LSTM responsible for modeling temporal dependencies to obtain a video spatio-temporal feature map. The spatio-temporal feature map is a representation of objects, actions, and scene changes in the video, containing spatial information about each frame of image and temporal change information across frames. The fusion feature vector is dimensionally expanded through replication and concatenation operations to expand it to the same spatio-temporal dimensions as the video spatio-temporal feature map. In the feature fusion stage, element-by-element addition operation is performed on the video spatio-temporal feature map and the expanded fusion feature map to obtain a joint feature map. Element-by-element addition operation realizes the comprehensive representation of two kinds of features by adding the feature values at corresponding positions, so that the joint feature map contains both the spatio-temporal feature information of the video and the multi-modal interaction information of the user. Based on the joint feature map, spatial attention weights and channel attention weights are calculated to obtain an attention-enhanced joint feature map. Spatial attention weights are used to highlight important spatial regions in the video, such as the user's gaze or the location of significant objects; while channel attention weights enhance or suppress specific feature dimensions by assigning different channel weights. The calculation of spatial attention is represented as:

[0089] ;

[0090] wherein, represents the attention weight at position , Indicates the joint feature map at location Features This is a mapping function used to calculate attention weights, such as a simple linear transformation. Through the attention mechanism, the system focuses more on important regions or features relevant to user interaction, resulting in more accurate feature representations. Temporal modeling is performed on the attention-enhanced joint feature map to capture the changes in the feature sequence over time, yielding temporally encoded features. Temporal modeling uses recurrent neural networks (RNNs) or Transformer structures to ensure the system can understand dynamic changes in the video. By linking features at different time steps through temporal modeling, a comprehensive understanding of the entire interaction process is obtained. Temporally encoded features contain information about temporal changes in the video, as well as the user's behavior and intent throughout the interaction. The temporally encoded features are input into a multi-task learning head. The multi-task learning head includes an intent classification branch and a response generation branch, where the intent classification branch predicts the user's interaction intent, and the response generation branch generates the corresponding interaction response strategy. Intent classification is implemented through fully connected layers and a softmax function. Fully connected layers map the temporally encoded features to a predefined intent category space, while the softmax function normalizes the score for each category, obtaining the probability distribution of the user's interaction intent. The formula for calculating the intent probability distribution is:

[0091] ;

[0092] in, Indicates the first The probability of an intention. It is the output score of the fully connected layer. This represents the total number of intent categories. The probability of each intent category is obtained using the softmax function, thus determining the most likely user interaction intent. Based on the probability distribution of user interaction intents, a set threshold is used to determine the final user interaction intent. For example, when the probability of a certain intent exceeds a preset threshold, that intent is determined to be the user's current intent. After determining the user interaction intent, an appropriate interaction response strategy template is selected according to a predefined intent-response mapping table. The intent-response mapping table is a mapping relationship that maps user interaction intents to specific response strategies, ensuring that different user intents trigger appropriate system responses. After selecting the appropriate interaction response strategy template, the temporal encoding features and the interaction response strategy template are input into a strategy refinement network to generate specific interaction response strategy parameters. The strategy refinement network is a neural network structure that can generate detailed response parameters that meet the user's current needs based on the input of temporal encoding features and response strategy templates. Through the strategy refinement network, high-level response intents are concretized into executable response strategies, such as adjusting playback content, switching interfaces, and selecting audio feedback, ultimately resulting in the interaction response strategy.

[0093] In one example, based on the user interaction intention and the interaction response strategy, the enhanced video stream is subjected to local contrast enhancement and motion compensation to obtain a high-quality video frame sequence, including:

[0094] According to the user interaction intention, corresponding local contrast enhancement parameters and motion compensation parameters are extracted from a preset intention-processing parameter mapping table;

[0095] The enhanced video stream is subjected to frame-level segmentation to obtain a video frame sequence, and each frame of image is subjected to adaptive blocking based on the local contrast enhancement parameters to obtain an image block set;

[0096] Local statistical features are calculated for each image block in the image block set, and an adaptive S-shaped curve mapping function is constructed according to the local statistical features and the local contrast enhancement parameters;

[0097] The adaptive S-shaped curve mapping function is applied to the corresponding image block to obtain a local contrast-enhanced image block, and the overlapping areas of adjacent image blocks are subjected to smooth transition to obtain a contrast-enhanced video frame sequence;

[0098] Based on the motion compensation parameters, block matching motion estimation is performed on adjacent frames in the contrast-enhanced video frame sequence to obtain a motion vector field, and the contrast-enhanced video frame sequence is subjected to motion compensation interpolation according to the motion vector field to generate a frame rate enhanced video sequence;

[0099] A time domain filter is applied to the frame rate enhanced video sequence to obtain a smooth high frame rate video sequence, and global brightness and saturation adjustment is performed on the smooth high frame rate video sequence according to the video enhancement level in the interaction response strategy to obtain a high-quality video frame sequence.

[0100] In this example, according to the user's interaction intention, corresponding local contrast enhancement parameters and motion compensation parameters are extracted from a preset intention-processing parameter mapping table. The preset mapping table defines suitable image processing parameters according to different user interaction intentions, so as to provide the best viewing experience for the user. The local contrast enhancement parameters are used to adjust the detail performance of the image, while the motion compensation parameters are used to improve the smoothness of the video and reduce motion blur. The enhanced video stream is subjected to frame-level segmentation to obtain a video frame sequence. The continuous video signal is divided into individual frames for independent processing and analysis of each frame. Based on the extracted local contrast enhancement parameters, each frame of image is subjected to adaptive blocking to obtain an image block set. Assuming that each frame of image is divided into image blocks, which are used for subsequent local contrast enhancement processing. For each image block, its local statistical features, such as brightness mean, standard deviation, contrast, etc., are calculated. Local statistical features are indicators used to describe the distribution and variation of pixel values in an image block, which can reflect the brightness, texture, and contrast of the image block. Based on the local statistical features and the extracted local contrast enhancement parameters, an adaptive S-shaped curve mapping function is constructed. The S-shaped curve mapping function is a nonlinear function used to adjust the distribution of pixel values in an image block to improve local contrast and detail performance. The adaptive S-shaped curve mapping function is represented as:

[0101] ;

[0102] wherein, represents the original pixel value at position in the image block, represents the enhanced pixel value, is the contrast enhancement parameter used to control the slope of the S-shaped curve, represents the brightness mean of the image block. Through nonlinear mapping, low and high brightness pixel values are compressed, while intermediate brightness pixels are stretched, thereby improving the local contrast of the image and making the details more obvious. The adaptive S-shaped curve mapping function is applied to the corresponding image block to obtain the local contrast-enhanced image block. To ensure smooth transitions between image blocks and avoid obvious stitching marks, smooth transition processing is performed on the overlapping areas of adjacent image blocks. Smooth transition is achieved through a weighted average method, i.e., the pixel values from different image blocks in the overlapping area are weighted and summed to eliminate boundary effects, resulting in a contrast-enhanced video frame sequence. Based on the motion compensation parameters, block matching motion estimation is performed on adjacent frames in the contrast-enhanced video frame sequence to obtain a motion vector field. The motion vector field is a feature map that describes the direction and speed of object motion in the video, calculated through a block matching algorithm. The block matching algorithm compares an image block in the current frame with image blocks at different positions in the previous frame, finds the most similar position, and determines the motion vector of the image block. The calculation of the motion vector is represented as:

[0103] ;

[0104] wherein, represents the motion vector of the image block at the current position, is the motion offset, and are the width and height of the image block, and respectively. By finding the offset that minimizes the inter-frame difference, the motion vector is obtained, and then motion compensation interpolation is performed on the video frames to generate the frame rate upsampled video sequence. Motion compensation interpolation generates new intermediate frames between adjacent frames, increasing the frame rate of the video, making the motion of the video more smooth and natural, and reducing the blur phenomenon caused by the rapid movement of objects. A temporal filter is applied to the frame rate upsampled video sequence to obtain a smooth high frame rate video sequence. The temporal filter smooths the temporal changes in the video frame sequence to eliminate artifacts and noise introduced by interpolation or other processing steps, thereby ensuring that the transition of the video in the time dimension is more natural and smooth. The temporal filter takes the form of a weighted moving average, and its calculation formula is:

[0105] ;

[0106] wherein, represents the frame after temporal filtering, represents the original frame at time , and is the size of the filter window. By weighted averaging of the current frame and several frames before and after it, the discontinuity between video frames is eliminated, and a smooth high frame rate video sequence is obtained. According to the video enhancement level in the interactive response strategy, the smooth high frame rate video sequence is adjusted in global brightness and saturation to obtain the final high quality video frame sequence. The video enhancement level is set according to the user's individual needs, including different brightness and saturation adjustment strategies, to adapt to different viewing environments and preferences. The adjustment of global brightness and saturation is realized by the following formula:

[0107] ;

[0108] wherein, represents the original pixel value, represents the adjusted pixel value, is a saturation adjustment coefficient that controls the intensity of color, is a brightness offset that controls the overall brightness of the picture. Through linear adjustment, the brightness and color performance of the video are flexibly adjusted to ensure that the final video output meets the user's viewing needs.

[0109] In one example, scene adaptive color space mapping and dynamic color temperature adjustment are performed on the high quality video frame sequence according to the real-time acquired ambient light information and the pre-stored display device color characteristics to obtain an output video stream, including:

[0110] Color space conversion is performed on the high quality video frame sequence to convert the RGB color space to the HSV color space to obtain hue H, saturation S and lightness V components;

[0111] The ambient light sensor is used to collect ambient light information in real time, including ambient light intensity and color temperature, and the current environment is classified as bright, moderate or dim scene according to a preset ambient light threshold, so as to obtain an ambient scene type;

[0112] According to the ambient scene type and the pre-stored display device color characteristics, corresponding color enhancement coefficients α, β and γ are selected from a preset scene-color mapping table;

[0113] The luminance V component is nonlinearly adjusted, and the formula V'=V γ Gamma correction is performed to obtain an adjusted luminance component V';

[0114] Based on the color enhancement coefficients α and β, the saturation S component is linearly adjusted, and the formula S'=α*S+β is used to obtain an adjusted saturation component S';

[0115] According to the ambient color temperature information, a hue offset ΔH is calculated, and the hue H component is adjusted, and the formula H'=H+ΔH is used to obtain an adjusted hue component H';

[0116] The adjusted hue component H', saturation component S' and luminance component V' are combined, and HSV to RGB color space inverse transformation is performed to obtain a color-enhanced video frame;

[0117] The Floyd-Steinberg dithering algorithm is applied to the color-enhanced video frame, the color depth is adjusted to the bit depth supported by the display device, an output video frame sequence is obtained, the output video frame sequence is encoded into a video stream in H.264 format, the bit rate and key frame interval of network transmission are set, and an output video stream is obtained.

[0118] In this example, a sequence of high-quality video frames is converted from the RGB color space to the HSV color space. The RGB color space representation is more direct for machines and display devices, but the HSV color space is more aligned with human perception of color and is converted to HSV for color adjustment. The HSV color space includes three components: hue (H), saturation (S), and value (V), where hue represents the type of color, such as red, green, blue, etc., saturation represents the purity of the color, and value represents the brightness or intensity of the color. Based on the real-time environmental light information collected by the ambient light sensor, including ambient light intensity and color temperature, and according to the pre-set ambient light threshold, the current environment is classified as bright, moderate, or dim scene, obtaining the environmental scene type. For example, when the ambient light sensor detects a higher light intensity, the current environment is classified as a bright scene, while when the light intensity is lower, it is classified as a dim scene. According to the different light conditions of the environment, the video content on the display device is intelligently adjusted to ensure that the user can have the best viewing experience in various light conditions. According to the environmental scene type and the pre-stored display device color characteristics, the corresponding color enhancement coefficients , β, and γ are selected from the pre-set scene-color mapping table. These color enhancement coefficients are used to adjust the components of the HSV color space to adapt to the current environmental light conditions. For bright scenes, a larger value of the value enhancement coefficient is selected to ensure that the image is still clear in strong light, while in dim scenes, an appropriate saturation enhancement coefficient is selected to improve the color performance of the picture, making the picture appear more vivid and vibrant. The value V component is adjusted non-linearly using the Gamma correction formula. The formula for Gamma correction is:

[0119] ;

[0120] where, represents the original value component, represents the adjusted value component, is the Gamma correction coefficient selected from the scene-color mapping table. Gamma correction adjusts the brightness non-linearly to adapt to different display environments. By adjusting the value of , different degrees of adjustment of light and dark parts are achieved. For example, when watching a video in a dim environment, a lower value is selected to enhance the performance of the bright part, so that the image is clearer in a dark environment; while in a strong light environment, a higher value is selected to prevent the image from being too bright and to maintain details. After completing the value adjustment, the saturation S component is linearly adjusted using the following formula:

[0121] ;

[0122] in, Represents the original saturation component. This represents the adjusted saturation component. and These are the saturation enhancement factor and offset selected from the scene-color map. Linear adjustment makes the colors in the image more saturated or softer by changing the saturation value. For example, in a dark environment, increasing... The saturation value is increased to make the colors more vibrant, thus compensating for the visual color loss caused by insufficient ambient light. In brighter environments, the saturation is reduced to prevent the colors from being too glaring and to ensure visual comfort. The hue shift is calculated based on the color temperature data from the ambient light information. The hue H component is then adjusted to obtain the adjusted hue component. The hue adjustment formula is:

[0123] ;

[0124] in, Represents the original hue components. This indicates the adjusted hue components. This represents the hue shift, calculated based on the ambient color temperature. For example, when the ambient color temperature is high and the light appears cool, the hue shift is reduced... The value is used to make the image hue warmer, thus neutralizing the visual discomfort caused by cool light; while when the ambient color temperature is low, the light is warmer, which increases... The value of makes the image hue cooler, in order to maintain color balance and naturalness. The adjusted hue components... saturation component and brightness components The video frames are color enhanced by recombining and performing HSV to RGB color space inverse transformation. The color performance of the video is adaptively adjusted according to the current ambient light conditions, making the output video content more consistent with human visual perception and providing a better viewing experience. Floyd-Steinberg dithering algorithm is applied to the color-enhanced video frames to adjust the color depth to the bit depth supported by the display device. Floyd-Steinberg dithering is an image processing algorithm used to reduce visual artifacts introduced by quantization when reducing color depth by propagating errors between pixels, making the image visually smoother and more natural. For example, if the display device can only support 8-bit color depth, and the original video frame is 10-bit color depth, the 10-bit color depth video frame is effectively converted to 8-bit color depth through the dithering algorithm, while the visual quality of the image is not significantly affected. After completing the color depth adjustment, the output video frame sequence is encoded into a video stream in H.264 format. H.264 is a coding standard applied to video compression, which can achieve efficient compression while ensuring high image quality. The bit rate and key frame interval for network transmission are set during the encoding process to ensure smooth transmission of video content under different network conditions. For example, in the case of limited bandwidth network conditions, a lower bit rate and longer key frame interval are selected to reduce the burden of data transmission; while in the case of sufficient bandwidth, a higher bit rate and shorter key frame interval are selected to provide higher image quality and better user experience.

[0125] Referring to Figure 2 The embodiment provides a television box intelligent interaction control unit image processing device, which comprises:

[0126] A feature analysis module 1 is configured to perform feature analysis on the input video stream of the television box to obtain a video feature data structure.

[0127] A processing module 2 is configured to perform overlapping image block division, independent processing and weighted reorganization on the input video stream according to the video feature data structure to obtain an enhanced video stream.

[0128] A fusion module 3 is configured to collect visual and voice interaction information of a user based on the intelligent interaction control unit, and input the visual and voice interaction information into a double-branch Transformer model to perform multi-modal information fusion and obtain a fusion feature vector.

[0129] A joint analysis module 4 is configured to perform joint analysis on the fusion feature vector and the enhanced video stream to obtain a user interaction intention and an interaction response strategy.

[0130] A compensation module 5 is configured to perform local contrast enhancement and motion compensation on the enhanced video stream based on the user interaction intention and the interaction response strategy to obtain a high-quality video frame sequence.

[0131] The adjusting module 6 is configured to perform scene adaptive color space mapping and dynamic color temperature adjustment on the high-quality video frame sequence according to the real-time acquired ambient light information and the pre-stored display device color characteristics, to obtain an output video stream.

[0132] In the embodiment, the specific implementation of each unit in the device embodiment is described above in the method embodiment, and will not be described here.

[0133] With reference to Figure 3 , the embodiment of the present application also provides a computer device, which can be a server, and the internal structure thereof can be as shown in Figure 3 . The computer device comprises a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store the corresponding data in the embodiment. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement the above method.

[0134] Those skilled in the art can understand that Figure 3 the structure shown in the embodiment is only a block diagram of part of the structure related to the present application scheme, and does not constitute a limitation on the computer device to which the present application scheme is applied.

[0135] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by the processor to implement the above method. It can be understood that the computer readable storage medium in the embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium provided by the present application and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM, etc.

[0137] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, device, article or method that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, device, article or method. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, device, article or method that includes the element.

[0138] The above description is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, based on the content of the present application specification and drawings, are also included in the patent protection scope of the present application.

Claims

1. A television box intelligent interaction control unit image processing method, characterized in that, The method comprises the following steps: characteristic analysis is performed on the input video stream of the television box to obtain a video feature data structure; Specifically, the input video stream of the television box is frame-level segmented to obtain a video frame sequence, and spatial domain analysis is performed on each frame in the video frame sequence to obtain spatial feature parameters including resolution, color space, brightness distribution, contrast, and saturation; difference operation is performed on adjacent frames in the video frame sequence to obtain an inter-frame difference map, and a motion vector is calculated based on the inter-frame difference map to obtain motion feature parameters; two-dimensional discrete Fourier transform is performed on the video frame sequence to obtain frequency domain feature parameters, and the spatial feature parameters, the motion feature parameters, and the frequency domain feature parameters are combined to construct a multi-dimensional feature vector; adaptive principal component analysis is performed on the multi-dimensional feature vector to obtain a reduced feature vector, and high-level semantic features corresponding to the reduced feature vector are extracted; the high-level semantic features, the spatial feature parameters, the motion feature parameters, and the frequency domain feature parameters are weighted and fused through an attention mechanism to obtain a video feature data structure; According to the video feature data structure, the input video stream is divided into overlapping image blocks, independently processed, and weighted and reorganized to obtain an enhanced video stream; specifically, according to the resolution parameter in the video feature data structure, each frame in the input video stream is grid-divided to obtain N×M basic image blocks, and the N×M basic image blocks are expanded by 50% overlap to obtain (2N-1)×(2M-1) overlapping image blocks, each of which has a size of 2 times that of the original basic image block; according to the brightness distribution parameter in the video feature data structure, each overlapping image block is adaptively histogram equalized to obtain a brightness-equalized overlapping image block; based on the frequency domain feature parameters in the video feature data structure, the brightness-equalized overlapping image block is wavelet transformed to extract multi-scale features, and the high-frequency sub-band is soft threshold denoised to obtain a denoised overlapping image block; according to the motion feature parameters in the video feature data structure, the denoised overlapping image block is motion compensated, the motion vector is calculated through a block matching algorithm, and sub-pixel level interpolation is performed to obtain a motion-compensated overlapping image block; based on the contrast parameter in the video feature data structure, a local contrast enhancement algorithm is applied to the motion-compensated overlapping image block to adjust the pixel intensity through adaptive gamma correction to obtain a contrast-enhanced overlapping image block; the contrast-enhanced overlapping image block is weighted to obtain a weighted overlapping image block, and the weighted overlapping image block is pixel-level weighted averaged in the overlap region according to the original position to reconstruct a complete video frame, and the reconstructed video frame sequence is combined to obtain an enhanced video stream; The visual and voice interaction information of the user is collected based on the intelligent interaction control unit, and the visual and voice interaction information is input into a double-branch Transformer model for multi-modal information fusion to obtain a fusion feature vector. Specifically, the visual input sequence is obtained by preprocessing the collected visual information of the user based on the intelligent interaction control unit, and the voice input sequence is obtained by preprocessing the collected voice information of the user; the visual input sequence is input into a visual branch Transformer encoder in the double-branch Transformer model, the visual branch Transformer encoder includes six encoding layers, each encoding layer includes a multi-head self-attention sublayer and a feedforward neural network sublayer, and a visual feature representation is obtained by calculation; the voice input sequence is input into a voice branch Transformer encoder in the double-branch Transformer model, the voice branch Transformer encoder includes six encoding layers, the structure of each encoding layer is the same as that of the visual branch, and a voice feature representation is obtained by calculation; the visual feature representation and the voice feature representation are subjected to position encoding, position information is injected into the feature representations, an enhanced visual feature representation and an enhanced voice feature representation are obtained; the enhanced visual feature representation and the enhanced voice feature representation are input into a cross-attention layer, the attention weight between the modes is calculated, the interaction of the visual and voice information is realized, and a preliminary fusion feature is obtained; the preliminary fusion feature is input into a fusion Transformer encoder in the double-branch Transformer model, the fusion Transformer encoder includes four encoding layers, each encoding layer includes a multi-head self-attention sublayer, a cross-attention sublayer and a feedforward neural network sublayer, and a deep fusion feature is obtained by calculation; a global average pooling operation is performed on the deep fusion feature, the sequence feature is compressed into a fixed-dimension vector representation, and a fusion feature vector is obtained. The fusion feature vector and the enhanced video stream are jointly analyzed to obtain a user interaction intention and an interaction response strategy; specifically, spatiotemporal feature extraction is performed on the enhanced video stream to obtain a video spatiotemporal feature map, and the fusion feature vector is dimensionally expanded by copying and splicing operations to expand the fusion feature vector to the same spatiotemporal dimensions as the video spatiotemporal feature map, thereby obtaining an expanded fusion feature map; feature fusion is performed on the video spatiotemporal feature map and the expanded fusion feature map, and a joint feature map is obtained by element-by-element addition operation; based on the joint feature map, spatial attention weights and channel attention weights are calculated to obtain an attention-enhanced joint feature map; the attention-enhanced joint feature map is subjected to time series modeling to obtain time series encoding features, and the time series encoding features are input into a multi-task learning head including an intention classification branch and a response generation branch to obtain a user interaction intention probability distribution through a fully connected layer and a softmax function; based on the user interaction intention probability distribution, a user interaction intention is determined through threshold judgment, and a corresponding interaction response strategy template is selected according to a predefined intention-response mapping table; the time series encoding features and the interaction response strategy template are input into a strategy refinement network to generate specific interaction response strategy parameters, thereby obtaining an interaction response strategy; Based on the user interaction intention and the interaction response strategy, local contrast enhancement and motion compensation are performed on the enhanced video stream to obtain a high-quality video frame sequence; specifically, corresponding local contrast enhancement parameters and motion compensation parameters are extracted from a pre-defined intention-processing parameter mapping table according to the user interaction intention; frame-level segmentation is performed on the enhanced video stream to obtain a video frame sequence, and adaptive blocking is performed on each frame of image based on the local contrast enhancement parameters to obtain an image block set; local statistical features are calculated for each image block in the image block set, and an adaptive S-shaped curve mapping function is constructed according to the local statistical features and the local contrast enhancement parameters; the adaptive S-shaped curve mapping function is applied to the corresponding image block to obtain a locally contrast-enhanced image block, and the overlapping areas of adjacent image blocks are smoothly transitioned to obtain a contrast-enhanced video frame sequence; based on the motion compensation parameters, block matching motion estimation is performed on adjacent frames in the contrast-enhanced video frame sequence to obtain a motion vector field, and motion compensation interpolation is performed on the contrast-enhanced video frame sequence according to the motion vector field to generate a frame rate improved video sequence; a temporal filter is applied to the frame rate improved video sequence to obtain a smooth high frame rate video sequence, and global brightness and saturation adjustment is performed on the smooth high frame rate video sequence according to the video enhancement level in the interaction response strategy to obtain a high-quality video frame sequence; According to the real-time acquired ambient light information and the pre-stored display device color characteristics, scene adaptive color space mapping and dynamic color temperature adjustment are performed on the high-quality video frame sequence to obtain an output video stream; specifically comprising: color space conversion is performed on the high-quality video frame sequence to convert the RGB color space into the HSV color space to obtain hue H, saturation S and lightness V components; ambient light information including ambient light intensity and color temperature is collected in real time based on an ambient light sensor, and the current environment is classified as a bright, moderate or dim scene according to a pre-set ambient light threshold to obtain an ambient scene type; according to the ambient scene type and the pre-stored display device color characteristics, corresponding color enhancement coefficients α, β and γ are selected from a pre-set scene-color mapping table; non-linear adjustment is performed on the lightness V component, and Gamma correction is performed using the formula V'=V γ non-linear adjustment is performed on the lightness V component, and Gamma correction is performed using the formula V'=V γ linear adjustment is performed on the saturation S component based on the color enhancement coefficients α and β using the formula S'=α*S+β to obtain an adjusted saturation component S'; according to the ambient color temperature information, a hue offset ΔH is calculated, and adjustment is performed on the hue H component using the formula H'=H+ΔH to obtain an adjusted hue component H'; the adjusted hue component H', saturation component S' and lightness component V' are combined, and color space inverse conversion from HSV to RGB is performed to obtain color-enhanced video frames; the Floyd-Steinberg dithering algorithm is applied to the color-enhanced video frames to adjust the color depth to a bit depth supported by the display device to obtain an output video frame sequence, and the output video frame sequence is encoded into a video stream in H.264 format, the bit rate and key frame interval of network transmission are set, and an output video stream is obtained.

2. A television set-top box intelligent interactive control unit image processing apparatus, characterized in that, The device comprises: a feature analysis module for performing feature analysis on the input video stream of the TV box to obtain a video feature data structure; a processing module configured to perform overlapped image block division, independent processing and weighted reorganization on the input video stream according to the video feature data structure, to obtain an enhanced video stream; a fusion module configured to collect visual and voice interaction information of a user based on an intelligent interaction control unit, input the visual and voice interaction information into a double-branch Transformer model for multi-modal information fusion, and obtain a fusion feature vector; a joint analysis module configured to perform joint analysis on the fusion feature vector and the enhanced video stream, to obtain a user interaction intention and an interaction response strategy; a compensation module configured to perform local contrast enhancement and motion compensation on the enhanced video stream based on the user interaction intention and the interaction response strategy, to obtain a high-quality video frame sequence; an adjustment module configured to perform scene adaptive color space mapping and dynamic color temperature adjustment on the high-quality video frame sequence according to real-time acquired ambient light information and pre-stored display device color characteristics, to obtain an output video stream.

3. A computer device comprising a memory and a processor, the memory having stored therein a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of claim 1.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of claim 1.

Images