Method, device and equipment for adjusting background light of terminal with screen and storage medium

Abstract

The invention discloses a method, device and equipment for adjusting background light of a terminal with a screen and a storage medium, and relates to the technical field of intelligent hardware. The method comprises the following steps: acquiring figure video data, text data and audio data in a video; inputting the character video data, the text data and the audio data into a sentiment analysis model, extracting modal features and inter-modal features of the character video data, the text data and the audio data through the sentiment analysis model, and calculating decoupling feature input according to a relation among the modal features, the inter-modal features and the decoupling features; outputting a target sentiment score predicted value according to a mapping relationship between the decoupling features and the sentiment score predicted value; and adjusting the background light according to the target emotion score predicted value. According to the embodiment of the invention, the real-time adaptive adjustment of the background light is realized, and the background light interaction adjustment capability is improved.

Description

Technical Field

[0001] This application belongs to the field of smart hardware technology, and in particular relates to a method, device, equipment and storage medium for controlling the backlight of a terminal with a screen. Background Technology

[0002] With the development of technology, users are spending more and more time watching screen terminals. However, prolonged screen use has potential impacts on users' eyesight and health. In order to improve the viewing experience of screen terminals, it is often necessary to adjust the backlight of the terminal.

[0003] Existing technologies for controlling the backlight of terminals with screens rely on setting up the terminal's light sensing module to actively adjust the screen's brightness and color temperature range based on the ambient light intensity. However, they do not consider the impact of screen content on the user's display. If the user is dissatisfied, manual intervention is required. The technology cannot adaptively adjust the screen's backlight in real time based on the content the user is viewing, resulting in poor interactive backlight adjustment capabilities and a poor viewing experience for the user. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for adjusting the backlight of a terminal with a screen, in order to solve the problem that the prior art cannot adaptively adjust the screen backlight in real time based on the content viewed by the user, resulting in a poor viewing experience for the user.

[0005] In a first aspect, embodiments of this application provide a method for adjusting the backlight of a terminal with a screen, the method comprising:

[0006] Acquire video data, text data, and audio data of people in a video;

[0007] The sentiment analysis model inputs video data, text data, and audio data of the person into the sentiment analysis model. The model extracts modal features and intermodal features from the video data, text data, and audio data. Based on the relationship between modal features, intermodal features, and decoupling features, the decoupling features are calculated. The decoupling features are then input into the feedforward neural network of the sentiment analysis model. Based on the mapping relationship between the pre-trained decoupling features in the feedforward neural network and the predicted sentiment score, the target predicted sentiment score is output.

[0008] The background light is adjusted based on the predicted target sentiment score.

[0009] Secondly, embodiments of this application provide a device for adjusting the backlight of a screen-equipped terminal, the device comprising:

[0010] The acquisition module is used to acquire video data, text data, and audio data of people in the video.

[0011] The analysis module is used to input video data, text data, and audio data of the person into the sentiment analysis model. The sentiment analysis model extracts modal features and inter-modal features of the video data, text data, and audio data of the person. Based on the relationship between modal features, inter-modal features, and decoupling features, the module calculates decoupling features and inputs the decoupling features into the feedforward neural network of the sentiment analysis model. Based on the mapping relationship between the pre-trained decoupling features in the feedforward neural network and the predicted sentiment score, the module outputs the predicted target sentiment score.

[0012] The adjustment module is used to adjust the background light based on the predicted target sentiment score.

[0013] Thirdly, embodiments of this application provide a terminal device, which includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements a method for adjusting the backlight of a screen terminal as described in the first aspect.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement a method for adjusting the backlight of a screen-equipped terminal as described in the first aspect.

[0015] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform a method for adjusting the backlight of a screen terminal as described in the first aspect.

[0016] This application provides a method, apparatus, device, and storage medium for adjusting the backlight of a terminal with a screen. The method first acquires video data, text data, and audio data of a person in a video, obtaining the three main modalities of the video. By providing complementary information through different modalities, it reduces potential misjudgments from a single data source. Next, the video data, text data, and audio data are input into a sentiment analysis model. The sentiment analysis model extracts modal features and inter-modal features from the video data, text data, and audio data. Based on the relationship between modal features, inter-modal features, and decoupling features, decoupling features are calculated and input into a feedforward neural network. A target sentiment score prediction value is output based on the mapping relationship between the pre-trained decoupling features and the predicted sentiment score value in the feedforward neural network. By calculating the predicted sentiment score value through modal features, inter-modal features, and decoupling features, modal feature complementarity is achieved, improving the accuracy of sentiment recognition. Then, the backlight is controlled based on the target sentiment score prediction value. This achieves intelligent control of the backlight based on the emotions of the person in the video, enabling real-time adaptive adjustment of the backlight, improving the interactive adjustment capability of the backlight, and enhancing the user experience. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the method for adjusting the backlight of a screen-equipped terminal according to an embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating an implementation method for calculating decoupling features based on the relationship between modal features, inter-modal features, and decoupling features, provided in an embodiment of this application.

[0020] Figure 3 This is a flowchart illustrating an implementation method of obtaining a predicted sentiment score by inputting decoupled features into a feedforward neural network, as provided in an embodiment of this application.

[0021] Figure 4 This is a schematic diagram of the structure of the device for adjusting the backlight of a screen-equipped terminal provided in an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0023] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0024] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0025] Existing methods for adjusting backlight on screen-equipped terminals mostly involve manual adjustment via a screen brightness dropdown menu or active adjustment of the screen brightness and color temperature range based on ambient light using the terminal's light sensor. These methods do not consider the impact of screen content on user display and cannot adaptively adjust the backlight in real time based on the content viewed by the user. Users often need to actively intervene to adjust the backlight, resulting in poor interactive backlight adjustment capabilities and a poor user viewing experience.

[0026] To address the shortcomings of existing technologies, this application provides a method for adjusting the background light of a terminal with a screen. The method first acquires video data, text data, and audio data of a person in a video, identifying the three main modalities of the video. By providing complementary information through different modalities, it reduces potential misjudgments from a single data source. Next, the video data, text data, and audio data are input into a sentiment analysis model. The sentiment analysis model extracts modal features and inter-modal features from the video data, text data, and audio data. Based on the relationship between modal features, inter-modal features, and decoupling features, decoupling features are calculated and input into a feedforward neural network. The target sentiment score prediction value is output based on the mapping relationship between the pre-trained decoupling features and the predicted sentiment score value in the feedforward neural network. By calculating the predicted sentiment score value through modal features, inter-modal features, and decoupling features, modal feature complementarity is achieved, improving the accuracy of sentiment recognition. Then, background light control is performed based on the target sentiment score prediction value. This method achieves intelligent control of the background light by adjusting the background light according to the emotions of the person in the video, enabling real-time adaptive adjustment of the background light, improving the interactive adjustment capability of the background light, and enhancing the user experience.

[0027] The method for adjusting the backlight of a screen-equipped terminal provided in the embodiments of this application will be described below.

[0028] Figure 1 This illustration shows a flowchart of a method for adjusting the backlight of a screen-equipped terminal according to an embodiment of this application. Figure 1 As shown, the method may include the following steps: S101 to S104.

[0029] S101, acquire video data, text data and audio data of the people in the video.

[0030] In some embodiments, a preset facial recognition model can be used to crop faces from each video frame in the acquired video, and the cropped face images can be constructed into a sequence of face frame images in chronological order to obtain the person video data.

[0031] In some embodiments, after acquiring audio data from a video, speech-to-text technology can be used to convert the audio into a text sequence as text data.

[0032] By simultaneously acquiring video data, text data, and audio data of individuals in a video, multi-dimensional information input is provided for the sentiment analysis model, which helps to more comprehensively capture and analyze the emotional state of individuals.

[0033] S102, input the person's video data, text data, and audio data into the sentiment analysis model, extract the modal features and inter-modal features of the person's video data, text data, and audio data through the sentiment analysis model, calculate the decoupling features based on the relationship between the modal features, inter-modal features, and decoupling features, input the decoupling features into the feedforward neural network of the sentiment analysis model, and output the target sentiment score prediction value based on the mapping relationship between the pre-trained decoupling features in the feedforward neural network and the predicted sentiment score value.

[0034] In some embodiments, when extracting modal features and intermodal features from video data, text data, and audio data of people, the extraction can be performed using a key-sparse self-attention KS-transformer encoder.

[0035] By extracting and fusing intermodal features from different modalities, it is helpful to obtain the connections between features of different modalities and improve the accuracy of the predicted target sentiment score.

[0036] S103, adjust the background light based on the predicted target sentiment score.

[0037] In some embodiments, adjusting the background light according to the target emotion score prediction value may include: determining the target background light parameters corresponding to the target emotion score prediction value based on the relationship between the emotion score prediction value and a preset range of background light parameters;

[0038] Adjust the background light according to the target background light parameters.

[0039] In some embodiments, the target background light parameter within the range of background light parameters is determined by interpolation, and the background light parameter corresponding to the value position within the range of background light parameters is estimated based on the value position of the predicted sentiment score within the range of sentiment score. Algorithms such as linear interpolation, polynomial interpolation, or piecewise linear interpolation can be used to determine this parameter.

[0040] This application embodiment acquires three main modalities from the video, providing complementary information through different modalities to reduce potential misjudgments from a single data source. Next, the video data, text data, and audio data of the person are input into a sentiment analysis model. The sentiment analysis model extracts modal features and inter-modal features from the video, text, and audio data. Based on the relationship between modal features, inter-modal features, and decoupling features, decoupling features are calculated, and these decoupling features are input into a feedforward neural network to obtain a predicted sentiment score. By calculating the predicted sentiment score using modal features, inter-modal features, and decoupling features, modal feature complementarity is achieved, improving the accuracy of sentiment recognition. Then, background light control is performed based on the predicted target sentiment score. This achieves intelligent background light control by adjusting the background light according to the emotions of the person in the video, enabling real-time adaptive adjustment of the background light, improving the interactive adjustment capability of the background light, and enhancing the user experience.

[0041] In some embodiments, extracting modal and intermodal features from video, text, and audio data using a sentiment analysis model may include:

[0042] Extract features from video, text, and audio data of individuals and add classification labels;

[0043] The features and classification labels of the video data, text data and audio data of the person are input into the multi-head self-attention layer of the sentiment analysis model, respectively. The attention weight of each element of each feature with each other element of the feature is calculated and weighted summed to obtain the modal features of the video data, text data and audio data of the person.

[0044] The features and classification labels of the video, text, and audio data of the person are input into the multi-head self-attention layer of the sentiment analysis model. The attention weights of each element of each feature with each other element of the same feature are calculated and summed in a weighted manner to obtain the intermodal features of the video, text, and audio data of the person.

[0045] Existing methods for analyzing human emotions often extract features from only a single modality, potentially overlooking important information from other modalities. This can lead to an incomplete understanding of the overall emotion and increase the likelihood of recognition errors. Alternatively, while extracting information from multiple modalities of emotional expression, each modality requires a separate single-modal emotion analysis algorithm, resulting in independence between the modalities and hindering the fusion and complementarity of multimodal features.

[0046] This application provides multi-dimensional information input by extracting video data, text data, and audio data of people as input modalities; it extracts modal features and inter-modal features with modal relationships for each modality, and by combining modal features and inter-modal features, the model can obtain a more optimized feature representation and improve the accuracy of the analysis model.

[0047] In some embodiments, when extracting modal features from person video data, text data, and audio data, the person video data, text data, and audio data are respectively input into the KS-transformer encoder, and feature extraction is performed on the data of the three modalities respectively. The query vector, key vector, and value vector input to the KS-transformer encoder are data from the same modality.

[0048] In this process, after the feature data is input into the KS-transformer encoder, the input data is used as query vectors, key vectors, and value vectors and then fed into a multi-head attention layer. The attention weights of each element of each feature with every other element of that feature are calculated and summed in a weighted manner. The output of the multi-head attention layer is added to the input feature data and normalized. The normalized result is then fed into a feedforward neural network, which performs weighted summation and nonlinear transformation operations. The output of the feedforward neural network is added to the normalized result again and normalized once more. This second normalized result is used as the modal feature output by the KS-transformer encoder.

[0049] In some embodiments, when extracting inter-modal features from person video data, text data, and audio data, the person video data, text data, and audio data are simultaneously input into the KS-transformer encoder. The person video data, text data, and audio data are used in turn as query vectors for the KS-transformer encoder, and the remaining data are used as key vectors and value vectors. The output inter-modal features are the inter-modal features of the data used as query vectors.

[0050] In some embodiments, the video data of the person is input into a pre-trained Frequency-Aware Binarized Network (FABNet) for feature extraction to obtain the features of the video data of the person.

[0051] In some embodiments, inputting person video data into a pre-trained frequency-aware binarization network for feature extraction to obtain features of the person video data may include:

[0052] Wavelet transform is used to decompose the video data of people into multiple scales to obtain low-frequency and high-frequency features;

[0053] A pre-defined binarization network structure is used to process low-frequency features and high-frequency features respectively, resulting in low-frequency features containing texture information and high-frequency features containing detail information;

[0054] By fusing low-frequency features containing texture information and high-frequency features containing detail information, the features of the person video data are obtained.

[0055] In some embodiments, text data is input into a pre-trained Robustly Optimized BERT Approach (RoBERTa) network for feature extraction to obtain the features of the text data.

[0056] In some embodiments, inputting text data into a pre-trained, robust, optimized bidirectional encoder representation network for feature extraction to obtain features of the text data may include:

[0057] The text data is segmented using the RoBERTa network's pre-defined word segmenter;

[0058] The segmented text data is input into the RoBERTa network, and the text data features are obtained through forward propagation.

[0059] In some embodiments, audio data is input into a pre-trained Waveform-to-Vector (wav2vec) network to extract features from the audio data.

[0060] In some embodiments, such as Figure 2 As shown, calculating decoupling features based on the relationship between modal features, inter-modal features, and decoupling features may include the following steps:

[0061] S201, calculate the average value of the modal features of the person's video data, text data, and audio data as the common modal features;

[0062] S202, subtract the common modal features from the modal features of the person video data, text data, and audio data respectively to obtain the noise features of the person video data, text data, and audio data;

[0063] S203, the noise features of the person video data, text data and audio data are multiplied by preset coefficients and then added to the modal common features to obtain the decoupled features of the person video data, text data and audio data.

[0064] By calculating the average value of modal features as common modal features, the commonalities between different modal data can be captured. Subtracting the common modal features from the original features of each modality yields modality-specific noise features, which contain information or details not shared with other modalities. Finally, multiplying these noise features by preset coefficients and adding them to the common modal features results in decoupled features that retain both the common information between modalities and the unique information of each modality, thus helping to improve the accuracy of sentiment analysis models.

[0065] In some embodiments, the preset coefficient is a pre-defined coefficient that satisfies a Gaussian distribution and has sparse characteristics, that is, the coefficient is close to zero in many dimensions and has significant non-zero values ​​in only a few dimensions, which is used to multiply with the noise features to enhance the decoupled features.

[0066] By introducing a certain degree of randomness, variable irrelevant noise is generated, which affects the training of the model. The common features of the relevant modalities will counteract the variable irrelevant noise. By training against this variable irrelevant noise, the learned features can be made to have strong robustness.

[0067] In some embodiments, such as Figure 3 As shown, inputting decoupled features into a feedforward neural network to obtain predicted sentiment scores can include the following steps:

[0068] S301, Multiply the feature elements of the decoupled features of the video data, text data and audio data of the person by element to obtain the fused features;

[0069] S302, the fused features are input into the feedforward neural network, and the intermediate features are obtained by multiplying the fused features with the pre-trained weight matrix;

[0070] S303 inputs intermediate features into a pre-trained activation function and outputs the target sentiment score prediction value based on the mapping relationship between intermediate features and sentiment score prediction values.

[0071] By performing element-wise multiplication of decoupled features from video, text, and audio data, multimodal feature fusion is achieved. This approach not only considers the complementarity between different modalities but also enhances the interaction between features through element-wise multiplication, helping the model to capture emotional information more comprehensively. Pre-trained weight matrices transform the fused features into deeper feature representations, facilitating the extraction of emotion-related information. The activation function introduces nonlinearity into the neural network, enabling it to learn and simulate complex nonlinear relationships, thereby outputting a predicted target emotion score.

[0072] In some embodiments, the method may further include, before inputting the person video data, text data, and audio data into the sentiment analysis model:

[0073] Obtain the training sample set, which includes training data of people's videos, text, and audio, as well as preset sentiment scores.

[0074] The training data of the person's video, text, and audio are input into the sentiment analysis model to obtain the modal training features, intermodal training features, noise training features, and predicted sentiment scores of the person's video, text, and audio data. The noise training features are determined by the preset relationship between the modal features, intermodal features, and noise features.

[0075] The target loss function value is calculated based on the relationship between intermodal training features, noise training features, predicted sentiment scores, preset sentiment scores, and target loss function value.

[0076] The gradient of the model parameters is calculated based on the target loss function value. The parameters of the sentiment analysis model are updated according to the gradient, and the sentiment analysis model is iteratively trained until the target loss function value does not exceed the set threshold.

[0077] By establishing a target loss function and training the sentiment analysis model with the minimum value of the target loss function as the optimization objective, the prediction error of the model can be gradually reduced, and the accuracy of the model in the sentiment analysis process can be improved.

[0078] In some embodiments, when training a sentiment analysis model with the goal of minimizing the target loss function, the Adaptive Moment Estimation (Adam) optimizer is used to train the sentiment analysis model.

[0079] In some embodiments, training a sentiment analysis model using the Adam optimizer may include the following steps:

[0080] Set the loss function and hyperparameters of the Adam optimizer;

[0081] Backpropagation is performed based on the target loss function value to calculate the gradient of each model parameter;

[0082] The Adam optimizer is used to compute the first-order and second-order moment estimates of the gradient of each model parameter, and bias corrections are applied.

[0083] The model parameters are updated using the first-order moment estimate, the second-order moment estimate, and the bias correction value;

[0084] Repeat the above steps until the target loss function is minimized.

[0085] In some embodiments, calculating the target loss function value based on the relationship between inter-modal training features, noisy training features, predicted sentiment scores, preset sentiment scores, and the target loss function value may include the following steps:

[0086] The modal features of the three modalities are subtracted pairwise, the feature vector lengths of each difference are calculated and summed to obtain the value of the modal feature optimization loss function.

[0087] Calculate the vector length of the noisy training features for each modality and sum them to obtain the modality feature optimization loss function value;

[0088] The value of the emotion prediction optimization function is obtained by subtracting the preset emotion score from the predicted emotion score and taking the absolute value.

[0089] Calculate the average values ​​of the inter-modal feature optimization loss function, the modal feature optimization loss function, and the sentiment prediction optimization function for all samples. Multiply the average values ​​by the corresponding preset weight parameters and sum them to obtain the target loss function value.

[0090] In some embodiments, the inter-modal feature optimization loss function can be:

[0091] L inter =||F T-inter -F A-inter ||2+||F T-inter -F V-inter ||2+||F A-inter -F V-inter ||2

[0092] Among them, L inter To optimize the loss function for inter-modal features, ||F T-inter -F A-inter ||2 is the feature vector length of the difference between the inter-modal features of the text data and the inter-modal features of the audio data, ||F T-inter -F V-inter ||2 is the feature vector length of the difference between the inter-modal features of the text data and the inter-modal features of the person video data, ||F A-inter -F V-inter ||2 represents the feature vector length of the difference between the intermodal features of the audio data and the intermodal features of the video data.

[0093] In some embodiments, the modality feature optimization loss function can be:

[0094] L intra =||F Noise-A ||2+||F Noise-V ||2+||F Noise-T ||2

[0095] Among them, L intra To optimize the loss function for modal features, ||F Noise-A ||2 is the vector length of the noise training features of the audio data, ||F Noise-V ||2 is the vector length of the noisy training features of the person video data, ||F Noise-T ||2 represents the vector length of the noisy training features of the text data.

[0096] In some embodiments, the sentiment prediction optimization function can be:

[0097]

[0098] Among them, L m Optimize the function for sentiment prediction. y is the predicted sentiment score, and y is the preset sentiment score value.

[0099] In some embodiments, the target loss function can be:

[0100]

[0101] Among them, L all Let N be the target loss function, and N be the total number of training samples. Let L be the value of the sentiment prediction optimization function for the i-th sample, α be the weight parameter of the inter-modal feature optimization loss function, and L be the value of the th sample. inter i Let L be the inter-modal feature optimization loss function value for the i-th sample, β be the weight parameter for the inter-modal feature optimization loss function value, and L be the weight parameter for the inter-modal feature optimization loss function value. intra i Optimize the loss function value for the modal features of the i-th sample.

[0102] In some embodiments, adjusting the background light based on the target sentiment score prediction may include the following steps:

[0103] Determine if the predicted sentiment score falls within the target sentiment score range of the preset sentiment score range; obtain the corresponding range of background light parameters based on the target sentiment score range.

[0104] The target background light parameters within the range of background light parameters are determined by interpolation to predict the sentiment score.

[0105] Adjust the background light according to the target background light parameters.

[0106] In existing sentiment analysis methods, sentiment scores only represent the degree of negativity or positivity of a person's emotions. For example, negative numbers represent negativity, and the smaller the negative number, the higher the degree of negativity. Positive numbers represent positivity, and the larger the integer, the higher the degree of positivity. Furthermore, only multi-level dimming is supported. When the sentiment score is within a preset score range, the background light is adjusted to the adjustment scheme corresponding to that score range, and there are relatively few background light adjustment schemes.

[0107] This application embodiment determines the specific target background light parameters within the background light parameter range by using an interpolation method to predict the sentiment score. This allows for a smooth transition within the background light parameter range, rather than jumping between several predefined schemes. It provides continuous background light adjustment for continuous changes in sentiment score, achieving fine-grained adjustment and stepless dimming of the background light, thus improving the precision of the background light adjustment.

[0108] In some embodiments, the emotion score output by this application is a continuous emotion score, and each preset emotion score range corresponds to a different emotion category and background light parameter adjustment range.

[0109] In one embodiment, background light parameters may include color temperature and brightness.

[0110] In one example, the correspondence between score range, emotion category, color temperature range, and brightness range is shown in Table 1. The preset emotion score range of -4 to -3.5 indicates disgust, the color temperature range is 2500K-3000K, and the brightness range is 20%-40%. The preset emotion score range of -3.5 to -2.5 indicates fear, the color temperature range is 3500K-4000K, and the brightness range is 30%-50%.

[0111] Table 1: Correspondence between score range, emotion category, color temperature range, and brightness range

[0112]

[0113] In one example, a predicted sentiment score of 3.75 falls between 3.5 and 4, corresponding to the emotion of excitement. The color temperature range for excitement is 6500K-7000K, and the brightness range is 80%-100%. Then, using linear interpolation, the specific color temperature and brightness values ​​are determined within these ranges, resulting in a color temperature of 6750K and a brightness of 90% for a sentiment score of 3.75.

[0114] Figure 4 This illustration shows a structural schematic of a device 400 for adjusting the backlight of a screen-equipped terminal according to an embodiment of this application. Figure 4 As shown, the device may include:

[0115] The acquisition module 401 is used to acquire video data, text data, and audio data of people in the video.

[0116] Analysis module 402 is used to input video data, text data and audio data of the person into the sentiment analysis model, extract modal features and inter-modal features of the video data, text data and audio data of the person through the sentiment analysis model, calculate decoupling features according to the relationship between modal features, inter-modal features and decoupling features, input the decoupling features into the feedforward neural network of the sentiment analysis model, and output the target sentiment score prediction value according to the mapping relationship between the pre-trained decoupling features in the feedforward neural network and the sentiment score prediction value.

[0117] Adjustment module 403 is used to adjust the background light based on the predicted value of the target sentiment score.

[0118] In some embodiments, the acquisition module is further configured to extract features from the video data, text data, and audio data of the person, and add classification labels;

[0119] Correspondingly, the device 400 for adjusting the backlight of the screen terminal may also include a calculation module, which is used to input the features and classification labels of the person video data, text data and audio data into the multi-head self-attention layer of the sentiment analysis model, calculate the attention weight of each element of each feature with each other element of the feature and sum them up in a weighted manner to obtain the modal features of the person video data, text data and audio data.

[0120] Correspondingly, the calculation module is also used to calculate the attention weight of each element of each feature with each other element of the feature in the multi-head self-attention layer of the features, text data and audio data, and classification labels, and then sum them up in a weighted manner to obtain the inter-modal features of the video data, text data and audio data.

[0121] In some embodiments, the computing module may also be used for:

[0122] The average value of the modal features of the person's video data, text data, and audio data is calculated as the common modal feature;

[0123] The common modal features are subtracted from the modal features of the video data, text data, and audio data of the person, respectively, to obtain the noise features of the video data, text data, and audio data of the person;

[0124] The noise features of the video data, text data, and audio data of the person are multiplied by preset coefficients and then added to the common modal features to obtain the decoupled features of the video data, text data, and audio data of the person.

[0125] In some embodiments, the computing module may also be used for:

[0126] The feature elements of the decoupled features of the video data, text data and audio data are multiplied element by element to obtain the fused features;

[0127] The fused features are input into the feedforward neural network, and the intermediate features are obtained by multiplying the fused features with the pre-trained weight matrix.

[0128] Correspondingly, the analysis module is also used to input intermediate features into a pre-trained activation function and output the target sentiment score prediction value based on the mapping relationship between the intermediate features and the sentiment score prediction value.

[0129] In some embodiments, the acquisition module 401 is further configured to acquire a training sample set, which includes training data of person videos, training data of text, training data of audio, and preset emotion scores.

[0130] The analysis module 402 is also used to input the character video training data, text training data and audio training data into the sentiment analysis model to obtain the modal training features, intermodal training features, noise training features and sentiment score prediction values ​​of the character video data, text data and audio data. The noise training features are determined by the preset relationship between the modal features, intermodal features and noise features.

[0131] The calculation module is also used to calculate the target loss function value based on the relationship between intermodal training features, noise training features, predicted sentiment scores, preset sentiment scores, and target loss function values.

[0132] Correspondingly, the device 400 for adjusting the backlight of the screen terminal may also include a training module for calculating the gradient of the model parameters based on the target loss function value, updating the parameters of the sentiment analysis model according to the gradient, and iteratively training the sentiment analysis model until the target loss function value does not exceed a set threshold.

[0133] In some embodiments, the computing module may also be used for:

[0134] The modal features of the three modalities are subtracted pairwise, the feature vector lengths of each difference are calculated and summed to obtain the value of the modal feature optimization loss function.

[0135] Calculate the vector length of the noisy training features for each modality and sum them to obtain the modality feature optimization loss function value;

[0136] The value of the emotion prediction optimization function is obtained by subtracting the preset emotion score from the predicted emotion score and taking the absolute value.

[0137] Calculate the average values ​​of the inter-modal feature optimization loss function, the modal feature optimization loss function, and the sentiment prediction optimization function for all samples. Multiply the average values ​​by the corresponding preset weight parameters and sum them to obtain the target loss function value.

[0138] In some embodiments, the device 400 for adjusting the backlight of a screen terminal may further include a determining module for determining that the predicted emotion score value belongs to a target emotion score range within a preset emotion score range; and obtaining the corresponding backlight parameter range based on the target emotion score range.

[0139] The determination module is also used to determine the target background light parameters within the range of background light parameters by interpolation for the predicted sentiment score;

[0140] The adjustment module 403 is also used to adjust the background light according to the target background light parameters.

[0141] Figure 4 The various modules in the illustrated device can achieve Figure 1 The various steps involved, and the corresponding technical effects achieved, will not be elaborated upon here for the sake of brevity.

[0142] Figure 5 A schematic diagram of the hardware structure of the terminal device provided in an embodiment of this application is shown.

[0143] The terminal device may include a processor 501 and a memory 502 storing computer program instructions.

[0144] Specifically, the processor 501 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0145] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 502 may include removable or non-removable (or fixed) media, or memory 502 may be non-volatile solid-state storage. Memory 502 may be internal or external to the integrated gateway disaster recovery device.

[0146] In one example, memory 502 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method for adjusting backlight of a screened terminal according to this disclosure.

[0147] The processor 501 reads and executes computer program instructions stored in the memory 502 to achieve... Figure 1 The method for adjusting the backlight of a terminal with a screen in the illustrated embodiment.

[0148] In one example, the terminal device may also include a communication interface 503 and a bus 504. Wherein, for example... Figure 5 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 504 and complete communication with each other.

[0149] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0150] Bus 504 includes hardware, software, or both, that couples components of an end device together. For example, and not as a limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 504 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0151] Furthermore, in conjunction with the backlight adjustment method for a screen-equipped terminal in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the backlight adjustment methods for a screen-equipped terminal in the above embodiments.

[0152] This application also provides a computer program product, including a computer program, which, when executed, implements any of the methods for adjusting the backlight of a screen-equipped terminal described in the above embodiments.

[0153] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0154] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or text segments used to perform the required tasks. Programs or text segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Text segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0155] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0156] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0157] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for adjusting the backlight of a terminal with a screen, characterized in that, The method includes: Acquire video data, text data, and audio data of people in a video; The video data, text data, and audio data of the person are input into the sentiment analysis model. The modal features and inter-modal features of the video data, text data, and audio data are extracted through the sentiment analysis model. The decoupling features are calculated based on the relationship between the modal features, inter-modal features, and decoupling features. The decoupling features are input into the feedforward neural network of the sentiment analysis model. The target sentiment score prediction value is output based on the mapping relationship between the pre-trained decoupling features in the feedforward neural network and the predicted sentiment score value. The background light is adjusted based on the predicted target sentiment score.

2. The method for adjusting the backlight of a terminal with a screen according to claim 1, characterized in that, The extraction of modal and inter-modal features from the sentiment analysis model of video, text, and audio data includes: Extract features from video, text, and audio data of individuals and add classification labels; The features and classification labels of the video data, text data and audio data of the person are input into the multi-head self-attention layer of the sentiment analysis model, respectively. The attention weight of each element of each feature with each other element of the feature is calculated and weighted summed to obtain the modal features of the video data, text data and audio data of the person. The features and classification labels of the video, text, and audio data of the person are input into the multi-head self-attention layer of the sentiment analysis model. The attention weights of each element of each feature with each other element of the same feature are calculated and summed in a weighted manner to obtain the inter-modal features of the video, text, and audio data of the person.

3. The method for adjusting the backlight of a terminal with a screen according to claim 1, characterized in that, The calculation of decoupling features based on the relationship between modal features, inter-modal features, and decoupling features includes: The average value of the modal features of the person's video data, text data, and audio data is calculated as the common modal feature; The common modal features are subtracted from the modal features of the video data, text data, and audio data of the person, respectively, to obtain the noise features of the video data, text data, and audio data of the person; The noise features of the video data, text data, and audio data of the person are multiplied by preset coefficients and then added to the common modal features to obtain the decoupled features of the video data, text data, and audio data of the person.

4. The method for adjusting the backlight of a terminal with a screen according to claim 1, characterized in that, The step of inputting decoupling features into a feedforward neural network and outputting a target sentiment score prediction value based on the pre-trained mapping relationship between the decoupling features and the sentiment score prediction value in the feedforward neural network includes: The feature elements of the decoupled features of the video data, text data and audio data are multiplied element by element to obtain the fused features; The fused features are input into the feedforward neural network, and the intermediate features are obtained by multiplying the fused features with the pre-trained weight matrix. The intermediate features are input into a pre-trained activation function, and the target sentiment score prediction is output based on the mapping relationship between the intermediate features and the predicted sentiment score.

5. The method for adjusting the backlight of a terminal with a screen according to claim 1, characterized in that, Before inputting the aforementioned video data, text data, and audio data into the sentiment analysis model, the method further includes: Obtain a training sample set, which includes training data of people's videos, text training data, and audio training data, as well as preset emotion scores. The training data of the person's video, text, and audio are input into the sentiment analysis model to obtain the modal training features, intermodal training features, noise training features, and predicted sentiment scores of the person's video, text, and audio data. The noise training features are determined by a preset relationship between the modal features, intermodal features, and noise features. The target loss function value is calculated based on the relationship between the intermodal training features, noise training features, predicted sentiment scores, preset sentiment scores, and target loss function values. The gradient of the model parameters is calculated based on the target loss function value. The parameters of the sentiment analysis model are updated according to the gradient, and the sentiment analysis model is iteratively trained until the target loss function value does not exceed a set threshold.

6. The method for adjusting the backlight of a terminal with a screen according to claim 5, characterized in that, The step of calculating the target loss function value based on the relationship between the inter-modal training features, noise training features, predicted sentiment scores, preset sentiment scores, and the target loss function value includes: The modal features of the three modalities are subtracted pairwise, the feature vector lengths of each difference are calculated and summed to obtain the value of the modal feature optimization loss function. Calculate the vector length of the noisy training features for each modality and sum them to obtain the modality feature optimization loss function value; The value of the emotion prediction optimization function is obtained by subtracting the preset emotion score from the predicted emotion score and taking the absolute value. Calculate the average values ​​of the inter-modal feature optimization loss function, the modal feature optimization loss function, and the sentiment prediction optimization function for all samples. Multiply the average values ​​by the corresponding preset weight parameters and sum them to obtain the target loss function value.

7. The method for adjusting the backlight of a terminal with a screen according to claim 1, characterized in that, The adjustment of the background light based on the predicted target sentiment score includes: The predicted emotion score is determined to belong to a target emotion score range within a preset emotion score range; the corresponding background light parameter range is obtained based on the target emotion score range; The target background light parameters within the range of background light parameters are determined by interpolation to predict the sentiment score. The background light is adjusted according to the target background light parameters.

8. A device for adjusting the backlight of a terminal with a screen, characterized in that, The device includes: The acquisition module is used to acquire video data, text data, and audio data of people in the video. The analysis module is used to input the video data, text data, and audio data of the person into the sentiment analysis model, extract the modal features and inter-modal features of the video data, text data, and audio data of the person through the sentiment analysis model, calculate the decoupling features according to the relationship between the modal features, inter-modal features, and decoupling features, input the decoupling features into the feedforward neural network of the sentiment analysis model, and output the target sentiment score prediction value according to the mapping relationship between the pre-trained decoupling features in the feedforward neural network and the sentiment score prediction value. The adjustment module is used to adjust the background light based on the predicted target sentiment score.

9. A terminal device, characterized in that, The device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the method for adjusting the backlight of a screen terminal as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the method for adjusting the backlight of a screen-equipped terminal as described in any one of claims 1-7.

11. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the method for adjusting the backlight of a screen terminal as described in any one of claims 1-7.

Images