Method and system for optimizing video content for elderly users based on multi-modal fusion

Abstract

This application discloses a method and system for optimizing video content for elderly users based on multimodal fusion, relating to the field of data processing technology. The method includes: first, acquiring user feedback and multimodal data of the video to be evaluated to extract multimodal feature vectors; then, inputting these vectors into a multimodal fusion model containing a feature extraction layer, a modality fusion layer, and an output layer containing a main task regression module and an auxiliary task classification module; obtaining a fit score and fuzzy segment localization results; and generating specific optimization suggestions based on these results, which are then pushed to the creation terminal for video modification. This method achieves quantifiable evaluation of video understanding and fit for elderly users, identifies comprehension difficulties, generates targeted optimization suggestions, improves video suitability for elderly users, automates evaluation and optimization, reduces manual costs, and improves the efficiency and effectiveness of content optimization.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for optimizing video content for elderly users based on multimodal fusion. Background Technology

[0002] Currently, the evaluation and optimization of video content mainly revolves around the general user group. For video content optimization specifically for elderly users, existing practices mostly rely on shallow data indicators to make a preliminary judgment on content quality, while combining the subjective experience of domain experts to complete the understanding and optimization guidance. At the data processing level, user feedback data and video content feature data are processed separately. User feedback is analyzed only through simple keyword statistics, while the focus is on technical parameter detection and analysis of video content features.

[0003] Current practices fail to design specific assessment dimensions tailored to the unique cognitive characteristics of elderly users. Relying solely on superficial indicators such as viewing time and click-through rates cannot reflect the actual understanding and experience of elderly users. The reliance on subjective expert evaluations suffers from low scoring consistency, high manual assessment costs, and difficulty in large-scale application to massive amounts of content, thus failing to establish a standardized assessment system. The fragmented processing of user feedback and video content characteristics fails to establish an effective mapping relationship and cannot uncover the deeper connections between the two. Furthermore, the lack of targeted optimization suggestions makes it impossible to pinpoint the specific content segments and core dimensions that cause comprehension difficulties for elderly users, hindering content creators from making precise modifications and effectively improving the understanding and adaptability of video content for elderly users.

[0004] Therefore, how to achieve correlation modeling and automated evaluation of feedback from elderly users and multimodal features of video content, and generate actionable optimization suggestions that are located at specific segments and dimensions, has become an urgent problem to be solved. Summary of the Invention

[0005] The main purpose of this application is to provide a method and system for optimizing video content for elderly users based on multimodal fusion, aiming to solve the technical problem of how to automatically evaluate and optimize the understanding and adaptation of video content for elderly users.

[0006] To achieve the above objectives, this application proposes a method for optimizing video content for elderly users based on multimodal fusion, including:

[0007] Obtain user feedback data and multimodal data of video content for the video to be evaluated;

[0008] Multimodal feature vectors are extracted based on the user feedback data and the multimodal data of the video content;

[0009] The multimodal feature vectors are input into a preset multimodal fusion model to obtain video content understanding and adaptation scores and conceptually ambiguous segment localization results. The preset multimodal fusion model includes a feature extraction layer, a modality fusion layer, and an output layer. The output layer includes a main task regression module and an auxiliary task classification module.

[0010] Based on the video content understanding and adaptation score and the location results of the conceptually ambiguous segments, optimization suggestions are generated for specific segments and understanding dimensions.

[0011] The optimization suggestions are sent to the content creation terminal so that the content creation terminal can modify the video to be evaluated based on the optimization suggestions.

[0012] In one embodiment, the step of obtaining user feedback data and multimodal data of video content for the video to be evaluated includes:

[0013] The raw user feedback data of the video to be evaluated is collected from the user interaction logs of the video playback platform and the terminal. The data is divided into explicit feedback data and implicit feedback data according to the form of feedback expression. The explicit feedback is the timestamp feedback submitted by the user when clicking the one-click feedback button on the screen during the viewing process. The implicit feedback includes user exit behavior logs and text in the comment section.

[0014] The explicit feedback data and the implicit feedback data are respectively subjected to deduplication, outlier removal and timestamp precision calibration to obtain standardized user feedback data;

[0015] The video to be evaluated is subjected to audio-video decoupling and text extraction operations to obtain raw audio data, raw visual data and raw text data, which are combined into raw multimodal data of video content;

[0016] The original multimodal data of the video content is normalized in format and validated for data validity to obtain standardized multimodal data of video content;

[0017] The standardized user feedback data and the standardized video content multimodal data are precisely aligned along the playback timeline to obtain the user feedback data and video content multimodal data of the video to be evaluated.

[0018] In one embodiment, the step of extracting multimodal feature vectors based on the user feedback data and the video content multimodal data includes:

[0019] The user feedback data is split into explicit feedback data and implicit feedback data;

[0020] Based on the explicit and implicit feedback data, the feedback density, exit rate curve, and sentiment-semantic vector are extracted and feature quantization is completed. The results are then integrated to obtain the user feedback modality feature set.

[0021] The video content multimodal data is split into audio data, visual data, and text data;

[0022] Based on the audio data, visual data, and text data, audio features, visual features, and text features are extracted and quantized respectively, and integrated to obtain a video content modal feature set. The audio features include speech rate, pause frequency, and pitch fluctuation. The visual features include subtitle clarity and frame complexity. The text features include professional vocabulary density, example richness, and syntactic complexity.

[0023] Calculate the correlation coefficients between each feature in the user feedback modal feature set and the video content modal feature set and the video content understanding and adaptation, select features whose absolute values ​​of correlation coefficients meet the preset correlation threshold, and integrate them to obtain a multimodal key feature set;

[0024] Normalization processing is performed on each feature in the multimodal key feature set, and the processed features are mapped to a preset unified feature space to obtain standardized multimodal features;

[0025] The standardized multimodal features are subjected to feature concatenation and dimension normalization to obtain multimodal feature vectors.

[0026] In one embodiment, before the step of inputting the multimodal feature vector into a preset multimodal fusion model to obtain the video content understanding fit score and the result of fuzzy segment localization, the method includes:

[0027] A labeled sample set is constructed by retrieving multimodal feature data of a predetermined number of video segments from the labeled sample library, along with corresponding manually labeled fit labels and conceptually ambiguous segment labels. At the same time, a labeled sample set is constructed by retrieving multimodal feature data of a predetermined number of unlabeled video segments.

[0028] Define a dual-task total loss function, where the dual-task total loss function is a weighted sum of regression loss and classification loss;

[0029] Build an initial multimodal fusion model;

[0030] The initial multimodal fusion model is trained in a supervised manner based on the labeled sample set to obtain the trained model;

[0031] The trained model is used to generate pseudo-labels for the unlabeled sample set, and iterative training is performed using a hybrid loss function to obtain an optimized model, wherein the hybrid loss function is a weighted sum of the labeled sample loss and the unlabeled sample loss.

[0032] The optimization model is validated for performance, and target optimization models that meet the preset performance threshold are selected.

[0033] The target optimization model is used as a preset multimodal fusion model.

[0034] In one embodiment, the step of inputting the multimodal feature vector into a preset multimodal fusion model to obtain a video content understanding fit score and a conceptually ambiguous segment localization result includes:

[0035] The multimodal feature vectors are input into the feature extraction layer of the preset multimodal fusion model to obtain the projection features of each modality;

[0036] The projection features of each modality are input into the modal fusion layer of the preset multimodal fusion model to obtain global fusion features;

[0037] The global fusion features are input into the main task regression module of the output layer of the preset multimodal fusion model to obtain the video content understanding and adaptation score.

[0038] The global fusion features are input into the output layer auxiliary task classification module of the preset multimodal fusion model to obtain the concept fuzzy fragment localization result.

[0039] In one embodiment, the step of inputting the global fusion features into the main task regression module of the output layer of the preset multimodal fusion model to obtain the video content understanding and adaptation score includes:

[0040] The global fusion features are input into the first fully connected layer of the main task regression module for feature dimension compression to obtain the first regression features;

[0041] The first regression feature is input into the batch normalization layer of the main task regression module for feature distribution normalization to obtain the second regression feature;

[0042] The second regression feature is input into the activation function layer of the main task regression module for nonlinear mapping to obtain the third regression feature;

[0043] The third regression feature is input into the second fully connected layer of the main task regression module and mapped to a preset scoring range to obtain the video content understanding and adaptation score.

[0044] The step of inputting the global fusion features into the output layer auxiliary task classification module of the preset multimodal fusion model to obtain the concept fuzzy fragment localization result includes:

[0045] The global fusion features are input into the convolutional layer of the auxiliary task classification module to extract fragment-level local features, thus obtaining the first classification features;

[0046] The first classification feature is input into the attention mechanism layer of the auxiliary task classification module to calculate the attention weight of each segment feature, and the second classification feature is obtained.

[0047] The second classification feature is input into the batch normalization layer of the auxiliary task classification module for feature distribution normalization to obtain the third classification feature;

[0048] The third classification feature is input into the Sigmoid activation layer of the auxiliary task classification module and mapped to the 0-1 interval to obtain the blur probability of each video segment.

[0049] The fuzzy probability is compared with a preset fuzzy threshold, and segments with fuzzy probabilities higher than the preset fuzzy threshold are selected to obtain the concept fuzzy segment localization result.

[0050] In one embodiment, the step of generating optimization suggestions for specific segments and understanding dimensions based on the video content understanding fit score and the conceptually ambiguous segment localization result includes:

[0051] The video content understanding and adaptation score is compared with a preset adaptation threshold. When the video content understanding and adaptation score is lower than the preset adaptation threshold, the key optimized segments and corresponding segment location information in the conceptual ambiguity segment localization results are extracted.

[0052] The multimodal feature values ​​of the key optimization segment are analyzed and matched with a preset understanding dimension feature library to locate abnormal understanding dimensions in the key optimization segment whose feature values ​​exceed a preset feature threshold. The multimodal feature values ​​include professional vocabulary density, example richness, speech rate, pause frequency and subtitle clarity.

[0053] Retrieve a preset feature-adaptation association rule library and match it with the optimization rule corresponding to the anomaly understanding dimension, wherein the optimization rule includes feature adjustment criteria and specific optimization methods for each understanding dimension;

[0054] By combining the key optimization segments, the anomaly understanding dimensions, and the optimization rules, dimensional adjustment requirements for each key optimization segment are generated;

[0055] By integrating the location information of all key optimization segments and the aforementioned dimensional adjustment requirements, and organizing them into structured text according to the video playback sequence, optimization suggestions are obtained.

[0056] Furthermore, to achieve the above objectives, this application also proposes a multimodal fusion-based video content optimization system for elderly users, which includes:

[0057] The acquisition module is used to acquire user feedback data and multimodal data of video content for the video to be evaluated;

[0058] The feature extraction module is used to extract multimodal feature vectors based on the user feedback data and the multimodal data of the video content;

[0059] The evaluation module is used to input the multimodal feature vector into a preset multimodal fusion model to obtain a video content understanding and adaptation score and a conceptually ambiguous segment localization result. The preset multimodal fusion model includes a feature extraction layer, a modality fusion layer and an output layer. The output layer includes a main task regression module and an auxiliary task classification module.

[0060] The optimization module is used to generate optimization suggestions for specific segments and understanding dimensions based on the video content understanding and adaptation score and the location results of the conceptually ambiguous segments.

[0061] The sending module is used to send the optimization suggestions to the content creation terminal, so that the content creation terminal can modify the video to be evaluated according to the optimization suggestions.

[0062] Furthermore, the feature extraction module is also used to split the user feedback data into explicit feedback data and implicit feedback data; extract feedback density, exit rate curves, and sentiment-semantic vectors based on the explicit and implicit feedback data and complete feature quantization, integrating them to obtain a user feedback modal feature set; split the video content multimodal data into audio data, visual data, and text data; extract audio features, visual features, and text features based on the audio data, visual data, and text data respectively and complete feature quantization, integrating them to obtain a video content modal feature set, wherein the audio features include speech rate, pause frequency, and pitch fluctuation, and the visual features... The features include subtitle clarity and frame complexity. The text features include professional vocabulary density, example richness, and syntactic complexity. The correlation coefficients between each feature in the user feedback modality feature set and the video content modality feature set and the video content understanding and adaptation are calculated. Features whose absolute values ​​of correlation coefficients meet the preset correlation threshold are selected and integrated to obtain a multimodal key feature set. Normalization processing is performed on each feature in the multimodal key feature set, and the processed features are mapped to a preset unified feature space to obtain standardized multimodal features. Feature concatenation and dimension normalization processing are performed on the standardized multimodal features to obtain a multimodal feature vector.

[0063] Furthermore, the evaluation module is also used to input the multimodal feature vectors into the feature extraction layer of the preset multimodal fusion model to obtain the projection features of each modality; input the projection features of each modality into the modal fusion layer of the preset multimodal fusion model to obtain the global fusion features; input the global fusion features into the main task regression module of the output layer of the preset multimodal fusion model to obtain the video content understanding and adaptation score; and input the global fusion features into the auxiliary task classification module of the output layer of the preset multimodal fusion model to obtain the concept fuzzy segment localization result.

[0064] This application addresses the video content optimization needs of elderly users. It first obtains user feedback and multimodal data of the video to be evaluated, extracts multimodal feature vectors, and then inputs them into a multimodal fusion model containing a feature extraction layer, a modality fusion layer, and an output layer. This yields an adaptation score and fuzzy segment localization results, which are then used to generate specific optimization suggestions and push them to the creation terminal for video modification. This achieves a quantifiable evaluation of video comprehension and adaptation for elderly users, accurately identifies comprehension difficulties, generates targeted optimization suggestions, improves video suitability for elderly users, and automates the evaluation and optimization process, reducing manual costs and improving the efficiency and effectiveness of content optimization. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a flowchart illustrating the first embodiment of the method for optimizing video content for elderly users based on multimodal fusion in this application;

[0067] Figure 2 This is a flowchart illustrating the second embodiment of the method for optimizing video content for elderly users based on multimodal fusion in this application;

[0068] Figure 3 This is a schematic diagram of the module structure of the video content optimization system for elderly users based on multimodal fusion in this application;

[0069] Figure 4 This is a schematic diagram of the device structure of the hardware operating environment involved in the method for optimizing video content for elderly users based on multimodal fusion in the embodiments of this application.

[0070] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0071] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0072] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0073] Currently, the evaluation and optimization of video content mainly revolves around the general user group. For video content optimization specifically for elderly users, existing practices mostly rely on shallow data indicators to make a preliminary judgment on content quality, while combining the subjective experience of domain experts to complete the understanding and optimization guidance. At the data processing level, user feedback data and video content feature data are processed separately. User feedback is analyzed only through simple keyword statistics, while the focus is on technical parameter detection and analysis of video content features.

[0074] Current practices fail to design specific assessment dimensions tailored to the unique cognitive characteristics of elderly users. Relying solely on superficial indicators such as viewing time and click-through rates cannot accurately reflect their actual understanding and experience. Methods relying on subjective expert evaluation suffer from low scoring consistency, high manual assessment costs, and difficulty in large-scale application to massive amounts of content, hindering the formation of a standardized assessment system. The fragmented processing of user feedback and video content features fails to establish an effective mapping relationship and uncover the deeper connections between the two. Furthermore, the lack of targeted optimization suggestions prevents the identification of specific content segments and core dimensions that cause comprehension difficulties for elderly users, making it difficult for content creators to make precise modifications and effectively improve the understanding and adaptability of video content for elderly users. Therefore, how to achieve correlation modeling and automated assessment of the multimodal features of elderly user feedback and video content, and generate actionable optimization suggestions that pinpoint specific segments and dimensions, is a pressing issue that needs to be addressed.

[0075] Based on the above, this application also provides a method for optimizing video content for elderly users based on multimodal fusion, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the method for optimizing video content for elderly users based on multimodal fusion in this application.

[0076] In this embodiment, the method for optimizing video content for elderly users based on multimodal fusion includes steps S10 to S50:

[0077] Step S10: Obtain user feedback data and multimodal data of video content for the video to be evaluated.

[0078] It should be noted that step S10 includes: collecting raw user feedback data of the video to be evaluated from user interaction logs of the video playback platform and terminals, and dividing it into explicit feedback data and implicit feedback data according to the form of feedback expression; performing deduplication, outlier removal, and precise timestamp calibration on the explicit feedback data and implicit feedback data respectively to obtain standardized user feedback data; performing audio-video decoupling and text extraction operations on the video to be evaluated to obtain raw audio data, raw visual data, and raw text data, which are combined into raw multimodal data of video content; performing format normalization and data validity verification on the raw multimodal data of video content to obtain standardized multimodal data of video content; and performing precise alignment processing on the playback timeline of the standardized user feedback data and the standardized multimodal data of video content to obtain user feedback data and multimodal data of video content of the video to be evaluated.

[0079] It's important to understand that explicit feedback refers to the timestamp-based feedback submitted by users clicking the one-click feedback button on the screen during viewing. This directly reflects the user's subjective understanding and experience; all one-click submissions of user comprehension status while watching the video fall into this category. Implicit feedback includes user exit behavior logs and comment text. Audio-video decoupling refers to separating the composite data of the video to be evaluated, separating the integrated audio and visual information into independent audio and visual data. Text extraction refers to the operation of extracting textual information from the video to be evaluated, including video subtitles, audio-to-text transcription, and other text-related content. Original multimodal video content data refers to a dataset composed of original audio, visual, and text data, encompassing multiple dimensions of original information from video audio, visual, and text dimensions. This is the foundation of multimodal video content data processing. Format normalization refers to the process of unifying the original multimodal video content data of different formats and specifications, ensuring consistency in the format and specifications of all data types. Data validity verification refers to screening and verifying the original multimodal data of video content to determine whether the data is complete and usable, and to remove invalid, incomplete, or erroneous data. Precise alignment of the playback timeline refers to the process of matching and associating standardized user feedback data with standardized multimodal data of video content according to the video's playback time dimension, ensuring complete consistency in the time dimension information of the two types of data.

[0080] Specifically, firstly, raw data is collected from user interaction logs and terminals on the video playback platform. This data is categorized into explicit feedback (feedback submitted by users actively clicking the "Understood / Didn't Understand" button) and implicit feedback (user exit behavior, viewing time, comment text, etc.). This is because the two types of feedback reflect comprehension states differently and require differentiated processing. Secondly, explicit feedback undergoes deduplication (removing duplicate submissions within 1 second), outlier removal (excluding accidental touches), and timestamp calibration (aligning client and server time differences). Implicit feedback is processed by removing invalid sessions with viewing time less than 10 seconds and calibrating timestamps to obtain standardized user feedback data. This ensures the accuracy and time consistency of the feedback data. Then, the video to be evaluated undergoes audio-video decoupling, separating the audio stream and video frame sequence, and subtitles are extracted using OCR. Text and speech recognition extract spoken text, yielding raw audio, visual, and text data, which are then combined to form raw multimodal data of the video content. This process is used to obtain the raw material for multimodal analysis. Next, the audio, visual, and text data undergo format normalization and data validity verification (removing silent frames, black screen frames, and segments without subtitles) to obtain standardized video content multimodal data. This eliminates interference from format differences and invalid data in subsequent processing. Finally, based on the video playback timeline, the standardized user feedback data is precisely aligned with the standardized video content multimodal data, matching each feedback timestamp with the corresponding video content time point. This results in user feedback data and video content multimodal data for the video to be evaluated. This establishes a one-to-one correspondence between feedback and content, laying the foundation for subsequent feature extraction and correlation analysis.

[0081] Step S20: Extract multimodal feature vectors based on user feedback data and multimodal video content data.

[0082] It should be noted that, firstly, user feedback data is split into explicit feedback data and implicit feedback data; based on the explicit and implicit feedback data, feedback density, exit rate curves, and sentiment-semantic vectors are extracted and feature quantization is performed, and integrated to obtain the user feedback modality feature set. Specifically, in this embodiment, feedback density is the feedback submitted by the user by actively clicking the "Understood / Didn't Understand" button, let's assume... For timestamps User feedback (unit: seconds) Indicates "understood". This indicates "I didn't understand" (users can submit this by clicking the button on the screen while watching). Define the feedback density feature as a video segment The percentage of feedback indicating "didn't understand" is calculated using the following formula:

[0083]

[0084] For example, a video in If there are 15 instances of "didn't understand" feedback within the second 30-second segment, then... The exit rate curve is defined as follows: , For viewing time The number of users, For the initial number of viewers, Sampling was performed in 30-second segments to obtain the sequence. ( (Total video duration), reflecting user churn at different segments (e.g., after a certain concept appears). A sharp increase indicates that the segment is difficult to understand. Let the comment section contain 10 comments, extract each comment Keywords (such as "too fast", "good example", "can't understand") are encoded into vectors using a pre-trained BERT model. ,but The mean of all keyword vectors:

[0085] .

[0086] Then, the multimodal video content data was split into audio, visual, and text data. Audio features, visual features, and text features were extracted from each data separately, and feature quantization was performed. These features were then integrated to obtain the video content modal feature set. Audio features included speech rate, pause frequency, and pitch variation; visual features included subtitle clarity and frame complexity; and text features included vocabulary density, example richness, and syntactic complexity. Speech rate... This represents the number of words per unit of time. ,in This represents the total number of characters after audio-to-text conversion. Video duration (seconds). Pause frequency. The number of valid pauses per unit time (pauses > 0.5 seconds are considered valid). ,in The number of effective pauses (pauses help older users understand) Too low a pitch can lead to information overload. (Pitch fluctuation) The standard deviation of pitch in an audio signal. ,in For the first The fundamental frequency of the frame audio. The average fundamental frequency (too low a fluctuation will result in monotony, too high a fluctuation will cause auditory fatigue). Subtitle clarity characteristics. Total subtitle size (pixel height) ), font contrast (brightness difference between subtitles and background) The indicators of ) ,in The height of the video (in pixels). ( The higher the frame complexity, the easier the subtitles are to read. This represents the edge density of a video frame, calculated using the Sobel operator to determine the percentage of edge pixels. ,in The number of pixels at the edge. For frame resolution (older users find highly complex frames (such as dense charts) more difficult to understand). Terminology density. The percentage of technical terms in a text. ,in To match the number of words in a domain thesaurus (such as "targeted therapy" and "blood oxygen saturation" in the health field), Total number of words ( The higher the value, the higher the conceptual complexity. Example richness The ratio of relatable real-life examples to abstract concepts. ,in The number of relatable real-life examples (such as "taking body temperature with a thermometer") For abstract concepts (such as "the principle of temperature conduction") The higher the level, the easier the concept is to understand. (Syntactic complexity) It is a weighted sum of average sentence length and the proportion of modifying elements. ,in Total number of sentences Total word count The number of words used to modify elements (such as attributives and adverbs) The higher the level, the more difficult the sentence is to understand.

[0087] Next, the correlation coefficients between each feature in the user feedback modal feature set and the video content modal feature set and the understanding fit of the video content are calculated. Features with absolute correlation coefficients that meet the preset correlation threshold are selected and integrated to obtain a multimodal key feature set. Specifically, based on the manually labeled gold standard for understanding fit, which is a standardized evaluation system constructed for elderly users to quantify the degree of matching between video content and the understanding ability of elderly users, it serves as the basis for manually labeled video content understanding fit and the label foundation for subsequent model training. It consists of a weighted overall index composed of four understanding dimensions: conceptual clarity, rhythm comfort, example effectiveness, and emotional acceptance. The Pearson correlation coefficients between each feature in the user feedback modal feature set and the video content modal feature set and the understanding fit are calculated. Features with absolute correlation coefficients greater than 0.3 are selected to form the multimodal key feature set, specifically expressed as: features with Pearson correlation coefficients that meet the preset correlation threshold are selected to form a multimodal key feature set. Significantly relevant features, let feature and The correlation coefficient is :

[0088]

[0089] The screening criteria are Preserve key features (such as This indicates that the density of specialized vocabulary is negatively correlated with the fit. This indicates that example richness is positively correlated with fit. This is done to eliminate redundant features that are irrelevant to the understanding of elderly users, reduce model complexity, and improve relevance.

[0090] Next, normalization is performed on each feature in the multimodal key feature set, and the processed features are mapped to a pre-defined unified feature space to obtain standardized multimodal features. Finally, the standardized multimodal features are concatenated and their dimensions are normalized to obtain multimodal feature vectors. Specifically, Z-score normalization is performed on each feature in the multimodal key feature set to eliminate dimensional differences. The processed features are then mapped to a 128-dimensional pre-defined unified feature space to obtain standardized multimodal features. This is done to ensure comparability between different features and to adapt to the model input requirements. Finally, the standardized multimodal features are concatenated in modal order, and their dimensions are normalized through principal component analysis or linear discriminant analysis to obtain fixed-dimensional multimodal feature vectors. This is done to form a structured model input format, which facilitates subsequent fusion calculations.

[0091] Step S30: Input the multimodal feature vector into the preset multimodal fusion model to obtain the video content understanding and adaptation score and the result of fuzzy segment localization.

[0092] It should be noted that the pre-defined multimodal fusion model refers to a model that has been pre-determined after training and performance verification based on labeled and unlabeled sample sets. It can fuse multimodal feature information to achieve video content understanding and adaptation score prediction and conceptually ambiguous segment localization. It is the core of completing the evaluation task. The pre-defined multimodal fusion model includes a feature extraction layer, a modality fusion layer and an output layer. The output layer includes a main task regression module and an auxiliary task classification module.

[0093] Specifically, firstly, the multimodal feature vectors are input into the feature extraction layer of a pre-defined multimodal fusion model. User feedback modal features are mapped to a 128-dimensional space using a two-layer MLP. Audio modal features are captured using BiLSTM to capture temporal dependencies. Visual modal features are extracted using a TimeSformer spatiotemporal attention module to extract spatiotemporal relationships. Text modal features are dimensionality-reduced using an MLP to obtain projection features for each modality. This is done to transform heterogeneous multimodal data to a unified dimension while preserving the unique information structure of each modality. Secondly, the projection features of each modality are input into the modality fusion layer to construct a 4×128 modality feature matrix. The attention weights between modalities are calculated through a cross-attention mechanism, enabling the weighted fusion of different modal features. The first 256 dimensions of the attention output are then concatenated to obtain the global fusion feature. This is done to achieve deep interactive association between user feedback and content features, audio and text, and other cross-modal information. Then, the global fusion features are input to the main task regression module of the input-output layer, mapped to the [1,10] interval through two fully connected layers, batch normalization, and Sigmoid activation to obtain a video content understanding suitability score. This is done to quantitatively assess the degree of matching between the overall video and the understanding ability of elderly users. Finally, the global fusion features are segmented into 30-second segments and input into the auxiliary task classification module. Local features are extracted through convolutional layers, segment weights are calculated using an attention mechanism, and Sigmoid activation is used to map probability values. Segments with a probability greater than 0.5 are identified as ambiguous segments, resulting in the localization result of conceptually ambiguous segments. This is done to accurately locate specific time periods of difficulty in understanding while simultaneously evaluating the overall picture, achieving collaborative optimization between the overall and local aspects.

[0094] Furthermore, before step S30, the method further includes: retrieving multimodal feature data from a predetermined number of video segments in the labeled sample library, along with corresponding manually labeled fit labels and conceptually ambiguous segment labels to construct a labeled sample set; simultaneously, retrieving multimodal feature data from a predetermined number of unlabeled video segments to construct an unlabeled sample set. In this embodiment, the predetermined number is 10,000. This is done to establish a supervised learning foundation using the gold standard of manually labeled data, while simultaneously expanding the unlabeled data to improve the model's generalization ability.

[0095] Define a dual-task total loss function, where the dual-task total loss function is a weighted sum of the regression loss and the classification loss, and the specific formula is as follows:

[0096]

[0097] in, This represents the total loss function for the two tasks. Indicates regression loss, Represents classification loss. This represents the weighting coefficient, with a value range of [0, 1]. This indicates the preset quantity, which is the number of video samples used in the calculation. Indicates the first The weights of each sample, , Indicates the first The video content understanding and fit score for each manually labeled sample. Indicates the first The video content understanding fit score predicted for each sample Indicates the total number of time segments. Indicates the first A true label for a time segment Indicates a blurred segment, Indicates a non-fuzzy segment, Indicates the predicted first The probability that a time segment belongs to a fuzzy segment. Indicates the focus parameter;

[0098] An initial multimodal fusion model is constructed, including a feature extraction layer (containing user feedback MLP, audio BiLSTM, visual TimeSformer, and text MLP), a modality fusion layer (cross-attention mechanism), and an output layer (main task regression module + auxiliary task classification module). This is done to build an end-to-end multimodal understanding and adaptation evaluation architecture. Specifically, the feature extraction layer obtains user feedback modal projection features, audio modal projection features, visual modal projection features, and text modal projection features. , , Concatenate into a vector ( The user feedback modal projection features are obtained by mapping them to a unified space through a two-layer MLP. , , .Will Concatenate into a vector Temporal dependencies are captured using BiLSTM (64 hidden layer dimensions), and audio modal projection features are output. .Will The visual modality projection features are concatenated with the frame features extracted by ResNet-50 and output through the spatiotemporal attention module of TimeSformer. .Will Concatenated with BERT text vectors, text modality projection features are obtained through MLP dimensionality reduction. Then, a cross-attention mechanism is used to fuse multimodal features. Let the modal feature matrix be... Calculate the intermodal attention weights:

[0099]

[0100] in , The fusion process yields a global fusion feature. (The first 256 dimensions of the attention output are spliced ​​together). The main task of the output layer is to obtain the video content understanding and adaptation score. Scaled to Sigmoid : , , The auxiliary task is to locate conceptually ambiguous segments. The video is divided into... Each 30-second clip has the following characteristics. Output fuzzy probability , , ( (Consider it a blurry fragment).

[0101] The initial multimodal fusion model is trained in a supervised manner based on the labeled sample set to obtain the trained model. Specifically, the Adam optimizer (learning rate 0.001, batch size 32, training for 100 epochs) is used to update the model parameters by minimizing the loss function to obtain the trained model. This is done to enable the model to learn the mapping relationship between multimodal features and understand the fitness.

[0102] The trained model is used to generate pseudo-labels for the unlabeled sample set, and then iteratively trained using a mixture loss function to obtain an optimized model. The mixture loss function is a weighted sum of the losses from labeled samples and unlabeled samples, and its specific formula is as follows:

[0103]

[0104] in This represents the hybrid loss function for semi-supervised learning. This indicates the number of samples in the labeled sample set. Indicates the first Multimodal data of labeled samples, Indicates the first The true labels of the labeled samples Indicates the first The total loss of each labeled sample This represents the number of samples in the unlabeled sample set. Indicates the first Multimodal data of unlabeled samples, Indicates the first A pseudo-label for an unlabeled sample. Indicates the first Consistency loss of unlabeled samples, This represents the weighting coefficient for unlabeled data.

[0105] The optimization is then validated to select target optimization models that meet the preset performance thresholds. Specifically, the optimization models are further validated by using the mean absolute error (MAE) to evaluate the main task and the accuracy (Acc) and F1 score to evaluate the auxiliary task. Target optimization models with MAE < 0.85, Acc > 0.85, and F1 > 0.80 are selected. This is done to ensure that the model's prediction accuracy and reliability meet practical standards.

[0106] Finally, the target optimization model is used as the preset multimodal fusion model. Specifically, the target optimization model is used as the preset multimodal fusion model, the model parameters and architecture configuration are fixed, and it is deployed to the video content evaluation service port. This is done so that the trained model can be put into practical application to provide standardized understanding and adaptation assessment and fuzzy segment localization services for the videos to be evaluated.

[0107] Furthermore, the pre-defined multimodal fusion model was validated. The experiment used a constructed dataset on video content understanding suitability for elderly users, containing 10,000 labeled videos and 10,000 unlabeled videos. The labeled videos covered four high-frequency content areas for the elderly: health and wellness (30%), smart device usage (25%), financial fraud prevention (20%), and cultural entertainment (25%). Each video was 3-5 minutes long and was manually labeled by 200 elderly users aged 55-75 (male-to-female ratio 1:1, education level covering primary to university). Video content understanding suitability scores (S) and four dimensions (conceptual clarity (C), rhythm comfort (R), example effectiveness (E), and emotional acceptance (A)) were obtained. The labeling quality control requirement was an intra-group correlation coefficient (ICC) > 0.85. Three sets of comparative models were set up: Baseline 1 was a BERT regression model using only text features; Baseline 2 was a simple multimodal concatenation model (without attention fusion mechanism); and Baseline 3 was a single-task model (only optimizing regression loss, without fuzzy segment localization auxiliary task). This embodiment employs a dual-task collaborative training strategy, combining semi-supervised learning with unlabeled data. Table 1 shows the performance comparison results of each model.

[0108] Table 1. Performance Comparison Results of Each Model

[0109]

[0110] As can be seen, the model in this embodiment achieves a MAE of 0.81 on the main task, a reduction of 48.1% compared to Baseline 1, 34.1% compared to Baseline 2, and 17.3% compared to Baseline 3. On the auxiliary task, the accuracy reaches 0.87 and the F1-score reaches 0.83, significantly outperforming the comparison models. These results validate the effectiveness of the multimodal attention fusion mechanism in capturing cross-modal associations, and the contribution of dual-task collaborative training to improving overall performance.

[0111] Step S40: Based on the video content understanding and adaptation score and the results of locating conceptually ambiguous segments, generate optimization suggestions for specific segments and understanding dimensions.

[0112] It should be noted that step S40 includes: First, comparing the video content understanding fit score with a preset fit threshold. When the video content understanding fit score is lower than the preset fit threshold, extracting key optimization segments and corresponding segment location information from the conceptually ambiguous segment localization results. Specifically, comparing the video content understanding fit score with a preset fit threshold of 6 determines whether the overall video understanding fit is in a low fit range. This is done to quickly identify video content that needs optimization. Second, when the video content understanding fit score is lower than 6, selecting high-confidence segments with an ambiguity probability greater than 0.7 from the conceptually ambiguous segment localization results as key optimization segments, and extracting the start and end timestamps of each key optimization segment as location information. This is done to focus on the period with the most severe problems, avoid ineffective optimization of low-probability noise segments, and clearly mark the modification location for creators to locate. Finally, integrating the ambiguity probability values, time location information, and corresponding original values ​​of multimodal features of all key optimization segments to form a structured optimization candidate set. This is done to provide accurate data input for subsequent abnormal understanding dimension analysis and optimization rule matching, ensuring the pertinence and feasibility of optimization suggestions.

[0113] Secondly, the multimodal feature values ​​of key optimization segments are analyzed and matched against a pre-defined comprehension dimension feature library to identify abnormal comprehension dimensions in key optimization segments whose feature values ​​exceed the pre-defined feature thresholds. These multimodal feature values ​​include professional vocabulary density, example richness, speech rate, pause frequency, and subtitle clarity. It should be noted that the pre-defined comprehension dimension feature library is derived from a manually annotated gold standard for comprehension fit, constructed by combining the cognitive habits of elderly users with the multimodal feature system of video content. The library clearly distinguishes audio, visual, and text-related comprehension dimensions, and assigns multimodal features such as professional vocabulary density, example richness, speech rate, pause frequency, and subtitle clarity to their corresponding comprehension dimensions, forming a stable feature-dimensional mapping relationship. This feature library adopts an iterative update approach. After adding a large number of valid annotated samples, the correspondence between multimodal features and comprehension dimensions is re-analyzed to verify and improve existing mapping relationships, while supplementing newly emerging features and comprehension dimensions, ensuring that the content in the library continuously meets the actual needs of elderly users in assessing video content comprehension. Anomaly understanding dimension refers to the understanding dimension in key optimized segments where the feature values ​​exceed the preset feature threshold. This dimension is the main reason why elderly users have difficulty understanding video segments.

[0114] Next, the pre-defined feature-fit correlation rule library is retrieved, and optimization rules corresponding to the anomaly understanding dimension are matched. The core source of the feature-fit correlation rule library is the gold standard of understanding-fit based on manual annotation. Combining the correlation analysis of multimodal feature values ​​and understanding-fit scores of a large number of annotated samples, the correspondence between each type of feature anomaly and the decline in fit is extracted and summarized. At the same time, incorporating the research results on the cognitive characteristics of elderly users, targeted feature adjustment rules are formulated and incorporated into the rule library after multiple rounds of verification. The update method combines regular updates and dynamic updates. New annotated samples and user feedback data are collected regularly to analyze new correlation patterns between features and fit and supplement new rules. At the same time, the implementation effect of optimization suggestions is monitored, and unreasonable rules are corrected to ensure that the rule library always meets the understanding needs of elderly users and remains synchronized with model evaluation results and actual optimization scenarios. The optimization rules include feature adjustment standards and specific optimization methods for each understanding dimension. Specifically, they include: when the density of the specialized vocabulary is greater than a preset density threshold, reducing the density of the specialized vocabulary to less than a first preset value; or, when the richness of the examples is less than a preset richness threshold, increasing the richness of the examples to greater than a second preset value; or, when the speaking speed is greater than a preset speaking speed threshold, reducing the speaking speed to a third preset value and increasing the pause frequency to greater than a fourth preset value; or, when the subtitle clarity is less than a preset clarity threshold, improving the subtitle clarity. Specifically, a preset feature-adaptation association rule base is retrieved. This rule base stores the mapping relationship between feature adjustment standards and specific optimization methods for each understanding dimension. This is done to transform abstract abnormal dimensions into executable modification instructions. Secondly, based on the dimensions of abnormal understanding identified in the positioning, corresponding optimization rules are matched: When the density of professional vocabulary is greater than 0.15, rule R1 is matched, suggesting reducing the density of professional vocabulary to less than 0.1 and adding everyday analogies to explain professional terms (such as using "retirement savings account" to explain the concept of "blockchain"); when the richness of examples is less than 0.5, rule R2 is matched, suggesting adding everyday examples to make the richness of examples greater than or equal to 1, ensuring that each abstract concept is equipped with at least one everyday example; when the speech rate is greater than 3.5 words / second, rule R3 is matched, suggesting reducing the speech rate to less than or equal to 2.5 words / second and increasing the pause frequency to more than 0.5 times / second, adding a pause of more than 0.5 seconds after key concepts; when the subtitle clarity is less than 0.6, rule R4 is matched, suggesting improving the clarity of subtitles, specifically including increasing the subtitle pixel height to more than or equal to 0.08 times the video height and increasing the brightness difference between subtitles and background to more than or equal to 180. This is done to provide precise, quantitative, and actionable modification solutions for different types of comprehension obstacles. Finally, the matched optimization rules are bound to the location information of the corresponding key optimization segments to form dimensional adjustment requirements that include problem location, root cause analysis and specific modification instructions. This is done so that content creators can directly execute modifications according to timestamps and numerical standards without secondary judgment or trial adjustments.

[0115] Then, combining key optimization segments, anomaly understanding dimensions, and optimization rules, dimensional adjustment requirements for each key optimization segment are generated. Finally, the location information and dimensional adjustment requirements of all key optimization segments are integrated and structured into text according to the video playback sequence to obtain optimization suggestions. Specifically, for each key optimization segment, its time location information, the located anomaly understanding dimension, and the matching optimization rule are bound together as a triple to generate the dimensional adjustment requirements for that segment. These requirements include the segment's start and end times, anomaly dimension type, current feature value, target feature value, and specific optimization operation instructions. This is done to integrate scattered diagnostic information into a complete solution for a single problem. Secondly, the dimensional adjustment requirements of all key optimization segments are sorted according to the video playback sequence, and multi-dimensional anomalies within the same time period (such as a segment having both excessively fast speech rate and insufficient examples) are merged and integrated into comprehensive adjustment requirements to avoid repeated modifications to the same time period. Then, the sorted adjustment requirements are converted into a structured text format, including four elements: "time period + problem description + modification instructions + expected effect." For example, "02:15-02:45 Abnormal conceptual clarity (professional vocabulary density 0.18→0.08): Replace 'blockchain' with an analogy of 'retirement savings account,' expected to improve comprehension by 1.5 points." This is done to help creators quickly understand the key points and execution methods for modification. Finally, a summary of the overall video evaluation (current fit score, target score, key issue statistics) is added before the structured text to form a complete optimization suggestion document, which is then output to the content creation terminal. This is done to help creators grasp the overall optimization direction and make precise modifications accordingly.

[0116] Step S50: Optimization suggestions are sent to the content creation terminal so that the content creation terminal can modify the video to be evaluated according to the optimization suggestions.

[0117] Specifically, the generated optimization suggestion document is transmitted to the content creation terminal via API interface or message queue. The transmitted content includes structured optimization suggestion text, timeline markers for key optimization segments, and corresponding raw multimodal feature data. This is to ensure that the creation terminal can fully receive and parse the optimization instructions. Secondly, after receiving the optimization suggestions, the content creation terminal parses the positional information and dimensional adjustment requirements of each key optimization segment, highlights the time segments to be modified in the video editing interface, and displays specific operation instructions for the corresponding optimization rules. This is to reduce the creator's understanding cost and achieve intuitive linkage between problem identification and editing operations. Then, the creator performs modification operations according to the terminal prompts: adjusting the audio speed of segments with abnormal speech rate to the target value and inserting pause markers; adding analogical explanations to segments with dense professional vocabulary using the text replacement function; adjusting subtitle style parameters for segments with insufficient subtitle clarity; and supplementing missing examples with everyday case video footage. This is to transform abstract optimization suggestions into concrete and executable editing actions. Finally, after the modifications are completed, the terminal automatically saves the new version of the video and triggers a secondary evaluation process. The modified video is then re-input into the multimodal fusion model to verify the improved fit, forming a closed loop of "evaluation-optimization-verification". This is done to ensure that the modifications are effective and to accumulate data feedback for subsequent optimization cases.

[0118] Furthermore, in this embodiment, 500 low-adaptability videos (video content comprehension adaptation score S < 6) were selected for optimization effect verification. The model of this embodiment was applied to generate optimization suggestions and guide the modification, and user feedback data before and after modification were compared. For example, the original video content comprehension adaptation score of a health science popularization video "Hypertension Prevention" was S = 5.2. The model identified the period from 2 minutes 15 seconds to 2 minutes 45 seconds as the key optimization segment, and diagnosed the abnormal dimensions as "abnormal concept clarity (professional vocabulary density 0.18)" and "abnormal rhythm comfort (speech rate 3.8 words / second)". The generated optimization suggestion was "replace 'blood pressure regulation mechanism' with the analogy of 'water pipe pressure' and reduce the speech rate to 2.3 words / second". After modification, the video content comprehension adaptation score improved to S = 8.3, and the "understood" feedback rate increased from 28% to 79%, verifying the feasibility and effectiveness of the optimization suggestions generated by the method of this application.

[0119] This embodiment addresses the video content optimization needs of elderly users. First, it acquires user feedback and multimodal data of the video to be evaluated, extracts multimodal feature vectors, and then inputs them into a multimodal fusion model containing a feature extraction layer, a modality fusion layer, and an output layer. This yields an adaptation score and fuzzy segment localization results, which are then used to generate specific optimization suggestions and push them to the creation terminal for video modification. This achieves a quantifiable evaluation of video understanding and adaptation for elderly users, accurately identifies comprehension difficulties, generates targeted optimization suggestions, improves video suitability for elderly users, and automates the evaluation and optimization process, reducing manual costs and improving the efficiency and effectiveness of content optimization.

[0120] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 The method for optimizing video content for elderly users based on multimodal fusion, step S30, further includes steps S201 to S204:

[0121] Step S201: Input the multimodal feature vector into the feature extraction layer of the preset multimodal fusion model to obtain the projection features of each modality.

[0122] Specifically, the multimodal feature vectors are split into four sub-vectors based on modality type: user feedback modality features, audio modality features, visual modality features, and text modality features. This is done to employ a dedicated processing method adapted to the data structure of different modalities. Next, the user feedback modality features are input into a two-layer Multilayer Perceptron (MLP). The first layer maps the input dimension to 256 dimensions, and after ReLU activation, the second layer compresses it to 128 dimensions, obtaining the user feedback projection features. This is done to transform heterogeneous feedback data into a dense vector representation with a unified dimension. Then, the audio modality features are input into a Bidirectional Long Short-Term Memory (BiLSTM) network. The hidden layer dimension is set to 64. The forward and backward LSTMs capture the forward and backward dependencies of the audio temporal sequence, respectively, and the final hidden states are concatenated to obtain the audio modality projection features. This is done to preserve the temporal dynamic characteristics of the audio features (speech rate, pauses, and tone). Next, the visual modality features are input into the TimeSformer spatiotemporal attention module, which divides the video frame sequence into spatiotemporal segments. The inter-frame relationships are calculated using a spatiotemporal joint attention mechanism, outputting visual modality projection features. This is done to simultaneously capture the spatiotemporal contextual information of the visual content. Then, the text modality features are input into a multilayer perceptron (MLP). A two-layer fully connected network reduces the dimensionality of the text semantic vectors extracted by BERT to 128 dimensions, yielding text modality projection features. This is done to maintain dimensionality consistency with other modalities and highlight key semantic information. Finally, the user feedback modality projection features, audio modality projection features, visual modality projection features, and text modality projection features are integrated to form a set of projection features for each modality. This provides a unified-dimensional feature input with modal characteristics for subsequent modality fusion layers, enabling effective cross-modal information interaction.

[0123] Step S202: Input the projection features of each modality into the modal fusion layer of the preset multimodal fusion model to obtain global fusion features.

[0124] Specifically, firstly, the modal projection features are stacked row-wise to construct a modal feature matrix. This is done to organize the scattered modal features into a structured tensor, facilitating matrix operations and attention calculations. Secondly, the modal feature matrix is ​​simultaneously used as the query matrix, key matrix, and value matrix as input to the cross-attention mechanism to calculate the attention score matrix. Then, it is normalized using the Softmax function to obtain the attention weight matrix. This is done to calculate the inter-modal correlation strength between different modal features, enabling the model to automatically learn the cross-modal correspondence between "user feedback" and "content features." Next, the attention weight matrix is ​​multiplied by the value matrix to obtain the attention output. The first 256 dimensions of the attention output are concatenated, and the training process is stabilized through layer normalization and residual connections to obtain the global fusion feature. This is done to compress multimodal information into a fixed-dimensional global representation while retaining the most critical cross-modal correlation information, providing a comprehensive decision-making basis for subsequent dual-task outputs.

[0125] Step S203: Input the global fusion features into the main task regression module of the output layer of the preset multimodal fusion model to obtain the video content understanding and adaptation score.

[0126] It should be noted that step S203 includes: inputting the global fusion features into the first fully connected layer of the main task regression module for feature dimension compression to obtain the first regression feature; inputting the first regression feature into the batch normalization layer of the main task regression module for feature distribution normalization to obtain the second regression feature; inputting the second regression feature into the activation function layer of the main task regression module for nonlinear mapping to obtain the third regression feature; and inputting the third regression feature into the second fully connected layer of the main task regression module to map to a preset scoring interval to obtain the video content understanding and adaptation score.

[0127] Specifically, the globally fused features are input into the first fully connected layer of the main task regression module. A linear transformation is performed using the weight matrix and bias vector to compress the feature dimension from 256 to 64, resulting in the first regression feature. This is done to reduce the feature dimensionality, extract the higher-order semantic information most relevant to understanding fit, and reduce subsequent computational complexity. Next, the first regression feature is input into a batch normalization layer to calculate the batch mean and variance, followed by normalization. Finally, scaling and shifting are performed using learnable parameters to obtain the second regression feature. This is done to stabilize the feature distribution, accelerate model convergence, and prevent gradient vanishing or exploding. Then, the second regression feature is input into a ReLU activation function layer to obtain the third regression feature. This introduces a non-linear transformation to enhance the model's ability to fit complex mapping relationships. Finally, the third regression feature is input into the second fully connected layer. The weight matrix and bias are calculated, and then mapped to the [0,1] interval by the Sigmoid function. The feature is then scaled to the preset scoring interval [1,10] by linear transformation to obtain the video content understanding and fit score. This is done to constrain the model output to a 1-10 score that conforms to the human annotation habits, so as to facilitate comparison with the gold standard and interpretation in actual business.

[0128] Step S204: Input the global fusion features into the output layer auxiliary task classification module of the preset multimodal fusion model to obtain the concept fuzzy fragment localization result.

[0129] It should be noted that step S204 includes: inputting the global fusion features into the convolutional layer of the auxiliary task classification module to extract segment-level local features, obtaining the first classification feature; inputting the first classification feature into the attention mechanism layer of the auxiliary task classification module to calculate the attention weight of each segment feature, obtaining the second classification feature; inputting the second classification feature into the batch normalization layer of the auxiliary task classification module to normalize the feature distribution, obtaining the third classification feature; inputting the third classification feature into the Sigmoid activation layer of the auxiliary task classification module to map to the 0-1 interval, obtaining the blur probability of each video segment; comparing the blur probability with a preset blur threshold, filtering out segments with blur probabilities higher than the preset blur threshold, and obtaining the concept blur segment localization result.

[0130] Specifically, the global fusion features are uniformly divided into n 30-second segments along the video timeline. These segments are then input into a one-dimensional convolutional layer with a kernel size of 3, a stride of 1, and 128 output channels. The convolutional kernel slides along the temporal dimension to calculate the weighted sum of features within the local receptive field, extracting local contextual information between adjacent segments to obtain the first classification feature. This is done to capture segment-level local temporal patterns and identify boundary points where the difficulty of comprehension changes abruptly. Next, the first classification feature is input into an attention mechanism layer to calculate the attention weight matrix between the features of each segment. This allows the model to focus on the key time-period features most relevant to the location of ambiguous segments, and then perform a weighted sum to obtain the second classification feature. This allows the model to automatically learn the differences in importance between different segments and focus on areas where comprehension is truly difficult. Then, the second classification feature is input into a batch normalization layer to calculate the mean and variance of each channel within the batch. After normalization, the mean and variance are scaled and shifted using learnable parameters to obtain a stable third classification feature. This is done to accelerate training convergence and prevent overfitting. Next, the third classification feature is input into the Sigmoid activation layer. A non-linear function independently maps the feature value of each segment to the 0-1 interval, calculating the blur probability of each video segment. This is done to transform segment features into interpretable probability values, representing the likelihood that the segment is conceptually ambiguous. Finally, the blur probability of each segment is compared one by one with a preset blur threshold of 0.5. A set of segment indices satisfying a blur probability > 0.5 is selected and marked as conceptually ambiguous segments. The start and end timestamps and blur probability values ​​of each ambiguous segment are recorded to obtain the conceptually ambiguous segment localization result. This is done to output clear binary classification judgment and localization information, providing accurate problem time period labeling for subsequent optimization suggestion generation.

[0131] In this embodiment, multimodal feature vectors are sequentially input into the feature extraction layer and modality fusion layer of the model to obtain the projection features of each modality and the global fusion features. Then, through the main task regression module and the auxiliary task classification module of the output layer, the fitness score and fuzzy fragment localization results are obtained simultaneously. This realizes the deep fusion of multimodal features and dual-task synchronous reasoning, improves the evaluation accuracy and efficiency, and provides a reliable basis for subsequent precise optimization.

[0132] Based on the first embodiment of this application, this application also provides a video content optimization system for elderly users based on multimodal fusion. Please refer to [link / reference]. Figure 3 The device includes:

[0133] The acquisition module 10 is used to acquire user feedback data and multimodal data of video content of the video to be evaluated.

[0134] The feature extraction module 20 is used to extract multimodal feature vectors based on user feedback data and multimodal data of video content.

[0135] Evaluation module 30 is used to input multimodal feature vectors into a preset multimodal fusion model to obtain video content understanding and adaptation scores and conceptually ambiguous segment localization results. The preset multimodal fusion model includes a feature extraction layer, a modality fusion layer, and an output layer. The output layer includes a main task regression module and an auxiliary task classification module.

[0136] The optimization module 40 is used to generate optimization suggestions for specific segments and understanding dimensions based on the video content understanding and adaptation score and the location results of conceptually ambiguous segments.

[0137] The sending module 50 is used to send optimization suggestions to the content creation terminal so that the content creation terminal can modify the video to be evaluated according to the optimization suggestions.

[0138] The video content optimization system for elderly users based on multimodal fusion provided in this application, employing the video content optimization method for elderly users based on multimodal fusion described in the above embodiments, can solve the technical problem of how to automatically assess and optimize the understanding and adaptation of video content for elderly users. Compared with the prior art, the beneficial effects of the video content optimization system for elderly users based on multimodal fusion provided in this application are the same as those of the video content optimization method for elderly users based on multimodal fusion provided in the above embodiments, and other technical features of the video content optimization system for elderly users based on multimodal fusion are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0139] In one example, the feature extraction module 20 is further configured to split the user feedback data into explicit feedback data and implicit feedback data; extract feedback density, exit rate curve, and sentiment-semantic vector based on the explicit and implicit feedback data and perform feature quantization, integrating them to obtain a user feedback modal feature set; split the video content multimodal data into audio data, visual data, and text data; extract audio features, visual features, and text features based on the audio data, visual data, and text data respectively and perform feature quantization, integrating them to obtain a video content modal feature set, wherein the audio features include speech rate, pause frequency, and pitch fluctuation, and the visual features include... The features include subtitle clarity and frame complexity. The text features include professional vocabulary density, example richness, and syntactic complexity. The correlation coefficients between each feature in the user feedback modality feature set and the video content modality feature set and the video content understanding and adaptation are calculated. Features whose absolute values ​​of correlation coefficients meet the preset correlation threshold are selected and integrated to obtain a multimodal key feature set. Normalization processing is performed on each feature in the multimodal key feature set, and the processed features are mapped to a preset unified feature space to obtain standardized multimodal features. Feature concatenation and dimension normalization processing are performed on the standardized multimodal features to obtain a multimodal feature vector.

[0140] In one example, the evaluation module 30 is further configured to input the multimodal feature vector into the feature extraction layer of the preset multimodal fusion model to obtain the projection features of each modality; input the projection features of each modality into the modal fusion layer of the preset multimodal fusion model to obtain the global fusion features; input the global fusion features into the main task regression module of the output layer of the preset multimodal fusion model to obtain the video content understanding and adaptation score; and input the global fusion features into the auxiliary task classification module of the output layer of the preset multimodal fusion model to obtain the concept fuzzy segment localization result.

[0141] This application provides a video content optimization device for elderly users based on multimodal fusion. The video content optimization device for elderly users based on multimodal fusion includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the video content optimization method for elderly users based on multimodal fusion in the above embodiment 1.

[0142] The following is for reference. Figure 4The diagram illustrates a structural schematic suitable for implementing the video content optimization device for elderly users based on multimodal fusion in the embodiments of this application. The video content optimization device for elderly users based on multimodal fusion in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The illustrated video content optimization device for elderly users based on multimodal fusion is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments in this application.

[0143] like Figure 4 As shown, a video content optimization device for elderly users based on multimodal fusion may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the video content optimization device for elderly users based on multimodal fusion. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the multimodal fusion-based video content optimization device for elderly users to exchange data wirelessly or via wired communication with other devices. Although various multimodal fusion-based video content optimization devices for elderly users are shown in the figures, it should be understood that implementation or possession of all of them is not required. More or fewer of these devices may be implemented alternatively.

[0144] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0145] The video content optimization device for elderly users based on multimodal fusion provided in this application, employing the video content optimization method for elderly users based on multimodal fusion described in the above embodiments, can solve the technical problem of how to automatically assess and optimize the understanding and adaptation of video content for elderly users. Compared with the prior art, the beneficial effects of the video content optimization device for elderly users based on multimodal fusion provided in this application are the same as those of the video content optimization method for elderly users based on multimodal fusion provided in the above embodiments, and other technical features in this video content optimization device for elderly users based on multimodal fusion are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0146] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0147] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0148] This application provides a computer-readable medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the multimodal fusion-based video content optimization method for elderly users in the above embodiments.

[0149] The computer-readable medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, or any combination thereof. More specific examples of computer-readable media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable medium may be any tangible medium containing or storing a program that can be executed by instructions, used by a device, or used in conjunction with it. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0150] The aforementioned computer-readable medium may be included in a multimodal fusion-based video content optimization device for elderly users; or it may exist independently and not be assembled into a multimodal fusion-based video content optimization device for elderly users.

[0151] The aforementioned computer-readable medium carries one or more programs that, when executed by a multimodal fusion-based video content optimization device for elderly users, enable the device to write computer program code for performing the operations of this application in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this application. In this regard, all blocks in the flowcharts or block diagrams may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that all blocks in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using dedicated hardware-based implementations that perform the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0153] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0154] The readable medium provided in this application is a computer-readable medium, which stores computer-readable program instructions (i.e., a computer program) for executing the above-described method for optimizing video content for elderly users based on multimodal fusion. This solves the technical problem of how to automatically assess and optimize the understanding and adaptation of video content for elderly users. Compared with the prior art, the beneficial effects of the computer-readable medium provided in this application are the same as those of the method for optimizing video content for elderly users based on multimodal fusion provided in the above embodiments, and will not be repeated here.

[0155] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method for optimizing video content for elderly users based on multimodal fusion.

[0156] The computer program product provided in this application can solve the technical problem of how to automatically assess and optimize the understanding and adaptation of video content for elderly users. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the multimodal fusion-based video content optimization method for elderly users provided in the above embodiments, and will not be repeated here.

[0157] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for optimizing video content for elderly users based on multimodal fusion, characterized in that, The method includes: Obtain user feedback data and multimodal data of video content for the video to be evaluated; Multimodal feature vectors are extracted based on the user feedback data and the multimodal data of the video content; The multimodal feature vectors are input into a preset multimodal fusion model to obtain video content understanding and adaptation scores and conceptually ambiguous segment localization results. The preset multimodal fusion model includes a feature extraction layer, a modality fusion layer, and an output layer. The output layer includes a main task regression module and an auxiliary task classification module. Based on the video content understanding and adaptation score and the location results of the conceptually ambiguous segments, optimization suggestions are generated for specific segments and understanding dimensions. The optimization suggestions are sent to the content creation terminal so that the content creation terminal can modify the video to be evaluated according to the optimization suggestions; The step of inputting the multimodal feature vector into a preset multimodal fusion model to obtain the video content understanding and adaptation score and the result of locating conceptually ambiguous segments includes: The multimodal feature vectors are input into the feature extraction layer of the preset multimodal fusion model to obtain the projection features of each modality; The projection features of each modality are input into the modal fusion layer of the preset multimodal fusion model to obtain global fusion features; The global fusion features are input into the main task regression module of the output layer of the preset multimodal fusion model to obtain the video content understanding and adaptation score. The global fusion features are input into the output layer auxiliary task classification module of the preset multimodal fusion model to obtain the concept fuzzy fragment localization result; The step of generating optimization suggestions for specific segments and understanding dimensions based on the video content understanding fit score and the conceptually ambiguous segment localization result includes: The video content understanding and adaptation score is compared with a preset adaptation threshold. When the video content understanding and adaptation score is lower than the preset adaptation threshold, the key optimized segments and corresponding segment location information in the conceptual ambiguity segment localization results are extracted. The multimodal feature values ​​of the key optimization segment are analyzed and matched with a preset understanding dimension feature library to locate abnormal understanding dimensions in the key optimization segment whose feature values ​​exceed a preset feature threshold. The multimodal feature values ​​include professional vocabulary density, example richness, speech rate, pause frequency and subtitle clarity. Retrieve a preset feature-adaptation association rule library and match it with the optimization rule corresponding to the anomaly understanding dimension, wherein the optimization rule includes feature adjustment criteria and specific optimization methods for each understanding dimension; By combining the key optimization segments, the anomaly understanding dimensions, and the optimization rules, dimensional adjustment requirements for each key optimization segment are generated; By integrating the location information of all key optimization segments and the aforementioned dimensional adjustment requirements, and organizing them into structured text according to the video playback sequence, optimization suggestions are obtained.

2. The method as described in claim 1, characterized in that, The steps for obtaining user feedback data and multimodal data of video content to be evaluated include: The raw user feedback data of the video to be evaluated is collected from the user interaction logs of the video playback platform and the terminal. The data is divided into explicit feedback data and implicit feedback data according to the form of feedback expression. The explicit feedback is the timestamp feedback submitted by the user when clicking the one-click feedback button on the screen during the viewing process. The implicit feedback includes user exit behavior logs and text in the comment section. The explicit feedback data and the implicit feedback data are respectively subjected to deduplication, outlier removal and timestamp precision calibration to obtain standardized user feedback data; The video to be evaluated is subjected to audio-video decoupling and text extraction operations to obtain raw audio data, raw visual data and raw text data, which are combined into raw multimodal data of video content; The original multimodal data of the video content is normalized in format and validated for data validity to obtain standardized multimodal data of video content; The standardized user feedback data and the standardized video content multimodal data are precisely aligned along the playback timeline to obtain the user feedback data and video content multimodal data of the video to be evaluated.

3. The method as described in claim 1, characterized in that, The step of extracting multimodal feature vectors based on the user feedback data and the video content multimodal data includes: The user feedback data is split into explicit feedback data and implicit feedback data; Based on the explicit and implicit feedback data, the feedback density, exit rate curve, and sentiment-semantic vector are extracted and feature quantization is completed. The results are then integrated to obtain the user feedback modality feature set. The video content multimodal data is split into audio data, visual data, and text data; Based on the audio data, visual data, and text data, audio features, visual features, and text features are extracted and quantized respectively, and integrated to obtain a video content modal feature set. The audio features include speech rate, pause frequency, and pitch fluctuation. The visual features include subtitle clarity and frame complexity. The text features include professional vocabulary density, example richness, and syntactic complexity. Calculate the correlation coefficients between each feature in the user feedback modal feature set and the video content modal feature set and the video content understanding and adaptation, select features whose absolute values ​​of correlation coefficients meet the preset correlation threshold, and integrate them to obtain a multimodal key feature set; Normalization processing is performed on each feature in the multimodal key feature set, and the processed features are mapped to a preset unified feature space to obtain standardized multimodal features; The standardized multimodal features are subjected to feature concatenation and dimension normalization to obtain multimodal feature vectors.

4. The method as described in claim 1, characterized in that, Before the step of inputting the multimodal feature vector into a preset multimodal fusion model to obtain the video content understanding fit score and the result of fuzzy segment localization, the following steps are included: A labeled sample set is constructed by retrieving multimodal feature data of a predetermined number of video segments from the labeled sample library, along with corresponding manually labeled fit labels and conceptually ambiguous segment labels. At the same time, a labeled sample set is constructed by retrieving multimodal feature data of a predetermined number of unlabeled video segments. Define a dual-task total loss function, where the dual-task total loss function is a weighted sum of regression loss and classification loss; Build an initial multimodal fusion model; The initial multimodal fusion model is trained in a supervised manner based on the labeled sample set to obtain the trained model; The trained model is used to generate pseudo-labels for the unlabeled sample set, and iterative training is performed using a hybrid loss function to obtain an optimized model, wherein the hybrid loss function is a weighted sum of the labeled sample loss and the unlabeled sample loss. The optimization model is validated for performance, and target optimization models that meet the preset performance threshold are selected. The target optimization model is used as a preset multimodal fusion model.

5. The method as described in claim 1, characterized in that, The step of inputting the global fusion features into the output layer main task regression module of the preset multimodal fusion model to obtain the video content understanding and adaptation score includes: The global fusion features are input into the first fully connected layer of the main task regression module for feature dimension compression to obtain the first regression features; The first regression feature is input into the batch normalization layer of the main task regression module for feature distribution normalization to obtain the second regression feature; The second regression feature is input into the activation function layer of the main task regression module for nonlinear mapping to obtain the third regression feature; The third regression feature is input into the second fully connected layer of the main task regression module and mapped to a preset scoring range to obtain the video content understanding and adaptation score. The step of inputting the global fusion features into the output layer auxiliary task classification module of the preset multimodal fusion model to obtain the concept fuzzy fragment localization result includes: The global fusion features are input into the convolutional layer of the auxiliary task classification module to extract fragment-level local features, thus obtaining the first classification features; The first classification feature is input into the attention mechanism layer of the auxiliary task classification module to calculate the attention weight of each segment feature, and the second classification feature is obtained. The second classification feature is input into the batch normalization layer of the auxiliary task classification module for feature distribution normalization to obtain the third classification feature; The third classification feature is input into the Sigmoid activation layer of the auxiliary task classification module and mapped to the 0-1 interval to obtain the blur probability of each video segment. The fuzzy probability is compared with a preset fuzzy threshold, and segments with fuzzy probabilities higher than the preset fuzzy threshold are selected to obtain the concept fuzzy segment localization result.

6. A video content optimization system for elderly users based on multimodal fusion, characterized in that, The system is applied to the method for optimizing video content for elderly users based on multimodal fusion as described in any one of claims 1-5, and the system comprises: The acquisition module is used to acquire user feedback data and multimodal data of video content for the video to be evaluated; The feature extraction module is used to extract multimodal feature vectors based on the user feedback data and the multimodal data of the video content; The evaluation module is used to input the multimodal feature vector into a preset multimodal fusion model to obtain a video content understanding and adaptation score and a conceptually ambiguous segment localization result. The preset multimodal fusion model includes a feature extraction layer, a modality fusion layer and an output layer. The output layer includes a main task regression module and an auxiliary task classification module. The optimization module is used to generate optimization suggestions for specific segments and understanding dimensions based on the video content understanding and adaptation score and the location results of the conceptually ambiguous segments. The sending module is used to send the optimization suggestions to the content creation terminal, so that the content creation terminal can modify the video to be evaluated according to the optimization suggestions.

7. The system as described in claim 6, characterized in that, The feature extraction module is further configured to split the user feedback data into explicit feedback data and implicit feedback data; extract feedback density, exit rate curve, and sentiment-semantic vector based on the explicit and implicit feedback data and perform feature quantization, and integrate them to obtain a user feedback modal feature set; split the video content multimodal data into audio data, visual data, and text data; extract audio features, visual features, and text features based on the audio data, visual data, and text data respectively and perform feature quantization, and integrate them to obtain a video content modal feature set, wherein the audio features include speech rate, pause frequency, and pitch fluctuation; the visual features include subtitle clarity and frame complexity; and the text features include professional vocabulary density, example richness, and syntactic complexity features. Calculate the correlation coefficients between each feature in the user feedback modal feature set and the video content modal feature set and the video content understanding and adaptation, select features whose absolute correlation coefficients meet a preset correlation threshold, and integrate them to obtain a multimodal key feature set; perform normalization processing on each feature in the multimodal key feature set, and map the processed features to a preset unified feature space to obtain standardized multimodal features; perform feature concatenation and dimension normalization processing on the standardized multimodal features to obtain a multimodal feature vector.

8. The system as described in claim 6, characterized in that, The evaluation module is also used to input the multimodal feature vector into the feature extraction layer of the preset multimodal fusion model to obtain the projection features of each modality; and to input the projection features of each modality into the modal fusion layer of the preset multimodal fusion model to obtain the global fusion features; The global fusion features are input into the main task regression module of the output layer of the preset multimodal fusion model to obtain the video content understanding and adaptation score; the global fusion features are input into the auxiliary task classification module of the output layer of the preset multimodal fusion model to obtain the concept fuzzy segment localization result.

Images