Speech video intelligent scoring method based on multi-dimensional feature extraction

Abstract

The invention relates to the technical field of computer vision and natural language processing, in particular to a speech video intelligent scoring method based on multi-dimensional feature extraction, which comprises the following steps: firstly, extracting multi-modal features of a speech video, and generating innovative feature data by using a self-supervised meta-learning algorithm; and then virtual view conversion is carried out on the video data by adopting a generative adversarial network to enhance visual features. And performing joint scoring on the audio, video and text features through the neural symbol mixed evaluation model, and finally generating comprehensive scoring data. In order to ensure transparency and non-tampering performance of the scoring process, scoring data are encrypted through digital signatures and are uploaded and recorded. The scoring accuracy and stability are improved, the scoring process is automatic and traceable, and the problems of subjectivity, low efficiency, non-traceability and the like in an existing scoring method are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and natural language processing, and particularly relates to a speech video intelligent scoring method based on multi-dimensional feature extraction. BACKGROUND

[0002] Traditional speech scoring methods usually rely on a single modality or manual scoring, which is not only susceptible to subjective factors, but also cannot comprehensively evaluate the performance of speech, emotion, language fluency, body language and other dimensions, resulting in inaccurate and inconsistent scoring results. Existing technologies, such as single-modal audio or text analysis methods, can only score speeches from a certain perspective, ignoring the influence of other dimensions, often failing to accurately reflect the overall quality of speeches, and manual scoring is not only inefficient but also has a large subjective error. Based on these problems, developing a multi-modal, dynamic and adaptive scoring method that can comprehensively evaluate speech videos from different dimensions can not only improve the accuracy and consistency of scoring, but also improve the automation and efficiency of the scoring process, becoming an important direction of current research. SUMMARY

[0003] In view of the many problems existing in the prior art, the present application provides a speech video intelligent scoring method based on multi-dimensional feature extraction. The present application realizes intelligent scoring through multi-modal data fusion, dynamic weight adjustment and multi-task scoring method. First, video, audio and text features are extracted from the speech video and fused through a deep learning model. Next, a self-supervised meta-learning algorithm is used to generate innovative features, a generative adversarial network is used for visual enhancement, and a neural-symbol hybrid evaluation is used to comprehensively score multi-modal features. Finally, the scoring results are encrypted and stored by blockchain technology to ensure the transparency and tamper resistance of the scoring results. The present application effectively improves the accuracy and stability of scoring, while reducing manual intervention and improving scoring efficiency.

[0004] A speech video intelligent scoring method based on multi-dimensional feature extraction, the present application comprises the following steps:

[0005] Obtain multi-modal data of the speech scene, wherein the multi-modal data includes video data, audio data and text data; after preprocessing the multi-modal data, perform conventional feature extraction on the video, audio and text respectively, and perform deep fusion on the features of each modality to generate fused feature data;

[0006] Implementing innovative multi-modal feature reconstruction and joint fusion based on the fusion feature data, which includes: generating first innovative feature data from the fusion feature data using a self-supervised meta-learning algorithm; performing virtual view conversion on the video data using a generative adversarial network and extracting enhanced visual features to generate second innovative feature data; and integrating the first innovative feature data, second innovative feature data, and conventional extracted video, audio, and text features, and implementing joint fusion using neural-symbol hybrid evaluation to generate innovative fusion feature data;

[0007] Implementing dynamic adaptive multi-task scoring and transparent explanation feedback based on the innovative fusion feature data, which includes: generating dynamic weights according to data quality indicators, and scoring the features of video, audio, and text using multi-task scoring branches, and generating comprehensive scoring data by weighted aggregation according to the dynamic weights; analyzing the contribution of each scoring dimension to generate scoring basis information using an explanatory algorithm; and digitally signing and recording the comprehensive scoring data and scoring basis information on the chain.

[0008] Preferably, the multi-modal data includes video data, audio data, and text data of a speech, the video data is collected by a high-resolution camera and a depth sensor, the audio data is collected by a plurality of high-fidelity microphone arrays, and the text data is generated by real-time transcription of audio data using a speech recognition system based on a Transformer model.

[0009] Preferably, the preprocessing step includes the following processing of the multi-modal data: denoising the video data, optimizing image color using a color correction algorithm, and removing irrelevant frames through a key frame extraction algorithm; implementing noise suppression and frame division on the audio data to improve the signal-to-noise ratio; performing grammar correction, automatic punctuation completion, and splitting the text into lexical units through word segmentation processing on the text data.

[0010] Preferably, the multi-modal data is deeply fused through a cross-attention mechanism to generate unified fusion feature data containing cross-modal context associations for subsequent innovative feature generation and evaluation.

[0011] Preferably, the self-supervised meta-learning algorithm uses a model-independent meta-learning based training strategy to perform meta-training on the speech data to learn the ability to quickly adapt between different tasks, and the generated first innovative feature data can extract deep semantic and emotional information of multi-modal data and has cross-modal transferability.

[0012] Preferably, the generative adversarial network is used for virtual perspective conversion of the video data, and in the conversion process, the generator generates videos of multiple different perspectives according to the original video content while maintaining the diversity of the light, background and character angle in the scene, extracts enhanced visual features to generate second innovative feature data.

[0013] Preferably, the neural-symbol hybrid evaluation module includes two parts: a neural network part for preliminary evaluation of the input first innovative feature data, second innovative feature data and traditional extracted text, audio and video features based on a deep learning model, and a symbol logic reasoning part for reasoning and correction according to the grammar structure, emotional expression and body language rules of the speech content.

[0014] Preferably, the dynamic weight is generated by a lightweight gating network according to the data quality indicators, the initial dynamic weight data is adjusted according to the video quality, audio signal-to-noise ratio and text transcription confidence, and the weight data is iteratively updated by feedback residual in the scoring process, and finally the final dynamic weight data conforming to the performance of each modal feature is generated.

[0015] Preferably, the multi-task scoring branch includes a text scoring branch, an audio scoring branch and a video scoring branch, the text scoring branch evaluates the grammar structure, vocabulary richness and grammatical errors of the speech through a fully connected neural network, the audio scoring branch evaluates the emotional tendency, speech fluency and pause length of the speech through a long short-term memory network or a Transformer model, and the video scoring branch evaluates the body language and speech style of the speaker through a three-dimensional convolutional neural network or a Transformer model, and generates corresponding scoring data for each dimension.

[0016] Preferably, the comprehensive scoring data and scoring basis information are encrypted by digital signature technology and stored on a blockchain platform, the storage step includes hashing and digitally signing the comprehensive scoring data, scoring basis information and other related data, and then uploading the calculation results to the blockchain.

[0017] Compared with the prior art, the advantages and beneficial effects of the present application are that:

[0018] The present application solves the limitations of the prior art in multi-modal scoring by introducing innovative technical means.

[0019] The present application can deeply integrate audio, text and video data in the speech video from multiple dimensions by using self-supervised meta-learning algorithm, generative adversarial network (GAN) and neural-symbol hybrid evaluation model, and generate more accurate and stable scoring results.

[0020] The application realizes innovative reconstruction of multi-modal features by using a self-supervised meta-learning algorithm, so that the scoring model can quickly adapt to various speech scenes, and the accuracy and flexibility of scoring are improved.

[0021] The application can better capture the details of body language and the emotional changes of the speaker in the video data processing by generating virtual perspective conversion and enhanced visual feature extraction of the generative adversarial network, thereby improving the performance of video scoring.

[0022] The application combines symbolic logic reasoning and deep learning through the neural-symbol hybrid evaluation model, which can not only deeply analyze grammar, emotion and body language, but also ensure the rationality and consistency of the scoring process. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The flowchart of the application is shown in the figure.

[0024] Figure 2 The figure shows the generation of virtual perspective video and the extraction of enhanced visual features in the application.

[0025] Figure 3 The figure shows the neural-symbol hybrid evaluation in the application. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments can be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concept of the present disclosure.

[0027] The terms used herein are merely used to describe specific embodiments and are not intended to limit the present disclosure. The terms "include", "comprise" and the like used herein indicate the presence of the described features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0028] All terms used herein (including technical and scientific terms) have meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted to have meanings consistent with the context of the present specification, and should not be interpreted in an idealized or overly formal manner.

[0029] As Figure 1As shown, a speech video intelligent scoring method based on multi-dimensional feature extraction, the present application comprises the following steps:

[0030] Multi-modal data of the speech scene is acquired, wherein the multi-modal data comprises video data, audio data and text data; after pre-processing the multi-modal data, conventional feature extraction is performed on the video, audio and text respectively, and the features of each modality are deeply fused to generate fusion feature data;

[0031] The speech video intelligent scoring method based on multi-dimensional feature extraction aims to comprehensively acquire multi-modal data of the speech scene, and perform pre-processing, feature extraction and deep fusion on these data to form fusion feature data with cross-modal context association, thereby providing a solid data foundation and information carrier for subsequent innovative feature reconstruction, joint fusion and dynamic adaptive multi-task scoring. Overall, the present application comprises three main steps: first, multi-modal data of the speech scene is acquired, which includes video data, audio data and text data; second, the acquired multi-modal data is pre-processed, time-aligned and conventionally feature-extracted, and multi-modal Transformer model is used to deeply fuse the features of different modalities to generate fusion feature data; finally, based on the fusion feature data, self-supervised meta-learning, generative adversarial network and neural-symbol hybrid evaluation are used to reconstruct and jointly fuse the original features to form innovative fusion feature data, and dynamic weights are generated based on data quality indicators, and multi-task scoring branches are used to realize comprehensive scoring of text, audio and video information, and explanatory algorithms are used to generate scoring basis information, and the final comprehensive scoring data and its scoring basis are stored through digital signature on-chain, thereby realizing the non-tamperability and full-process transparent traceability of the scoring data.

[0032] The core of the present application is to use the multi-modal data acquired in the speech scene as input, ensure data quality through pre-processing and other conventional steps, and realize effective integration of cross-modal information through advanced deep fusion technology. Through deep fusion, subtle emotional changes and semantic relationships in the speech video can be captured, providing more rich and accurate information support for the scoring system. The present application not only solves the limitations of traditional single-modal or simple multi-modal processing in information integration, but also realizes the transition from traditional features to high-dimensional abstract features through subsequent innovative feature reconstruction and joint fusion, thereby significantly improving the scoring accuracy and system robustness.

[0033] From data acquisition to final scoring, the present application constitutes a complete closed-loop system. First, in the data acquisition stage, video, audio and text data obtained through high-precision equipment can fully reflect the speech situation on site; second, in the preprocessing stage, denoising, color correction, key frame extraction, and frame division, text correction and other technologies significantly improve the quality of the original data, and through time alignment ensure that the modal data strictly corresponds in time; third, in the feature extraction stage, by extracting visual features in video, acoustic features in audio and semantic features in text respectively, and using multi-modal Transformer to realize deep fusion, fusion feature data containing cross-modal context association is generated; finally, through innovative feature reconstruction and joint fusion, using self-supervised meta-learning and generative adversarial network to extract first and second innovative feature data from fusion feature data respectively, and combining with neural-symbol hybrid evaluation module to comprehensively evaluate all features, innovative fusion feature data is generated, then through dynamic weight and multi-task scoring branch to realize comprehensive scoring, and using explanatory algorithm to generate detailed scoring basis, finally through digital signature and blockchain on-chain storage to ensure the safety and transparency of the scoring data.

[0034] The present application not only adopts mature technical means in data acquisition, preprocessing and feature extraction, but also introduces cutting-edge self-supervised meta-learning, generative adversarial network and neural-symbol hybrid evaluation technology in subsequent innovative feature reconstruction, joint fusion and intelligent scoring links. The synergistic effect of these technologies enables the system to adaptively adjust when facing different speech scenarios, and effectively overcomes the influence of data noise, environmental changes and subjective scoring factors, thereby greatly improving the accuracy and robustness of the scoring. Through the whole process of data processing and intelligent scoring closed-loop design, the present method realizes the transparent traceability from the original data to the scoring result, and provides a feedback loop for the continuous optimization of subsequent models, reflecting the advancedness and operability of the overall design of the system.

[0035] Preferably, the multi-modal data includes video data, audio data and text data of the speech, the video data is collected by a high-resolution camera and a depth sensor, the audio data is collected by a plurality of high-fidelity microphone arrays, and the text data is generated by real-time transcription of the audio data by a speech recognition system based on a Transformer model.

[0036] The application preferably adopts a high-precision multi-modal data acquisition device to ensure comprehensive recording of the information in the speech site, wherein the video data is acquired by a high-resolution camera combined with a depth sensor, which can simultaneously capture the color image and depth information of the speaker, and realize accurate recording of the posture, facial expression and three-dimensional structure of the background. In specific operation, the camera device is set to high-resolution mode, and the depth map of each frame of image is obtained through the built-in depth sensor, while the timestamp of each frame is recorded to ensure the timing accuracy of the video data. The audio data is synchronously acquired by multiple high-fidelity microphone arrays, and the clear capture of the speaker's voice is realized by using hardware noise reduction and echo cancellation technology; in actual application, the arrangement of the microphone array is optimized to ensure that clear signals can be obtained from different directions, thereby reducing the interference of environmental noise. The text data is converted into real-time transcription of the audio data by an end-to-end speech recognition system based on Transformer, which outputs detailed recognition confidence during the transcription process, facilitating the subsequent judgment of data quality. In the application, the multi-modal data obtained by the above method has high signal-to-noise ratio and high timeliness in the acquisition stage, providing sufficient and accurate raw information for subsequent preprocessing, feature extraction and deep fusion.

[0037] In addition, in actual field application, the data acquisition device can be flexibly adjusted according to the lighting, noise and personnel distribution of the speech site, for example, for low-illumination scenes, the camera can automatically switch to high-gain mode; in a high-noise environment, the pickup direction and frequency response curve of the microphone array can also be dynamically adjusted according to the preset algorithm to ensure that the recorded data always reaches the expected quality. After preliminary detection, the collected data is transmitted to the central processing unit through a special data interface, providing a unified data source for subsequent data preprocessing and feature extraction. Overall, the preferred multi-modal data acquisition scheme realizes high-precision, multi-dimensional recording of the information in the speech site through device configuration, environmental adaptation and real-time monitoring, lays a solid foundation for subsequent processing steps of the system, and significantly improves the overall performance and robustness of the final scoring system.

[0038] Preferably, the preprocessing step includes the following processing of the multi-modal data: denoising the video data, optimizing the image color by using a color correction algorithm, and removing irrelevant frames by a key frame extraction algorithm; noise suppression and frame division operation are performed on the audio data to improve the signal-to-noise ratio; the text data is subjected to syntax error correction, automatic punctuation completion, and is split into word units by word segmentation processing.

[0039] In the present application, the design of the preprocessing step aims to strictly improve the quality and standardize the collected multi-modal data, so as to ensure the consistency and high reliability of the data used in the subsequent feature extraction and fusion link. The preferred preprocessing step includes processing video, audio and text data respectively using special algorithms.

[0040] For video data, first, digital filtering technology is used for denoising, such as Gaussian filtering or mean filtering, to eliminate image noise caused by sensor noise or low-illumination environment; then, color correction algorithm is used to adjust the color balance of the image, so that the images collected under different lighting conditions have uniform color performance; then, key frame extraction algorithm is used to automatically select representative image frames, and remove repeated or non-information frames, thereby reducing data redundancy while retaining key visual information.

[0041] For audio data, spectral noise suppression technology is preferably used, such as spectral subtraction, to suppress noise in audio signals, and then short-time Fourier transform is used to frame process continuous audio signals, usually using Hamming window function to window each frame of data to reduce edge effects, and signal normalization technology is used to ensure that the amplitude of each frame of data is in a unified standard.

[0042] For text data, first, a method based on statistical rules and deep learning is used to correct the syntax of transcribed text, automatically correcting recognition errors; second, punctuation completion algorithm is used to restore missing punctuation symbols, making the text semantics more coherent; finally, high-efficiency segmentation algorithm is used to cut the text into word units, usually using a method based on maximum matching and hidden Markov model to improve the accuracy of word segmentation.

[0043] In addition, in order to ensure the consistency of each modal data in time, dynamic time warping algorithm is used to align the time sequence of video, audio and text data in the preprocessing process. Dynamic time warping algorithm can achieve the best matching of different modal data under nonlinear time transformation, ensuring that each time point of video frame, audio frame and corresponding text content strictly corresponds, thereby generating aligned multi-modal data.

[0044] To objectively evaluate the effect of each data processing link, the system also performs data quality detection on the preprocessing results, calculates the definition index for video data, calculates the signal-to-noise ratio for audio data, and evaluates the recognition confidence for text data. These indicators form the data quality index data to provide feedback for subsequent processing. In the preferred embodiment, all preprocessing parameters can be adaptively adjusted according to the actual situation on site, such as filter window size, frame length, and dynamic time warping matching tolerance, to ensure that the system can achieve the best preprocessing effect in various collection environments. Through the above strict preprocessing measures, the original collected data has been significantly improved in noise, color, timing, and semantics, providing a high-quality, standardized data foundation for subsequent feature extraction and deep fusion, effectively reducing the risk of subsequent model error and information loss, and significantly improving the stability and accuracy of the overall intelligent scoring system.

[0045] Preferably, the multi-modal data is deep fused through a cross-attention mechanism to generate unified fusion feature data containing cross-modal context associations for subsequent innovative feature generation and evaluation.

[0046] The multi-modal deep fusion is achieved by using a cross-attention mechanism. This mechanism uses a multi-modal Transformer model to jointly model the text, audio, and video features extracted after preprocessing, thereby generating unified and richly cross-modal context-associated fusion feature data. Specifically, first, the features of each modality are converted into fixed-dimensional vector representations by a special embedding layer. These vectors serve as inputs to form queries (Q), keys (K), and values (V), respectively. The cross-attention mechanism calculates using the following standard formula:

[0047]

[0048] where Q represents the query matrix, K represents the key matrix, V represents the value matrix, and dk represents the dimension of the key matrix. This formula is used in the present application to calculate the attention weights between different modalities by calculating the inner product of the vector representations of each modality feature and dividing by the normalization factor Then, the softmax function is used to obtain the normalized weight distribution, which reflects the correlation between the information of each modality. After stacking multiple cross-attention modules, the features of each modality are deep complementary while retaining their unique information, and the interaction parameters are continuously optimized through backpropagation. In the preferred embodiment, the number of layers of the multi-modal Transformer model is usually set to between six and twelve to ensure that the model can sufficiently capture cross-modal context information while avoiding overfitting problems caused by too deep networks.

[0049] In the specific operation process, the fusion module first receives the text features, audio features and video features from the preprocessing stage, which are converted into fixed-dimensional representation vectors by the corresponding embedding layers, and then calculates the interaction matrix between the modalities through the cross-attention module. The finally generated fusion feature data not only contains the basic information of each modality, but also fuses the cross-modal context relationship, thereby providing high-quality input for subsequent innovative feature reconstruction. The effect of the preferred embodiment is that through deep fusion, information complementation and context enhancement are realized, greatly improving the discriminability and robustness of feature representation. In addition, the attention weight in the fusion process can also be used to generate subsequent explanatory feedback, helping to clearly explain the contribution of each modality feature to the final score result. In actual application, the system uses a large amount of labeled data to optimize the multi-modal Transformer end-to-end in the training stage, so that the parameters of each cross-attention layer are constantly updated in backpropagation, thereby gradually improving the adaptability of the model to complex speech scenarios and the accuracy of the comprehensive score. Through this cross-attention mechanism, the fusion feature data can reflect the subtle emotional changes, speech rhythm and semantic coherence in the speech video, providing sufficient and comprehensive information support for the intelligent scoring module, while also providing visualized explanations based on attention distribution for users, meeting the demand for scoring transparency. Overall, the preferred deep fusion method not only has a sufficient mathematical basis in theory, but also has high operability and effect stability in actual systems, providing a solid data foundation and information support for subsequent innovative feature reconstruction, score calculation and final transparent feedback.

[0050] As shown in Figure 2 Innovative multi-modal feature reconstruction and joint fusion are implemented based on the fusion feature data, which includes: generating first innovative feature data from the fusion feature data using a self-supervised meta-learning algorithm; performing virtual perspective conversion on the video data using a generative adversarial network and extracting enhanced visual features to generate second innovative feature data; and integrating the first innovative feature data, second innovative feature data and conventionally extracted video, audio and text features, and realizing joint fusion using neural-symbol hybrid evaluation to generate innovative fusion feature data.

[0051] The present application implements innovative multi-modal feature reconstruction and joint fusion based on the fusion feature data, and the core goal is to further reconstruct innovative features for the multi-modal data after deep fusion, forming a comprehensive information carrier that can fully capture the deep semantic, emotional and expression features of the speech content.

[0052] Firstly, the self-supervised meta-learning algorithm is used for re-learning of the fusion feature data, and the cross-modal rapid adaptation capability is obtained through meta-training, so that the first innovative feature data is generated; subsequently, the generative adversarial network is used for virtual view conversion for the video data, a plurality of virtual videos with different views, illuminations, backgrounds and character angles are generated, and enhanced visual features are extracted in each virtual video to generate second innovative feature data; finally, the first innovative feature data and the second innovative feature data are integrated with the conventional extracted text, audio and video features, and joint fusion is realized by using neural-symbol hybrid evaluation to generate the final innovative fusion feature data.

[0053] The innovative fusion feature data not only retains the basic information reflected in the traditional features, but also forms a deep abstract expression through reconstruction and joint fusion, which can more accurately describe the semantics, emotions and non-verbal expressions of the speaker, and further provide high-quality and comprehensive information input for subsequent dynamic adaptive multi-task scoring. The present application realizes the transition from low-dimensional conventional features to high-dimensional abstract features through multi-level feature reconstruction and fusion, provides a solid data foundation for intelligent scoring, and uses advanced learning and generative adversarial technology to make up for the shortcomings of traditional methods in cross-modal information integration, thereby improving the accuracy, robustness and interpretability of the scoring system.

[0054] The present application adopts strict time sequence alignment and quality detection means in each link of data preprocessing, feature extraction, deep fusion and innovative feature reconstruction, ensures the consistency and high quality requirements of each modal data before entering the reconstruction module, and realizes end-to-end optimization and closed-loop feedback in the whole scoring system. By comprehensively using self-supervised meta-learning, generative adversarial network and neural-symbol hybrid evaluation, the system can realize adaptive adjustment and real-time optimization when facing different speech scenes and data quality changes, so that the generated innovative fusion feature data has higher discriminant ability and generalization performance in semantic expression, emotion recognition and behavior evaluation, and provides a more solid and transparent basis for subsequent scoring algorithms. In summary, the present application realizes intelligent, automatic and transparent design from data acquisition to scoring closed loop through innovative feature reconstruction and joint fusion, not only greatly enriches the expression information of multi-modal data, but also effectively solves the problems of isolated modal information, feature redundancy and information loss in traditional methods, and provides a new, efficient and sustainable optimization technical scheme for speech video intelligent scoring.

[0055] Preferably, the self-supervised meta-learning algorithm adopts a training strategy based on model-independent meta-learning, and performs meta-training on the speech data to learn the ability to quickly adapt between different tasks, so that the first innovative feature data can extract deep semantic and emotional information of multi-modal data and has cross-modal migration capability.

[0056] The self-supervised meta-learning algorithm adopts a model-agnostic meta-learning (MAML) based training strategy, and performs meta-training on a large amount of speech data, so that the model can quickly adjust the parameters when facing different tasks, and then extract deep semantic and emotional information from the fused feature data to generate the first innovative feature data.

[0057] In the specific operation process, first, the fused feature data after preprocessing and deep fusion by the multi-modal Transformer is input into the self-supervised meta-learning module. The module adopts an iterative optimization learning strategy, and the goal is to make the model learn the general feature reconstruction rule through meta-training, so as to realize the effective transfer of cross-modal information in the limited labeled or unlabeled data environment. This process involves constructing multiple training tasks, each corresponding to a different scene or context in the speech data. Through rapid adaptation and parameter updating on these tasks, the model obtains cross-task migration capability. Mathematically, the basic steps of parameter updating in meta-training can be described by the following formula:

[0058]

[0059] wherein θ represents the initial model parameters, α is the learning rate, represents the loss function for task T i , and the updated parameter θ ′ is the rapid adaptation parameter on the task. After multi-task training, the initial parameters are further updated by backpropagation of the mean of each task loss to make the model have higher generalization ability. This self-supervised meta-learning strategy makes the generated first innovative feature data not only capture the basic information in the fused feature data, but also extract the deep abstract semantic and emotional change information hidden in the multi-modal data, thereby realizing significant information gain in cross-modal feature reconstruction. In the specific embodiment, in the speech data set, the model can quickly adapt to the speech video containing complex features such as speech speed, tone and semantic coherence, and generate highly discriminative feature representations after a certain number of meta-training rounds.

[0060] The core of the strategy is to make the model automatically optimize through internal supervision signals without a large amount of manual annotation through self-supervised learning, and to obtain cross-task rapid adaptation ability through meta-training, thereby greatly improving the stability and migration ability of generating the first innovative feature data. Through the application, the system can maintain high robustness in different speech scenes, and even in the case of data scarcity or high noise, stable feature representation can be extracted to provide high-quality input for subsequent joint fusion. The preferred embodiment further supports online updating and incremental learning, which can continuously update the parameters of the meta-model according to newly collected data to adapt to changing speech styles and data distribution, thereby realizing long-term stable operation and performance optimization of the intelligent scoring system.

[0061] Preferably, the generative adversarial network is used for virtual view conversion of the video data, and in the conversion process, the generator generates a plurality of videos of different views according to the original video content while maintaining the diversity of light, background and character angle in the scene, extracts enhanced visual features therein to generate second innovative feature data.

[0062] The generative adversarial network is used for virtual view conversion of the video data, and the core of the generative adversarial network is to generate a plurality of videos of different views according to the original video content through the generator network, and maintain the diversity of light, background and character angle in the scene in the process, so as to extract enhanced visual features to generate second innovative feature data. In a specific implementation, the generative adversarial network is composed of a generator and a discriminator, the generator takes the original video data as input, generates video output of different views through a series of convolution and deconvolution layers, and the discriminator is used to determine the similarity between the generated video and the real video, thereby continuously optimizing the output quality of the generator. The process follows the basic principle of adversarial training, and the loss function can be represented as:

[0063]

[0064] Where G represents the generator, D represents the discriminator, x represents the real video data, and z represents the random input vector of the generator. In the application, the generator encodes the input video data and generates a plurality of videos with different views using the decoding layer, and the process requires the generator to generate virtual videos by changing the parameters such as view, light and background while keeping the basic semantics of the original video content unchanged. In a specific implementation, each layer in the generator network adopts residual connection to ensure the continuity of information transmission, and batch normalization technology is used to stabilize the training process. Through repeated iterative training, the generator can generate virtual view videos with high quality and strong diversity.

[0065] Subsequently, a pre-trained three-dimensional convolutional neural network (such as an I3D network) is used to extract spatiotemporal visual features from each generated virtual perspective video. These features reflect the speaker's body movements, facial expressions and scene information under different perspectives. The extracted visual features are fused with the visual features of the original video after appropriate dimensionality reduction processing. The generated second innovative feature data not only contains direct features in the original data when describing the visual information, but also integrates enhanced features under the virtual perspective, thereby improving the robustness and discriminability of the visual features across scenes and lighting conditions. In practical applications, virtual perspective conversion can enable the system to maintain stable visual performance when facing complex scene changes, while providing richer visual information for subsequent neural symbolic hybrid evaluation. This preferred embodiment not only improves the limitations of traditional single-perspective data under changes in lighting and angles, but also realizes the dynamic expansion and enhancement of visual information through the adaptive learning of the generative adversarial network, thereby greatly improving the overall performance and scoring accuracy of the subsequent joint fusion module.

[0066] Preferably, Figure 3 As shown, the neural-symbolic hybrid evaluation module includes two parts: the neural network part is used to perform preliminary evaluation on the input first innovative feature data, the second innovative feature data and the traditionally extracted text, audio and video features based on the deep learning model; the symbolic logic reasoning part performs reasoning and correction based on the grammatical structure, emotional expression and body language rules of the speech content.

[0067] The neural-symbol hybrid evaluation module integrates the deep learning capabilities of neural networks with the rule-based reasoning capabilities of symbolic logic to comprehensively evaluate the input first innovation feature data, second innovation feature data, and conventionally extracted text, audio, and video features, achieving joint fusion. In specific implementation, the neural network part uses a combination of standard deep convolutional neural networks and recurrent neural networks to make preliminary predictions and scores on the input features; while the symbolic logic reasoning part has pre-defined rule sets, including key indicators such as the grammatical structure of the speech content, emotional expression, and body language. For example, the symbolic logic reasoning module has the rule "if the speech text has a grammatical error rate exceeding a certain threshold, reduce the grammar score; if long pauses are detected in the audio, modify the emotional score", which is calculated by the logic reasoning engine to correct the preliminary score of the neural network part. The overall module, in the input stage, splices the first innovation feature data and the second innovation feature data with the conventional features to form a unified feature vector, which is then output by the neural network part. Thereafter, the symbolic logic reasoning module modifies the score result according to the rules, and the final output of the comprehensive evaluation result is the innovation score data, along with detailed explanation information forming the innovation explanation data. This process can be mathematically described as follows: let the neural network prediction output be S NN , the symbolic reasoning correction factor be ΔS symbol , then the final score S can be expressed as:

[0068] S=S NN +ΔS symbol

[0069] where S NN reflects the preliminary evaluation result of deep learning on multi-modal features, and ΔS symbol adjusts S NN according to symbolic logic rules. In actual application, through the neural-symbol hybrid evaluation module, the advantages of neural networks in complex pattern recognition can be fully utilized, while the symbolic logic's explainability is used to correct and explain the score results, ensuring high accuracy of the evaluation results and providing transparent scoring basis. In the preferred embodiment, the module supports online adjustment, allowing the system to continuously update the symbolic rules and neural network weights according to new speech data to adapt to changing speech styles and data distribution, achieving continuous model optimization and system performance improvement. Overall, the neural-symbol hybrid evaluation module effectively compensates for the lack of pure neural networks in interpretability, ensuring that the intelligent scoring system has both high precision and clear scoring basis, making the scoring results easier to understand and trace, providing reliable and transparent intelligent scoring services for users.

[0070] Based on the innovative fusion feature data, dynamic adaptive multi-task scoring and transparent explanation feedback are implemented, which includes generating dynamic weights according to data quality indicators, scoring the features of video, audio and text respectively using multi-task scoring branches, and generating comprehensive scoring data by weighted aggregation according to the dynamic weights; using an explanatory algorithm to analyze the contribution of each scoring dimension to generate scoring basis information; and digitally signing and recording the comprehensive scoring data and scoring basis information on the chain.

[0071] Based on the innovative fusion feature data, the speech video intelligent scoring method implements dynamic adaptive multi-task scoring and transparent explanation feedback, aiming to comprehensively evaluate the multi-dimensional information in the speech video, including text, audio and video features, to achieve efficient and accurate scoring.

[0072] The process first generates dynamic weights using data quality indicators, adjusts the weights according to the quality of each modal feature, and updates the weights adaptively through feedback residuals; then, the system scores the features of video, audio and text through multi-task scoring branches, and generates comprehensive scoring data by weighted aggregation according to the dynamic weights of each modal. Finally, the scoring results are analyzed by an explanatory algorithm, and detailed scoring basis information is generated. To ensure the security and transparency of the scoring data, the comprehensive scoring data and scoring basis information will be encrypted by digital signature and uploaded to the blockchain platform for storage.

[0073] This process not only improves the accuracy and reliability of the scoring, but also provides transparency and traceability of the scoring, fully meeting the requirements of modern intelligent scoring systems for accuracy, robustness, explainability and data security. Overall, by introducing dynamic weight adjustment, multi-task scoring, multi-dimensional explanation and blockchain storage, etc., the invention provides a comprehensive and efficient speech video scoring method that can quickly adapt to different speech scenarios and conditions and produce stable and accurate scoring results.

[0074] Preferably, the dynamic weights are generated by a lightweight gating network according to the data quality indicators, and the initial dynamic weight data is adjusted according to the video quality, audio signal-to-noise ratio and text transcription confidence, and is iteratively updated in the scoring process through feedback residuals to generate the final dynamic weight data that meets the performance of each modal feature.

[0075] The dynamic weight generation method is realized by a lightweight gating network, which automatically generates initial dynamic weights using the data quality indicators, and iteratively updates the weight data in the scoring process through feedback residuals to ensure that the finally generated dynamic weights can adapt to the performance of each modal feature.

[0076] Specifically, the gating network generates initial dynamic weight data based on the acquired data quality indicators, including video quality, audio signal-to-noise ratio, and text transcription confidence. In this process, first, the video quality indicator is calculated to evaluate the clarity and stability of the video; second, the audio signal-to-noise ratio is used to measure the quality of the audio signal; finally, the text transcription confidence is used to measure the accuracy of speech recognition. According to these quality indicators, the network adjusts the initial weight value of each modality, ensuring that the modality with poor data quality has a low proportion in the scoring process. Then, during the scoring process, when the scoring data is iterated through the feedback loop, the residual error, i.e., the scoring error, is used to adjust the weights, optimizing the response and scoring accuracy of the system.

[0077] This weight adjustment process can be represented as:

[0078]

[0079] where w (t) is the current weight, η is the learning rate, represents the current scoring error, is the gradient of the loss function with respect to the weight, w (t+1) is the updated weight. This method iteratively adjusts the dynamic weights through feedback residuals, ensuring the adaptability and accuracy of the system in the face of different data quality situations. Finally, after multiple rounds of iterative optimization, the system will generate final dynamic weight data that conforms to the performance of each modality, maximizing the accuracy and robustness of the scoring results. In practical applications, this process realizes adaptive adjustment through an efficient gating network, not only optimizing the effect of multi-modal scoring, but also improving the stability and real-time performance of the scoring process.

[0080] Preferably, the multi-task scoring branch includes a text scoring branch, an audio scoring branch, and a video scoring branch. The text scoring branch evaluates the grammatical structure, vocabulary richness, and grammatical errors of the speech through a fully connected neural network. The audio scoring branch evaluates the emotional inclination, speech fluency, and pause length of the speech through a long short-term memory network or a Transformer model. The video scoring branch evaluates the speaker's body language and speech style through a three-dimensional convolutional neural network or a Transformer model, and generates corresponding scoring data for each dimension.

[0081] The multi-task scoring branch method divides the speech video scoring task into three independent but related sub-tasks: text scoring, audio scoring, and video scoring. Each sub-task is evaluated by a dedicated scoring branch and generates corresponding scoring data.

[0082] Specifically, the text scoring branch evaluates the syntax structure, lexical richness, and grammatical errors of the speech through a fully connected neural network. The network first encodes the input text features, extracting lexical embeddings and syntactic features, and then performs classification and regression through a multi-layer perceptron (MLP) to evaluate the quality of the text. For the audio scoring branch, this branch utilizes a long short-term memory network (LSTM) or a Transformer model to evaluate the sentiment orientation, speech fluency, and pause duration in the speech. The LSTM network models the time series of audio signals to identify fluctuations in emotional expression, while generating sentiment score data based on pause duration and speech fluency information in the audio. For the video scoring branch, this branch employs a three-dimensional convolutional neural network (3D CNN) or a Transformer model to evaluate the speaker's body language and speaking style. The 3D CNN models the spatiotemporal features of video frames to capture the speaker's body movements and facial expressions, and further evaluate their speaking style.

[0083] These scoring branches score different modal features respectively, but since the speech video is multi-modal, the output of each scoring branch will depend on the scoring results of other branches. During the calculation process, the output of each branch is weighted and aggregated with dynamic weights to reflect the relative importance of each modal feature in scoring. Finally, by combining the output from each task scoring branch, the comprehensive score data is calculated through a weighted aggregation algorithm. This multi-task scoring method not only independently evaluates the features of each modality, but also realizes the fusion of different modal information through weighted aggregation, fully utilizing the advantages of each modality and improving the comprehensiveness and accuracy of the scoring system. In practical applications, the system dynamically adjusts the scoring weights according to different speech video features to ensure that the scoring results are highly consistent with the actual speaking style and content.

[0084] Preferably, the comprehensive score data and scoring basis information are encrypted by digital signature technology and stored by the blockchain platform. The storage step includes hashing and digitally signing the comprehensive score data, scoring basis information, and other related data, and then uploading the calculation results to the blockchain.

[0085] Digital signature and blockchain storage technology plays an important role in ensuring the non-tamperability and process transparency of the scoring data in the present application. The comprehensive score data and scoring basis information are encrypted by digital signature technology and stored by the blockchain platform, ensuring that the scoring results can be traced back at any time node and cannot be tampered with.

[0086] In implementation, first, the comprehensive score data and the score basis information corresponding thereto, including the output of each score branch, dynamic weight data and score error, will be subjected to hash calculation to generate a unique digital fingerprint, and the digital fingerprint will be encrypted through digital signature technology to generate an encrypted signature. The signature is generated by a private key, ensuring that only authorized parties can generate and verify its authenticity. The generated signature and hash value will be uploaded to the blockchain platform to ensure the transparency and tamper resistance of the score data and score basis.

[0087] This evidence preservation process not only improves the security of the scoring system, but also increases the traceability of the system, ensuring that all scoring decisions can be traced back to the corresponding data sources and calculation processes. The tamper resistance of the blockchain ensures that the scoring data cannot be modified or deleted during the entire life cycle, avoiding the risk of manual intervention and data tampering. The evidence preservation step can be automatically executed through a smart contract, further reducing errors and biases in manual operations. Through this solution, all scoring processes will have legal validity and verifiability, especially when a review or dispute resolution is required, providing strong technical support for the system. In addition, the introduction of this technology not only improves the credibility of the scoring system, but also further enhances users' trust in the scoring results, ensuring the transparency and fairness of the entire intelligent scoring process.

[0088] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects.

[0089] The above is only an embodiment of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. A method for intelligent scoring of speech videos based on multidimensional feature extraction, characterized in that: The present invention comprises the following steps: Acquiring multimodal data of a speech scene, wherein the multimodal data includes video data, audio data, and text data; preprocessing the multimodal data, performing conventional feature extraction on the video, audio, and text, respectively, and deeply fusing the features of each modality to generate fused feature data; Implementing innovative multimodal feature reconstruction and joint fusion based on the fused feature data, the steps comprising: generating first innovative feature data from the fused feature data using a self-supervised meta-learning algorithm; performing virtual perspective conversion on the video data using a generative adversarial network and extracting enhanced visual features to generate second innovative feature data; and integrating the first innovative feature data and the second innovative feature data with conventionally extracted video, audio, and text features, and implementing joint fusion using neural-symbolic hybrid evaluation to generate innovative fused feature data; Based on the innovative fusion feature data, dynamic adaptive multi-task scoring and transparent explanatory feedback are implemented. This step includes: generating dynamic weights according to data quality indicators, and using multi-task scoring branches to score the features of video, audio and text respectively, and generating comprehensive scoring data based on weighted aggregation of the dynamic weights; using an explanatory algorithm to analyze the contribution of each scoring dimension to generate scoring basis information; and digitally signing the comprehensive scoring data and scoring basis information and recording them on the chain.

2. The method according to claim 1, characterized in that The multimodal data includes video data, audio data and text data of the speech. The video data is collected by a high-resolution camera and a depth sensor, the audio data is collected by multiple high-fidelity microphone arrays, and the text data is generated by real-time transcription of the audio data by a speech recognition system based on the Transformer model.

3. The method according to claim 1, characterized in that The preprocessing step includes performing the following processing on the multimodal data: denoising the video data, optimizing the image color using a color correction algorithm, and removing irrelevant frames using a key frame extraction algorithm; Noise suppression and framing operations are performed on audio data to improve the signal-to-noise ratio; grammatical correction and punctuation automatic completion are performed on text data, and the text is split into vocabulary units through word segmentation.

4. The method according to claim 1, wherein The multimodal data deeply fuses text features, audio features and video features through a cross-attention mechanism to generate unified fused feature data. The fused feature data contains the contextual association of cross-modal information for subsequent innovative feature generation and evaluation.

5. The method according to claim 1, wherein The self-supervised meta-learning algorithm adopts a training strategy based on model-independent meta-learning and performs meta-training on speech data to learn the ability to quickly adapt between different tasks. The generated first innovative feature data can extract deep semantic and emotional information from multimodal data and has cross-modal migration capabilities.

6. The method according to claim 1, characterized in that The generative adversarial network is used to perform virtual perspective conversion on the video data. During the conversion process, the generator generates multiple videos with different perspectives based on the original video content, while maintaining the diversity of lighting, background and character angles in the scene, and extracts enhanced visual features therefrom to generate second innovative feature data.

7. The method according to claim 1, characterized in that The neural-symbolic hybrid evaluation module includes two parts: the neural network part is used to perform preliminary evaluation of the input first innovative feature data, second innovative feature data and traditionally extracted text, audio and video features based on a deep learning model; the symbolic logic reasoning part performs reasoning and correction based on the grammatical structure, emotional expression and body language rules of the speech content.

8. The method according to claim 1, characterized in that The dynamic weight generates initial dynamic weights according to the data quality indicators through a lightweight gating network. The initial dynamic weight data is adjusted according to the video quality, audio signal-to-noise ratio and text transcription confidence, and the weight data is iteratively updated through feedback residuals during the scoring process, and finally the final dynamic weight data that conforms to the characteristic performance of each modality is generated.

9. The method according to claim 1, characterized in that The multi-task scoring branch includes a text scoring branch, an audio scoring branch and a video scoring branch. The text scoring branch evaluates the grammatical structure, vocabulary richness and grammatical errors of the speech through a fully connected neural network. The audio scoring branch evaluates the emotional tendency, speech fluency and pause duration of the speech through a long short-term memory network or a Transformer model. The video scoring branch evaluates the speaker's body language and speech style through a three-dimensional convolutional neural network or a Transformer model, and generates corresponding scoring data for each dimension.

10. The method according to claim 1, characterized in that The comprehensive scoring data and scoring basis information are encrypted through digital signature technology and stored on the blockchain platform. The storage step includes hashing and digitally signing the comprehensive scoring data, scoring basis information and other relevant data, and then uploading the calculation results to the blockchain.

Images