Speech video analysis method, computer device and storage medium

Abstract

The application relates to a speech video analysis method, a computer device and a storage medium. The method comprises the following steps: based on a preset multi-modal model, a to-be-analyzed speech video is split and extracted to obtain a plurality of information units of each mode at different scales; based on each information unit, an evidence item is respectively generated; based on each evidence item, an evaluation result under each preset evaluation dimension is respectively generated; and based on each evaluation result, a guidance suggestion of the to-be-analyzed speech video is generated. The method can improve the accuracy of model-generated conclusions.

Description

Technical Field

[0001] This application relates to the field of large language model technology, and in particular to a speech video analysis method, computer device, and storage medium. Background Technology

[0002] With the development of large language model technology, speech video analysis technology has emerged. This technology extracts multimodal information from videos, analyzes this information, and finally outputs conclusions such as a summary of the speech video or suggestions for improvement.

[0003] In traditional techniques, speech videos are typically preprocessed and transcribed, then multimodal information such as text, audio, and video is extracted. This information is then input into a multimodal analysis model or a large language model, which outputs a summary, score, modification suggestions, or natural language report.

[0004] However, traditional methods typically input the full multimodal analysis results directly when generating conclusions. The generated conclusions lack a reference relationship with the multimodal information, making it difficult to form a traceable chain of evidence, resulting in poor accuracy of the model's generated conclusions. Summary of the Invention

[0005] Therefore, it is necessary to provide a speech video analysis method, computer equipment, and storage medium that can improve the accuracy of model-generated conclusions in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a method for analyzing speech videos, the method comprising:

[0007] Based on a pre-defined multimodal model, the video of the speech to be analyzed is split and extracted to obtain multiple information units of each modality at different scales;

[0008] Evidence items are generated based on each of the aforementioned information units;

[0009] Based on each of the aforementioned evidence items, an evaluation result is generated for each preset evaluation dimension.

[0010] Based on the evaluation results, guidance suggestions are generated for the speech video to be analyzed.

[0011] In one embodiment, the preset multimodal model includes a segmentation and extraction module corresponding to each modality; the step of segmenting and extracting the speech video to be analyzed based on the preset multimodal model to obtain multiple information units of each modality at different scales includes:

[0012] Based on the preset module calling order, each splitting and extraction module in the preset multimodal model is called sequentially to split and extract the speech video to be analyzed once or multiple times, thereby obtaining multiple information units corresponding to the splitting and extraction module.

[0013] When the splitting and extraction result of the splitting and extraction module does not meet the preset conditions, the splitting and extraction module is processed based on the processing strategy of the intermediate state of the splitting and extraction module in the preset state transition rules.

[0014] In one embodiment, generating evidence items based on each of the information units includes:

[0015] Based on the information unit, evidence items in the unified evidence index are generated; the evidence item includes one or more of the following: unique evidence identifier, evidence type, source product, time range, text content, file path, and availability status.

[0016] In one embodiment, generating evaluation results for each preset evaluation dimension based on each of the evidence items includes:

[0017] Based on a preset evaluation framework document, at least one preset evaluation dimension is determined; the preset evaluation framework document includes evidence screening rules corresponding to each preset evaluation dimension.

[0018] Based on the evidence screening rules and each of the evidence items, an evidence package is generated for each of the preset evaluation dimensions;

[0019] Based on the preset evaluation framework document, the evaluation result of each evidence package is determined.

[0020] In one embodiment, the evidence screening rules include evidence source rules, auxiliary evidence restrictions, and prohibited evidence rules. The step of generating an evidence package for each of the preset evaluation dimensions based on the evidence screening rules and each of the evidence items includes:

[0021] Based on the evidence source rules and the prohibited evidence rules, each of the evidence items is filtered to obtain a first set of evidence items;

[0022] Based on the aforementioned auxiliary evidence constraints, each of the evidence items in the first set of evidence items is classified into primary and secondary categories to obtain a second set of evidence items, which serves as the evidence package under the preset evaluation dimension.

[0023] In one embodiment, the speech video analysis method further includes:

[0024] Availability verification is performed on each of the information units to obtain a first verification result; the availability verification includes one or more of the following: output format verification, field integrity verification, abnormal status marking, failure record, and insufficient information verification.

[0025] The evaluation results are verified to obtain a second verification result; the report verification includes one or more of the following: evidence boundary verification, output format verification, field integrity verification, label legality verification, auxiliary evidence limitation verification, conclusion without evidence verification, visual over-inference verification, and information insufficiency verification.

[0026] If the first verification result is abnormal, the evaluation result associated with the information unit corresponding to the first verification result will be constrained to a non-deterministic evaluation result.

[0027] If the second verification result is abnormal, the evaluation result corresponding to the second verification result will be constrained to a non-deterministic evaluation result.

[0028] In one embodiment, the guidance suggestions generated based on the evaluation results for the presentation video to be analyzed include:

[0029] Based on the preset typology framework document and the evaluation results, the typology profile of the speech video to be analyzed is determined.

[0030] Based on the aforementioned type profile, guidance suggestions are generated for the speech video to be analyzed.

[0031] In one embodiment, determining the type profile of the speech video to be analyzed based on a preset typological framework document and the evaluation results includes:

[0032] Based on a pre-defined typological framework document, the evaluation dimensions corresponding to different typological dimensions are determined.

[0033] Based on the evaluation results associated with the evaluation dimensions corresponding to each of the type dimensions, the type labels of the speech video to be analyzed under different type dimensions are determined;

[0034] Based on the type labels under different type dimensions, the type profile of the speech video to be analyzed is determined.

[0035] Secondly, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in any of the embodiments described above.

[0036] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the embodiments described above.

[0037] The aforementioned speech video analysis method, computer equipment, and storage medium, based on a preset multimodal model, decompose and extract information from the speech video to be analyzed, obtaining multiple information units for each modality at different scales; based on each information unit, evidence items are generated; based on each evidence item, evaluation results are generated for each preset evaluation dimension; based on each evaluation result, guidance suggestions for the speech video to be analyzed are generated. Compared to traditional techniques that directly generate corresponding guidance suggestions from the speech video, this method decomposes and extracts multimodal information from the speech video, uniformly encodes the obtained information into multiple evidence items, and performs corresponding evaluations and generates guidance suggestions based on these multiple evidence items. This allows each evaluation result and guidance suggestion to be specifically indexed to the corresponding evidence item, enabling the final guidance suggestions to combine the evaluation results corresponding to multiple evidence items to generate more accurate guidance suggestions, thereby achieving the technical effect of improving the accuracy of the model's generated conclusions. Attached Figure Description

[0038] Figure 1 This is a diagram illustrating the application environment of a presentation video analysis method in one embodiment.

[0039] Figure 2 This is a flowchart illustrating a presentation video analysis method in one embodiment;

[0040] Figure 3 Here is a system flowchart of a presentation video analysis method in one embodiment;

[0041] Figure 4 This is a diagram of a preset evaluation framework document and a preset typology framework document in one embodiment;

[0042] Figure 5 This is a schematic diagram of the interface of the speech video analysis method in one embodiment;

[0043] Figure 6 This is a schematic diagram of the interface displaying guidance and suggestions in one embodiment;

[0044] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] The presentation video analysis method provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. The data storage system can store information units, evidence items, evaluation results, and guidance suggestions required by server 104. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Server 104 can be used to split and extract the presentation video to be analyzed, generate evidence items, evaluation results, and guidance suggestions, and send these to the data storage system and terminal 102 for visualization on terminal 102. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0047] This application's embodiments introduce an Agent-Harness collaborative mechanism, an evidence boundary control mechanism, and a structured typological analysis method. The Agent can be an intelligent execution entity capable of planning tasks around predetermined goals, invoking models and tools, and dynamically adjusting the process based on execution status. The Harness can be a constraint and operational control framework built around the models and tools. The Agent can act as the global process orchestration entity for the speech video analysis task, scheduling models and program tools, and implementing continue, retry, degradation, or termination controls based on the execution status of each stage. The Harness performs format validation, field integrity checks, status marking, and exception handling on the model and tool outputs to prevent incomplete or invalid results from blocking the program or directly entering subsequent processes.

[0048] In one embodiment, such as Figure 2 As shown, a method for analyzing speech videos is provided, including the following steps:

[0049] Step S110: Based on the preset multimodal model, the video of the speech to be analyzed is split and extracted to obtain multiple information units of each modality at different scales.

[0050] The preset multimodal model can be an artificial intelligence model capable of simultaneously receiving, processing, understanding, and generating two or more different modal information and establishing intermodal mapping relationships. The modalities processed by the multimodal model can include, but are not limited to, one or more forms such as text, images, audio, video, and 3D point clouds. In some embodiments, the preset multimodal model can be an encoder that transforms different raw modal data such as text, images, and speech into feature vectors of a unified dimension and maps them to a shared representation space; relying on alignment, fusion, and generation mechanisms, it can mine the inherent relationships between different modal information to achieve tasks such as cross-modal retrieval, understanding, content generation, and question answering. In other embodiments, the preset multimodal model can also be a combination of one or more processing modules for different modalities. In this embodiment, the preset multimodal model can be used to perform processes such as duration statistics, speech-to-text conversion, extraction of key paragraphs or sentences, and extraction of action and facial expression information from the video to be analyzed.

[0051] The process of splitting and extracting the speech video to be analyzed can be carried out by splitting and extracting the multimodal information in the speech video according to a preset modality. Taking text as an example, the splitting and extraction of the multimodal information in the speech video to be analyzed can include performing speech recognition on the speech video to obtain speech-transcribed text, and performing OCR recognition on the subtitles in each image frame of the speech video to obtain image-recognized text.

[0052] For example, the different scales can be the division methods of each modality at different levels of fineness. Similarly, taking the extraction of text from the speech video to be analyzed as an example, the different scales can include, but are not limited to, text, paragraphs, sentences, words, characters, etc. Taking the preset modality as video as an example, different scales can correspond to different frame numbers, such as different video segments in units of frames, milliseconds, and seconds. Taking the preset modality as image as an example, the different scales can include the complete image and multiple corresponding image blocks of different sizes, and can also include resolution scaling processing of the image itself to obtain image data of the complete image at different resolutions.

[0053] An information unit can be a storage unit for the extracted results. For example, depending on the preset modality, an information unit can be used to store information including but not limited to video clips, image data, audio clips, text, and 3D point cloud data. Furthermore, depending on the scale, taking text as the preset modality as an example, each information unit can be used to store the aforementioned text, paragraphs, sentences, words, characters, and other information.

[0054] Step S120: Generate evidence items based on each of the information units.

[0055] Understandably, when obtaining the information unit, one or more of the following can also be obtained: the type of the information unit, the timestamp and / or time range in which the information unit appears in the video to be analyzed, the text content of the information unit, and the storage path of the information unit.

[0056] The evidence item can be the smallest referable object in the unified evidence index, used to record one or more of the following: unique evidence identifier, evidence type, time range, text content, and file path; the unified evidence index can be a structured index file used for evidence retrieval.

[0057] For example, generating evidence items based on each of the information units can be achieved by assigning a unique identifier to each information unit, combining one or more of the unique identifier, the type of the information unit, the time range text content, etc., into an evidence item, and then incorporating it into a unified evidence index.

[0058] Step S130: Based on each of the evidence items, generate evaluation results for each preset evaluation dimension.

[0059] The preset evaluation dimensions can be dimensions used to analyze the speech video. For example, the preset evaluation dimensions can be set according to actual needs. In some exemplary embodiments, the preset evaluation dimensions may include acoustic parameters, body language, speech strategies, and appeals.

[0060] Taking the evaluation of acoustic parameters as an example, the process of generating evaluation results for each preset evaluation dimension based on each piece of evidence can be as follows: First, based on the evaluation dimension of acoustic parameters, select the evidence items required for the evaluation dimension of acoustic parameters, such as acoustic features, noise quality, etc., and then evaluate according to the selected evidence items to obtain the evaluation results.

[0061] An evaluation framework can be a document that includes rules for interpreting evidence, a set of labels for each evaluation dimension, and evaluation logic, used to categorize evaluation results by label.

[0062] The evaluation results may include one or more of the following: a unique evaluation identifier, an evaluation conclusion, and the evidence used.

[0063] Step S140: Based on the evaluation results, generate guidance suggestions for the speech video to be analyzed.

[0064] The step of generating guidance suggestions for the speech video to be analyzed based on the evaluation results can be achieved by inputting the evaluation results into a guidance suggestion generation model to obtain guidance suggestions for the speech video to be analyzed.

[0065] This embodiment provides a speech video analysis method that, compared to traditional techniques that directly generate corresponding guidance suggestions from speech videos, extracts and decomposes multimodal information from the speech video and encodes the obtained information into multiple evidence items. Based on these evidence items, corresponding evaluations and guidance suggestions are generated. This allows each evaluation result and guidance suggestion to be specifically indexed to the corresponding evidence item, enabling the final guidance suggestion to combine the evaluation results corresponding to multiple evidence items to generate more accurate guidance suggestions, thereby improving the technical effect of improving the accuracy of the model's generated conclusions.

[0066] In one embodiment, the preset multimodal model includes a segmentation and extraction module corresponding to each modality; the step of segmenting and extracting the speech video to be analyzed based on the preset multimodal model to obtain multiple information units of each modality at different scales includes:

[0067] Based on the preset module calling order, each splitting and extraction module in the preset multimodal model is called sequentially to split and extract the speech video to be analyzed once or multiple times, thereby obtaining multiple information units corresponding to the splitting and extraction module.

[0068] When the splitting and extraction result of the splitting and extraction module does not meet the preset conditions, the splitting and extraction module is processed based on the processing strategy of the intermediate state of the splitting and extraction module in the preset state transition rules.

[0069] In this embodiment, the splitting and extraction of the speech video to be analyzed can be performed by using multiple splitting and extraction modules corresponding to different modalities.

[0070] Understandably, when there are multiple splitting and extraction modules, there may be interdependencies between different splitting and extraction modules. For example, the speech-to-text module and the acoustic analysis module depend on the speech extraction module. Therefore, in this embodiment, a preset module calling order is provided to determine the calling order of each splitting and extraction module.

[0071] Understandably, when the information unit includes multiple different scales, the speech video to be analyzed can be split and extracted once or multiple times. For example, in the first split and extraction, the text of the speech video to be analyzed can be obtained; in the second split and extraction, the paragraphs of the text can be obtained; in the third split and extraction, the sentences of the paragraphs can be obtained, and so on, until multiple information units of each modality at different scales are finally obtained.

[0072] In this embodiment, the preset conditions can be basic conditions used to determine whether an information unit is usable. For example, when the preset conditions are not met, it may correspond to situations such as extraction failure, non-existent target information, incomplete extraction of target information, or module abnormality.

[0073] The preset state transition rule can be a rule used to process the module that has an abnormality based on the corresponding abnormal state when an abnormality occurs in the splitting and extraction module; for example, the processing can include continuing, retrying, downgrading, terminating, etc.

[0074] The intermediate state can be the state that the splitting and extraction module changes to when it encounters one of the above-mentioned extraction failure, target information not found, target information incomplete extraction, or module abnormality. In an exemplary embodiment, the intermediate state of the splitting and extraction module can be one of the following states: extraction failure, target information not found, target information incomplete, or module abnormality. Taking the intermediate states of the splitting and extraction module as extraction failure, target information not found, target information incomplete, or module abnormality as an example, the processing strategy can be to retry when the splitting and extraction module is in the extraction failure state, skip when the target information does not found state, downgrade when the target information is incomplete state, and terminate when the module is in the abnormal state, etc., to process the corresponding splitting module.

[0075] This embodiment provides a speech video analysis method that sets the running order for each splitting and extraction module, allowing the model to run in a preset order, reducing the risk of the model running arbitrarily. Furthermore, it sets state transition rules during model operation, ensuring that the entire process is not blocked when a single model encounters an anomaly, thereby achieving the technical effect of improving system operating efficiency.

[0076] In one embodiment, generating evidence items based on each of the information units includes:

[0077] Based on the information unit, evidence items in the unified evidence index are generated; the evidence item includes one or more of the following: unique evidence identifier, evidence type, source product, time range, text content, file path, and availability status.

[0078] The unique identifier of the evidence can be a key used to retrieve the evidence item in a unified evidence index; the evidence type can be text, audio, image, keyframe, etc.; the source product can be the source of information unit decomposition and extraction, such as the original text, audio, video, and image frames of the speech video to be analyzed; the time range can be the time period or time point in which the evidence appears in the video to be analyzed; the text content can be the text information corresponding to the evidence item, the recognition result from speech, video, image, etc., or the output result obtained through image understanding model and / or video understanding model; and the availability status can be a flag indicating whether the evidence is currently available.

[0079] This embodiment provides a method for analyzing presentation videos. By organizing information units into evidence items and registering each evidence item using a unified evidence index, the original evidence can be quickly located through the unified evidence identifier, forming a traceable evidence chain, thereby improving the traceability and auditability of each piece of evidence.

[0080] In one embodiment, generating evaluation results for each preset evaluation dimension based on each of the evidence items includes:

[0081] Based on a preset evaluation framework document, at least one preset evaluation dimension is determined; the preset evaluation framework document includes evidence screening rules corresponding to each preset evaluation dimension.

[0082] Based on the evidence screening rules and each of the evidence items, an evidence package is generated for each of the preset evaluation dimensions;

[0083] Based on the preset evaluation framework document, the evaluation result of each evidence package is determined.

[0084] The evaluation framework document can have multiple preset evaluation dimensions, and determining at least one preset evaluation dimension can be done by determining the current evaluation dimension in the order of the evaluation framework document.

[0085] The evidence screening rules can be restriction rules used to limit the types of evidence that can be read by the evaluation dimension.

[0086] The step of generating an evidence package for each preset evaluation dimension based on the evidence screening rules and each of the evidence items can be achieved by reading the evidence items that can be read in the corresponding evaluation dimension according to the evidence screening rules, packaging the obtained evidence items, and obtaining an evidence package for each preset evaluation dimension.

[0087] The evaluation result for each evidence package is determined based on the preset evaluation framework document. The preset evaluation framework document can be a document containing evidence interpretation rules, evaluation logic, and a set of tags corresponding to each evaluation dimension, used for tagging the evaluation results. Understandably, each evaluation dimension can contain multiple tags to represent the specific classification under that dimension.

[0088] The evidence interpretation rules and evaluation logic can be used to determine the evaluation conclusion of the evaluation dimension. That is, the evidence items in the evidence package are interpreted through the evidence interpretation rules, and the corresponding evaluation conclusion is generated through the evaluation logic. The evaluation conclusion can be a conclusion obtained by summarizing the evidence items in the evidence package.

[0089] The evidence interpretation rules and evaluation logic can also be used to limit the label classification results of the evidence package, that is, to determine the label of the evaluation result based on the evidence types contained in the evidence package and the corresponding rules and logic; for example, when the evidence types contained in the evidence package are volume, pitch and timbre, and the evaluation dimension is the grammatical dimension, the label of the evaluation result can be tone.

[0090] The evaluation results may include one or more of the following: evaluation unique identifier, label, evaluation conclusion, evidence used, and evaluation framework version.

[0091] This embodiment provides a method for analyzing speech videos. By using evidence screening rules to determine the range of evidence that can be read for each evaluation dimension, an evidence package is generated based on the screened evidence, and evaluation results and labels are given for each evaluation dimension according to the evaluation framework document and the evidence package. This method can avoid information unrelated to the current evaluation dimension from being mixed into the model context, thereby improving the accuracy of the evaluation results.

[0092] In one embodiment, the evidence screening rules include evidence source rules, auxiliary evidence restrictions, and prohibited evidence rules. The step of generating an evidence package for each of the preset evaluation dimensions based on the evidence screening rules and each of the evidence items includes:

[0093] Based on the evidence source rules and the prohibited evidence rules, each of the evidence items is filtered to obtain a first set of evidence items;

[0094] Based on the aforementioned auxiliary evidence constraints, each of the evidence items in the first set of evidence items is classified into primary and secondary categories to obtain a second set of evidence items, which serves as the evidence package under the preset evaluation dimension.

[0095] The evidence source rule may be an admission rule that includes the types of evidence that can be read by the current evaluation dimension, and the evidence disabling rule may be a restriction rule that includes the types of evidence that cannot be read by the current evaluation dimension.

[0096] The supporting evidence restriction can be a classification rule used to indicate which types of evidence in the first set of evidence items should be used as supporting evidence and not as core evidence. For example, the supporting evidence restriction can store classification items that can be used as core evidence, or classification items that should be used as supporting evidence.

[0097] The classification of evidence items in the first set of evidence items based on the constraints of the auxiliary evidence can be based on the classification rules within the constraints. When the constraints store categories that should be considered auxiliary evidence, the evidence type matching the constraints is considered secondary evidence (i.e., auxiliary evidence), and the other evidence types are considered primary evidence (i.e., core evidence). Conversely, when the constraints store categories that can be considered core evidence, the evidence type matching the constraints is considered primary evidence (i.e., core evidence), and the other evidence types are considered secondary evidence (i.e., auxiliary evidence).

[0098] Furthermore, when determining the evaluation result of each evidence package, different weights can be set for primary and secondary evidence, thereby adjusting the influence of core evidence on the evaluation result and making the evaluation result more accurate.

[0099] This embodiment provides a speech video analysis method that restricts the types of evidence that can be read by the evaluation dimension through evidence source rules and prohibited evidence rules. This can prevent evidence unrelated to the current evaluation dimension from being mixed into the model context. At the same time, by restricting the primary and secondary relationships of evidence items through auxiliary evidence, the core evidence can have a greater impact on the evaluation results, thereby improving the accuracy of the evaluation results.

[0100] In one embodiment, the speech video analysis method further includes:

[0101] Availability verification is performed on each of the information units to obtain a first verification result; the availability verification includes one or more of the following: output format verification, field integrity verification, abnormal status marking, failure record, and insufficient information verification.

[0102] The evaluation results are verified to obtain a second verification result; the report verification includes one or more of the following: evidence boundary verification, output format verification, field integrity verification, label legality verification, auxiliary evidence limitation verification, conclusion without evidence verification, visual over-inference verification, and information insufficiency verification.

[0103] If the first verification result is abnormal, the evaluation result associated with the information unit corresponding to the first verification result will be constrained to a non-deterministic evaluation result.

[0104] If the second verification result is abnormal, the evaluation result corresponding to the second verification result will be constrained to a non-deterministic evaluation result.

[0105] The output format verification can be used to check whether the output format of the model conforms to the preset format rules; the field integrity verification can be used to check whether the fields output by the model are missing; the abnormal state marking can be used to mark the information unit according to the abnormal situation of the model; the failure record can be used to log the failure situation that occurs during the splitting and extraction process; and the information insufficiency verification can be used to check whether the information corresponding to the information unit exists in the video to be analyzed.

[0106] For example, the first verification result may include results such as pass, partial missing, insufficient information, and failure.

[0107] The evidence boundary check can be used to check whether the evaluation dimension reads evidence outside the limitations of the evidence rules. The output format verification can be used to check whether the format of the evaluation result meets the preset format. The field integrity check can be used to check whether any fields in the evaluation result are missing. The label legality verification can be used to check whether the labels in the evaluation result are within the label range of the evaluation dimension. The auxiliary evidence limitation verification can be used to check whether the evaluation result uses auxiliary evidence as core evidence. The no-evidence conclusion verification can be used to check whether there are any evaluation results that do not use any evidence. The visual over-inference verification can be used to check whether there are any evidence items in the evaluation result that use a single keyframe as evidence of action and expression. The information insufficiency verification can be used to check whether the evidence in the evaluation result is complete.

[0108] For example, the second verification result may include results such as pass, rejection, downgrade, insufficient information, and pending. Downgrade can reduce the weight of the current evaluation result in type classification, while pending can be marked as pending when the result cannot be classified as pass, rejection, downgrade, or insufficient information after verification, allowing for a second verification.

[0109] Specifically, constraining the evaluation result associated with the information unit corresponding to the first verification result to a non-deterministic evaluation result may mean that if the verification result of the information unit is not pass or fail, then when the evidence item generated based on the information unit is applied to the evaluation result, only the evaluation result may be saved, and the application of the evaluation result to subsequent processes may be restricted.

[0110] The method of constraining the evaluation result corresponding to the second verification result to a non-deterministic evaluation result can refer to the method of constraining the evaluation result associated with the information unit corresponding to the first verification result to a non-deterministic evaluation result as described above. This embodiment will not be elaborated here.

[0111] This embodiment provides a method for analyzing presentation videos. By verifying evidence during evidence generation and evaluation result generation, it can prevent abnormal evidence from entering the subsequent evaluation process and abnormal evaluation results from entering the subsequent suggestion generation process, thereby improving the accuracy of evaluation results and subsequent suggestion generation.

[0112] In one embodiment, the guidance suggestions generated based on the evaluation results for the presentation video to be analyzed include:

[0113] Based on the preset typology framework document and the evaluation results, the typology profile of the speech video to be analyzed is determined.

[0114] Based on the aforementioned type profile, guidance suggestions are generated for the speech video to be analyzed.

[0115] The preset typology framework can be a document used to classify the evaluation results, or a document used to determine the evaluation results as a type profile. For example, the preset typology framework can include type dimensions and optional subtypes, where the optional subtypes can be one or more specific optional types under each type dimension.

[0116] For example, determining the type profile of the speech video to be analyzed based on the preset typology framework document and each of the evaluation results may involve determining the evaluation dimension and the evaluation result corresponding to each type dimension for each type dimension, determining the subtype corresponding to each type dimension based on the evaluation result, and using the subtype corresponding to each type dimension as the type profile of the speech video to be analyzed.

[0117] In one specific embodiment, the guidance suggestions for generating the speech video to be analyzed based on the type profile can be as follows: when the embodied presentation type is stable explanation type, the rhetorical interaction type is expert explanation type, and the semantic content type is technical solution type, the system can generate suggestions such as increasing story entry points, reducing terminology density, and strengthening value resonance paragraphs.

[0118] This embodiment provides a speech video analysis method that generates a type profile of the speech video to be analyzed based on the evaluation results, and determines suggestions for the speech video to be analyzed based on the type profile. This allows the generation of speech suggestions to be dynamically adjusted according to the typological framework while being constrained by the speech video, thereby improving the system's flexibility while ensuring system stability.

[0119] In one embodiment, determining the type profile of the speech video to be analyzed based on a preset typological framework document and the evaluation results includes:

[0120] Based on a pre-defined typological framework document, the evaluation dimensions corresponding to different typological dimensions are determined.

[0121] Based on the evaluation results associated with the evaluation dimensions corresponding to each of the type dimensions, the type labels of the speech video to be analyzed under different type dimensions are determined;

[0122] Based on the type labels under different type dimensions, the type profile of the speech video to be analyzed is determined.

[0123] The step of determining the evaluation dimensions corresponding to different type dimensions based on a preset typology framework document can be achieved by reading the evaluation dimensions corresponding to different type dimensions in the preset typology framework document.

[0124] Understandably, each type dimension may include multiple type labels. In one specific embodiment, the type dimension may include embodied presentation type, rhetorical interaction type, and semantic content type, and the type labels may be, respectively, stable explanation type, high-energy mobilization type, or pause emphasis type under embodied presentation type; expert explanation type, problem-solving type, or story-driven type under rhetorical interaction type; and technical solution type, value-leading type, personal experience type, or public issue type under semantic content type.

[0125] The step of determining the type label of the speech video to be analyzed under different type dimensions based on the evaluation results associated with the evaluation dimensions corresponding to each type dimension can be achieved by matching the type labels with the evaluation results associated with the evaluation dimensions after determining the evaluation dimensions corresponding to the type dimensions, thereby determining the type labels of the speech video to be analyzed under different type dimensions.

[0126] The process of determining the type profile of the speech video to be analyzed based on the type tags under different type dimensions can be achieved by combining the type tags after determining the type tags under each type dimension to obtain the type profile of the speech video to be analyzed.

[0127] This embodiment provides a method for analyzing speech videos. By using type dimensions and their corresponding evaluation dimensions and type labels in a preset typological framework, a type profile of the speech video to be analyzed is generated. This method can determine more refined subtypes while constraining the type dimensions, avoiding coarse classification based solely on type dimensions. This makes the generation of type profiles more accurate, thereby improving the accuracy of guidance and suggestions.

[0128] To more clearly illustrate the technical solution of this application, a detailed embodiment is also provided.

[0129] Speech videos are a type of composite media that simultaneously carries linguistic semantics, auditory expression, and visual behavior. In addition to the explicit content expressed by the written language, the auditory and visual modalities contain supplementary expressive information, reflecting the speaker's emphasis, attitude, and persuasive organization, and influencing the audience's understanding and judgment of the speech content. The speaker's final presentation effect is also closely related to the multimodal information expression. Existing technologies typically preprocess the speech video and perform speech-to-text transcription, then extract multimodal information such as text, audio, and video, and input this information into a multimodal analysis model or a Large Language Model (LLM). The model then outputs a summary, score, modification suggestions, or a natural language report. These technologies can improve the automation of speech content understanding and expression evaluation, and also enable the system to utilize the text understanding, semantic induction, and natural language generation capabilities of the Large Language Model to generate richer analytical results.

[0130] However, existing technologies still have shortcomings: First, because the outputs of different stages are completed by different models, issues such as inconsistent formatting, missing fields, or invalid results may arise. The existing process lacks a unified orchestration body to schedule each processing stage, as well as constraint control mechanisms for verifying, marking, and handling fault tolerance of model and tool outputs. Therefore, an anomaly in any stage can easily cause program blocking. Second, when calling LLM to generate conclusions, existing solutions typically directly input the full multimodal analysis results without limiting the scope of citations based on the specific evaluation task. Information irrelevant to the current evaluation objective may be mixed into the model context, and evidence from different sources and with varying degrees of reliability may be cross-used, leading to judgments without evidence, mixed use of evidence, conclusion drift, or over-inference. Furthermore, the generated conclusions lack clear citation relationships with the original text, audio segments, video clips, and intermediate products, making it difficult to trace back to their factual basis and review them. Furthermore, existing solutions typically output summaries, scores, tags, or recommendations directly from LLM (Local Management Model). The analysis results are primarily free text, failing to organize multimodal observations into structured intermediate results or summarize the speaker's expressive characteristics according to a unified typological analysis process. Therefore, while existing technologies can identify speech content and some acoustic, emotional, or visual features, they struggle to generate stable typological judgments and produce targeted and traceable optimization recommendations based on a traceable chain of multimodal evidence.

[0131] To address the aforementioned issues, an Agent-Harness collaborative mechanism, an evidence boundary control mechanism, and a structured typological analysis method can be introduced into the existing multimodal analysis technology for speech videos. Based on this, the multimodal processing results are uniformly organized into traceable evidence items, and the scope of citationable evidence is limited according to the objectives of different evaluation tasks. Furthermore, a structured typological analysis method is introduced. Typology is a method that compresses complex experience and forms theoretical explanations by selecting key dimensions, comparing attribute combinations, and constructing analytical types. This method uses verified evaluation results as input for typological analysis. According to pre-set classification dimensions, the speaker's embodied presentation, rhetorical interaction, and semantic content are structured and summarized to form embodied presentation types, rhetorical interaction types, and semantic content types, and optimization suggestions corresponding to the speech expression types are generated accordingly.

[0132] Therefore, in one embodiment, such as Figure 3 As shown, a method for analyzing speech videos is provided, the method comprising the following steps:

[0133] Step 1, Task Control, includes task creation and status initialization; using a single English entrepreneur speech video uploaded by the user as the speech video to be analyzed, the task initialization module performs video access, task ID generation and creation time registration, establishes a task directory and records the video path, processing status and output location; when the video is unreadable or the audio track is missing, the task status is marked as failed or partially missing, and the available information is retained.

[0134] Step two, model scheduling, includes model selection, model routing, and model invocation; the media preprocessing module reads the video duration, resolution, frame rate, audio sampling rate, and audio / video stream information, extracts standard waveform audio files (WAV), and generates preprocessing metadata to provide a unified time reference for subsequent speech recognition, acoustic observation, and video segment extraction.

[0135] The speech recognition and time alignment module performs Automatic Speech Recognition (ASR) transcription on the audio, generating the original English transcription, sentence-level timelines, and word-level timestamps. The system retains sentences, keywords, and word-level time ranges as citationable objects, avoiding subsequent evaluations that only cite unverifiable natural language summaries.

[0136] The acoustics and tone observation module generates speech rate, pauses, pitch, volume, stress candidates, voice quality, sound events, and emotional tone observations based on audio and timelines. Acoustic and emotional results serve as evidence of expression methods and do not independently lead to conclusions about values, speaking strategies, or psychological states.

[0137] The text structure analysis module generates paragraph indexes, key sentence backfilling, expression habit diagnosis, dynamic chapters, and selected key sentences based on the original transcription and timeline. It also identifies text anchors needed for subsequent evaluation, such as speech strategies, thematic frameworks, values, appeals, grammar, and language resources.

[0138] The visual segment generation module extracts semantic short video clips and keyframes based on time windows of key sentences, memorable quotes, or important chapters. Short video clips are used to observe dynamic processes, while keyframes are mainly used for location and review, and are not used alone as a basis for judging action processes or psychological states.

[0139] The dynamic visual observation module analyzes short video clips, outputting only visible changes in facial expressions, gaze direction, posture openness, head orientation, gestures, and body movements. When the subject is not visible, the camera obstructs the view, the image is too far away, or the clip is insufficient, the system outputs insufficient information and does not interpret a single frame as a stable personality, genuine attention, or inner emotions.

[0140] In the process of multi-model processing, speech-to-text can use models such as Wisper large-v3, WisperX, or Wisper turbo; events and emotions can use models such as SenseVoice, Emotion2vec_plus, or MSPdimension; acoustics and noise can use models such as Librosa or Voice-Taxonomy-57; and vision and text can use models such as Qwen3.6-plus.

[0141] Step 3, output verification, includes usability checks such as structure and field verification, time and evidence boundary verification, and anomaly feedback. The preprocessed products are verified. Anomaly feedback occurs when downstream processes encounter problems or errors; the system not only records these locally but also sends the error signal back upstream so the source can correct, mark, or revoke them. The preprocessed products include speech-to-text products, text analysis products, acoustic analysis products, and visual analysis products. Speech-to-text products include ASR original transcription, sentence-level timelines, and word-level timestamps. Text analysis products include paragraph indexes, key sentence backfilling, expression habit diagnosis, dynamic chapters, and selected key sentences. Visual analysis products include semantic short videos, semantic keyframes, and dynamic visual observation.

[0142] Furthermore, the tool output can be validated; the tool output includes structured results, file paths, metadata, and execution status generated by media parsing, time alignment, fragment extraction, format conversion, index generation, or other program tools.

[0143] The unified evidence indexing module registers text, word-level timestamps, audio observations, sound events, short video clips, keyframes, paragraph structure, and chapter information as evidence items. Each evidence item includes at least a unique evidence identifier, evidence type, source material, start time, end time, text content or file path, and availability status.

[0144] Step four involves verifying the unified evidence index, including evidence integrity, ID uniqueness, and index consistency. Index consistency refers to whether the evidence obtained by searching the same index in different ways at different times is consistent.

[0145] Step 5, evaluation orchestration, includes evaluation framework reading, evidence package assembly, evaluation scheduling, and report verification. The evaluation framework reading module reads the current open evaluation framework document, such as speech_eval_labels_v0.1, to obtain the evaluation dimensions, candidate labels, label definitions, core evidence, auxiliary evidence, disabled evidence, and version number. This document limits the range of evidence that can be read for each evaluation dimension.

[0146] In one embodiment, such as Figure 4 As shown, the sources of evidence can be speech transcription, text analysis, acoustic analysis, and visual analysis; the evidence package assembly module generates multiple evidence packages according to the evaluation dimensions.

[0147] For example, acoustic parameters are based on acoustic features and noise quality observation as core evidence, while ASR original transcription, sentence-level timeline, word-level timestamp, paragraph index, noise profile summary, and acoustic event analysis are auxiliary evidence. Keyword backfilling, expression habit diagnosis, dynamic chapters, selection of golden sentences, emotion / tone analysis, semantic short videos, semantic keyframes, and dynamic visual observation are evidence of not using / disabling.

[0148] The action and expression dimension uses emotion / tone analysis, semantic short videos, and semantic keyframes as core evidence, and ASR original transcription, sentence-level timeline, word-level timestamp, paragraph index, acoustic features, noise quality observation, noise profile summary, acoustic event analysis, and dynamic visual observation as auxiliary evidence. Keyword backfilling, expression habit diagnosis, dynamic chapters, and selection of golden sentences are used as evidence of non-use / disabling.

[0149] The core evidence for the speech strategy dimension includes ASR original transcription, sentence-level timeline, word-level timestamp, paragraph index, keyword backfilling, expression habit diagnosis, dynamic chapters, and selection of key sentences. Acoustic features, noise quality observation, noise profile summary, and acoustic event analysis serve as auxiliary evidence. Emotion / tone analysis, semantic short videos, semantic keyframes, and dynamic visual observation serve as evidence for not using / disabling speech.

[0150] In the grammar and language resource dimension, key evidence includes keyword backfilling, expression habit diagnosis, dynamic chapters, and selection of golden sentences; auxiliary evidence includes ASR original transcription, sentence-level timeline, word-level timestamp, and paragraph index; and evidence of non-use / disabling includes acoustic features, noise quality observation, noise profile summarization, acoustic event analysis, emotion / tone analysis, semantic short videos, semantic keyframes, and dynamic visual observation.

[0151] The core evidence for the claims is the original ASR transcription, sentence-level timeline, word-level timestamp, and paragraph index. The evidence for not using / disabling the claims includes keyword backfilling, expression habit diagnosis, dynamic chapters, selection of key sentences, acoustic features, noise quality observation, noise profile summary, acoustic event analysis, emotion / tone analysis, semantic short videos, semantic keyframes, and dynamic visual observation.

[0152] The thematic framework uses paragraph indexing, keyword backfilling, expression habit diagnosis, dynamic chapters, and selection of key sentences as core evidence, and ASR original transcription, sentence-level timeline, and word-level timestamp as auxiliary evidence. Acoustic features, noise quality observation, noise profile summary, acoustic event analysis, emotion / tone analysis, semantic short video, semantic keyframe, and dynamic visual observation are used as evidence of non-use / disabling.

[0153] The value dimension uses paragraph indexes, keyword backfilling, expression habit diagnosis, dynamic chapters, and selected golden sentences as core evidence, acoustic features, noise quality observation, noise profile summaries, and acoustic event analysis as auxiliary evidence, and emotion / tone analysis, semantic short videos, semantic keyframes, and dynamic visual observation as evidence of non-use / disabling.

[0154] Each evaluation module generates intermediate evaluation results solely based on its own evidence package and the current evaluation framework document. Each intermediate evaluation result must include at least a unique evaluation identifier, a tag, a Chinese conclusion, a unique evidence identifier, a time range, source material, framework version, and output restrictions. Evaluation modules may not cite evidence outside of the evidence package, nor may they use supporting evidence as a standalone source of strong conclusions; the output restrictions may be rules limiting which fields, tags, and evidence can be included in the output.

[0155] Furthermore, the action and expression dimension first checks whether the short video clip covers the target action, and then combines dynamic visual observation to output the conclusion of visible action. For example, if the clip shows the speaker leaning forward and making gestures before a key sentence, the system can record "open posture" or "emphasis gesture"; if there is only a single keyframe, it can only be used as location evidence and cannot be written as an action process.

[0156] For conclusions such as "building rapport through personal experience," "establishing credibility through causal chains," and "driving persuasion through problem-solving structures," the evaluation results of the speech strategy dimension can be linked to evidence from the original English sentence, sentence-level time range, and paragraph structure; acoustics, tone, and sound events can only describe the atmosphere of expression and cannot support strategy judgments on their own.

[0157] The thematic framework, values, demands, and grammar and language resources are first matched and categorized within the current candidate tags. If a stable tag appears in the evidence package but is not currently covered by the configuration, the evaluation module outputs new tag candidates, along with the candidate tag name, definition, triggering evidence, similar existing tags, and a description of the differences; unconfirmed new tags are not written into the stable evaluation framework.

[0158] Harness performs one or more of the following checks on all evaluation results: evidence boundary checks, output format checks, field integrity checks, label validity checks, supporting evidence limitation checks, no-evidence conclusion checks, visual over-inference checks, and insufficient information checks. Unsatisfactory evaluation results are rejected, downgraded, or marked as insufficient information.

[0159] Furthermore, the evaluation results verified through the Harness report are organized into reviewable structured results for display in the evidence review interface. The review interface allows users to trace back from the conclusion to the original English sentence, audio time segment, short video clip, keyframe, or structured source document; this step only organizes and reviews the verified results and does not serve as a new entry point for freely generated analysis.

[0160] Step six, typological orchestration, includes typological framework reading, type mapping, and analysis scheduling.

[0161] The typology framework reading module reads an editable typology framework document, such as speech_typology_v0.1, to obtain the typology dimensions, typology labels, evidence requirements, exclusion conditions, recommended rules, and typology framework version. The typology dimensions include at least the embodied presentation dimension, the rhetorical interaction dimension, and the semantic content dimension. The typology analysis module only reads evaluation results that have passed report verification or the organized structured results, without rereading the original video, full transcription, or intermediate products that have not passed verification. The exclusion conditions are used to restrict the selection of subtypes based on the evaluation results. Even if the evaluation result matches a certain subtype, the subtype cannot be used as the final subtype. For example, a single increase in volume during a speech is not sufficient to classify it as "high-energy mobilization type." After the exclusion conditions are triggered, the system retains the original evaluation results and evidence and continues to judge other subtypes. If no candidate type meets the conditions, the system outputs "Insufficient information."

[0162] In one exemplary embodiment, such as Figure 4As shown, the typology analysis module generates typological profiles, including embodied presentation type, rhetorical interaction type, and semantic content type. Embodied presentation type can be categorized into stable explanatory type, high-energy mobilization type, or pause-emphasis type based on acoustic parameters and facial expressions. Rhetorical interaction type can be categorized into expert explanatory type, problem-solving type, or story-driven type based on speech strategies, grammar and language resources, and appeals. Semantic content type can be categorized into technical solution type, value-leading type, personal experience type, or public issue type based on thematic framework and values.

[0163] Furthermore, if a certain profile lacks sufficient evidence, such as insufficient visual fragments, low ASR transcription quality, or insufficient value evidence, the system outputs "pending" and explains the type of missing evidence.

[0164] Step seven, outcome control, includes outcome summarization, evidence retrospection, and recommended constraints.

[0165] The analysis and suggestion module generates guidance suggestions based on type profile analysis. For example, when the embodied presentation type is stable explanation type, the rhetorical interaction type is expert explanation type, and the semantic content type is technical solution type, the system can generate suggestions such as increasing story entry points, reducing terminology density, and strengthening value resonance paragraphs, and set more specific action paths.

[0166] Each suggestion is linked to a supporting type, a unique evaluation identifier, a unique evidence identifier, or a time range. Users can retrieve the corresponding English sentence, audio clip, short video clip, or structured evaluation results from the suggestions to determine whether the suggestions are supported by evidence.

[0167] In the running Peter Steinberger Technology, Entertainment, Design (TED) demo case, the system has generated output modules including acoustic parameters, presentation strategies, body language, thematic framework, values, appeals, grammar, and language resources. Some modules are incomplete, and items with insufficient or limited information are retained, indicating that the system will not force a complete conclusion when evidence is insufficient. This example demonstrates the process from multimodal data mining to type profiling analysis, and then to guidance and suggestions.

[0168] In one embodiment, such as Figure 5As shown, the speech video analysis interface includes a "Create New Task" button in the upper right corner and a "Task Center" on the left, displaying completed tasks (Sundar Pichai's Google I / O speech, Satya Nadella's Build 2024 speech, Elon Musk's product launch speech) or abnormal tasks (Reid Hoffhan's startup sharing). Each task includes its status and creation date. The currently executing task is Peter Steinberger's TED speech, which includes the video duration, task creation date, and buttons to stop the task and open the table of contents. Below is the task flow (input, preprocessing, evidence, evaluation, report) and the time consumption and outputs (segment.json, transcript.json, speakers.json, keypoints.json, evidence.json) of each stage (1. segmentation and alignment, 2. speech recognition (ASR), 3. speaker separation, 4. key information extraction, 5. evidence mapping and classification, 6. evidence deduplication and aggregation, 7. evidence quality assessment, 8. supporting relationship construction). Currently, we are in the evidence flow (generating the Evidence Packet, 16 / 22 steps completed). We have completed the stages from segmentation and alignment to key information extraction. We are currently in the evidence mapping and classification stage. The stages to be completed are evidence deduplication and aggregation and supporting relationship construction.

[0169] The right side displays the execution details, including the current step (Evidence Packet, step 5 / 22), the current elapsed time (12 minutes and 58 seconds), the estimated remaining time (4 minutes and 58 seconds), the input files (Peter_Steinberger_TED.mp4, Peter_Steinberger_TED.srt), and some output files (transcript.json, keypoints.json, evidence_map.json). It also includes warnings about two insufficient visual fragments, the lack of clear keyframes in some key visual content which may affect the integrity of the evidence, a "View Details" button, an "Expand Technical Log" button, and a "View Analysis Results" button at the bottom.

[0170] In one embodiment, such as Figure 6As shown, the analysis results interface is titled with the video title (Peter Steinberger's TED Talk). The left sidebar displays the analysis results, including an overview, timeline, seven-dimensional analysis (acoustic parameters, presentation strategy, body language, thematic framework, values, appeals, and language resources), evidence index, and a complete report. Under the presentation strategy dimension, it includes all evaluations, available evaluations, and evaluations indicating insufficient information. Among the all evaluations are: one tagged "Building Credibility" ("Building credibility through personal experience, quickly bridging the gap with the audience"), with the limitation "No live audience feedback observed, unable to assess the actual impact on trust"; another tagged "Stimulating Thinking" ("One question drives thinking, stimulating audience participation and curiosity"), with the limitation "Based on only a single question, unable to determine whether a continuous chain of thought is formed"; and yet another tagged "Simplifying Complexity" ("Using analogies and metaphors to help understand complex concepts, lowering the cognitive threshold"), with limitations. The description states "Insufficient background information to accurately match all audience members." Each review provides supporting evidence (original English text) and a timeframe, along with a "View Evidence" button. The right sidebar displays evidence and media, including the original speech video. Below the original speech video is the original text evidence (in English) and its timeframe within the original video. This evidence is considered core evidence, typed as textual evidence, along with its limitations and source file (Peter_Steinberger_TED_Talk.mp4). The right sidebar also includes buttons to jump to the video and open the source file. The bottom sidebar is divided into chapters by time (about my teaching experience, common programming pitfalls, how we limit ourselves, breaking the boundaries of thinking, action and change), key quotes, and pauses, tone information, and relevant video clips from the video, with zoom in, zoom out, and full-screen buttons.

[0171] This embodiment provides a speech video analysis method that, compared to traditional techniques that directly generate corresponding guidance suggestions from speech videos, extracts and decomposes multimodal information from the speech video and encodes the obtained information into multiple evidence items. Based on these evidence items, corresponding evaluations and guidance suggestions are generated. This allows each evaluation result and guidance suggestion to be specifically indexed to the corresponding evidence item, enabling the final guidance suggestion to combine the evaluation results corresponding to multiple evidence items to generate more accurate guidance suggestions, thereby improving the technical effect of improving the accuracy of the model's generated conclusions.

[0172] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0173] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a presentation video analysis method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0174] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0175] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the presentation video analysis method of any of the above embodiments:

[0176] Based on a pre-defined multimodal model, the video of the speech to be analyzed is split and extracted to obtain multiple information units of each modality at different scales;

[0177] Evidence items are generated based on each of the aforementioned information units;

[0178] Based on each of the aforementioned evidence items, an evaluation result is generated for each preset evaluation dimension.

[0179] Based on the evaluation results, guidance suggestions are generated for the speech video to be analyzed.

[0180] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the speech video analysis method of any of the above embodiments:

[0181] Based on a pre-defined multimodal model, the video of the speech to be analyzed is split and extracted to obtain multiple information units of each modality at different scales;

[0182] Evidence items are generated based on each of the aforementioned information units;

[0183] Based on each of the aforementioned evidence items, an evaluation result is generated for each preset evaluation dimension.

[0184] Based on the evaluation results, guidance suggestions are generated for the speech video to be analyzed.

[0185] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0186] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0187] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for analyzing speech videos, characterized in that, The speech video analysis method includes: Based on a pre-defined multimodal model, the video of the speech to be analyzed is split and extracted to obtain multiple information units of each modality at different scales; Evidence items are generated based on each of the aforementioned information units; Based on each of the aforementioned evidence items, an evaluation result is generated for each preset evaluation dimension. This generation of evaluation results for each preset evaluation dimension includes: determining at least one preset evaluation dimension based on a preset evaluation framework document; the preset evaluation framework document includes evidence screening rules corresponding to each preset evaluation dimension; generating an evidence package for each preset evaluation dimension based on the evidence screening rules and each of the aforementioned evidence items; and determining the evaluation result for each evidence package based on the preset evaluation framework document. Based on the evaluation results, guidance suggestions are generated for the speech video to be analyzed.

2. The speech video analysis method according to claim 1, characterized in that, The preset multimodal model includes a splitting and extraction module corresponding to each mode; The process involves splitting and extracting information from the speech video to be analyzed based on a pre-defined multimodal model, resulting in multiple information units for each modality at different scales, including: Based on the preset module calling order, each splitting and extraction module in the preset multimodal model is called sequentially to split and extract the speech video to be analyzed once or multiple times, thereby obtaining multiple information units corresponding to the splitting and extraction module. When the splitting and extraction result of the splitting and extraction module does not meet the preset conditions, the splitting and extraction module is processed based on the processing strategy of the intermediate state of the splitting and extraction module in the preset state transition rules.

3. The speech video analysis method according to claim 1, characterized in that, The generation of evidence items based on each of the information units includes: Based on the information unit, evidence items in the unified evidence index are generated; the evidence item includes one or more of the following: unique evidence identifier, evidence type, source product, time range, text content, file path, and availability status.

4. The speech video analysis method according to claim 1, characterized in that, The evidence screening rules include evidence source rules, auxiliary evidence restrictions, and prohibited evidence rules. The step of generating an evidence package for each of the preset evaluation dimensions based on the evidence screening rules and each of the evidence items includes: Based on the evidence source rules and the prohibited evidence rules, each of the evidence items is filtered to obtain a first set of evidence items; Based on the aforementioned auxiliary evidence constraints, each of the evidence items in the first set of evidence items is classified into primary and secondary categories to obtain a second set of evidence items, which serves as the evidence package under the preset evaluation dimension.

5. The speech video analysis method according to claim 1, characterized in that, The speech video analysis method also includes: Availability verification is performed on each of the information units to obtain a first verification result; the availability verification includes one or more of the following: output format verification, field integrity verification, abnormal status marking, failure record, and insufficient information verification. The evaluation results are verified to obtain a second verification result; the report verification includes one or more of the following: evidence boundary verification, output format verification, field integrity verification, label legality verification, auxiliary evidence limitation verification, conclusion without evidence verification, visual over-inference verification, and information insufficiency verification. If the first verification result is abnormal, the evaluation result associated with the information unit corresponding to the first verification result will be constrained to a non-deterministic evaluation result. If the second verification result is abnormal, the evaluation result corresponding to the second verification result will be constrained to a non-deterministic evaluation result.

6. The speech video analysis method according to claim 1, characterized in that, The guidance suggestions for generating the presentation video to be analyzed based on the evaluation results include: Based on the preset typology framework document and the evaluation results, the typology profile of the speech video to be analyzed is determined. Based on the aforementioned type profile, guidance suggestions are generated for the speech video to be analyzed.

7. The speech video analysis method according to claim 6, characterized in that, The process of determining the type profile of the speech video to be analyzed, based on the preset typology framework document and the evaluation results, includes: Based on a pre-defined typological framework document, the evaluation dimensions corresponding to different typological dimensions are determined. Based on the evaluation results associated with the evaluation dimensions corresponding to each of the type dimensions, the type labels of the speech video to be analyzed under different type dimensions are determined; Based on the type labels under different type dimensions, the type profile of the speech video to be analyzed is determined.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the speech video analysis method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the speech video analysis method according to any one of claims 1 to 7.

Images