Video quality diagnosis system and method based on multi-modal large model

Abstract

The application relates to the technical field of video diagnosis, and particularly discloses a video quality diagnosis system and method based on a multi-modal large model, which comprises a data acquisition module, a data set construction module, a model fine-tuning module, a preliminary diagnosis module and a deep diagnosis module, constructs a professional knowledge enhanced data set of a current video quality diagnosis process, fine-tunes a visual-linguistic base model in a field by using the professional knowledge enhanced data set, obtains a field video quality diagnosis model, inputs video frame sequences and equipment metadata in a to-be-diagnosed original video stream into the field video quality diagnosis model, and outputs a preliminary diagnosis result; deep root causes leading to fault phenomena in the preliminary diagnosis result are inferred to generate a structured comprehensive diagnosis report; and the application can improve the diagnosis efficiency of video quality.

Description

Technical Field

[0001] This invention relates to the field of video diagnostic technology, and in particular to a video quality diagnostic system and method based on a multimodal large model. Background Technology

[0002] Traditional video quality diagnostic technologies often rely on single visual feature detection and manual rule matching, making it difficult to jointly analyze video frame temporal information and device metadata. This results in a limited fault identification dimension, failing to comprehensively cover video quality anomaly types in complex scenarios, leading to omissions and misjudgments, and ultimately, low overall diagnostic accuracy. Furthermore, existing technologies lack optimization training based on expertise in video quality diagnostics, resulting in insufficient model generalization and domain adaptability. The inference of the correlation between fault phenomena and root causes is inefficient, hindering the rapid generation of structured diagnostic reports. Consequently, the overall efficiency of video quality diagnosis and problem tracing fails to meet the needs of large-scale operation and maintenance.

[0003] Existing video quality diagnostic solutions lack deep fusion applications of multimodal large models, enabling only superficial identification of fault phenomena and failing to infer deep root causes based on fault confidence. Diagnostic results are fragmented and lack a unified structured output standard. Furthermore, historical diagnostic data and professional knowledge have not been standardized into datasets to support model iteration, resulting in poor scalability and difficulty in adapting to the video quality diagnostic needs of different devices and scenarios. Long-term use exacerbates the accumulation of diagnostic errors, making it impossible to achieve continuous and stable high-quality video diagnostic services. Summary of the Invention

[0004] This invention provides a video quality diagnostic system and method based on a multimodal large model to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a video quality diagnostic system based on a multimodal large model, characterized in that the system includes a data acquisition module, a dataset construction module, a model fine-tuning module, a preliminary diagnosis module, and a deep diagnosis module, wherein: The data acquisition module is used to extract the video frame sequence containing timing information and the corresponding device metadata from the original video stream to be diagnosed; The dataset construction module is used to construct a professional knowledge-enhanced dataset for the current video quality diagnosis process based on the historical database accumulated during the historical video quality diagnosis process. The model fine-tuning module is used to fine-tune the open-source vision-language base model using the professional knowledge augmentation dataset to obtain a domain-specific video quality diagnosis model for the current video quality diagnosis process. The preliminary diagnosis module is used to input the video frame sequence and the device metadata into the domain-specific video quality diagnosis model, and output the preliminary diagnosis results of the current video quality diagnosis process; The deep diagnostic module is used to infer the underlying root cause of the fault phenomenon in the primary diagnostic result based on the fault phenomenon description and corresponding confidence level in the primary diagnostic result, so as to generate a structured comprehensive diagnostic report of the current video quality diagnostic process.

[0006] In a preferred embodiment, when the data acquisition module extracts the video frame sequence containing timing information and the corresponding device metadata from the original video stream to be diagnosed, it is specifically used for: The original video stream to be diagnosed is demultiplexed to obtain the video encoded stream and metadata stream of the original video stream; The video encoded stream is decoded to obtain the original video frame sequence of the original video stream; Record the playback timestamps of the original video frame sequence to obtain a video frame sequence containing timing information; Device metadata is parsed from the metadata stream. The device metadata includes device identifier, acquisition time, and video parameters.

[0007] In a preferred embodiment, when the dataset construction module constructs a professional knowledge-enhanced dataset for the current video quality diagnosis process based on a historical database accumulated during historical video quality diagnosis, it is specifically used for: Historical diagnostic cases are obtained from the historical database. Each historical diagnostic case includes historical video clips, descriptions of historical fault phenomena corresponding to the historical video clips, and the historical root causes that led to the descriptions of historical fault phenomena. Based on the description of the historical fault phenomena, corresponding professional questions are generated, and based on the historical root causes, expert answers to the professional questions are generated. The analysis chain from the description of the historical fault phenomenon to the root cause is obtained from the historical diagnostic cases, and the analysis chain is used as the reasoning path information connecting the professional problem and the expert answer; The historical video clips, the professional questions, the expert answers, and the reasoning path information are associated and stored to obtain a professional knowledge-enhanced dataset for the current video quality diagnosis process.

[0008] In a preferred embodiment, when the model fine-tuning module executes a domain-specific video quality diagnosis model based on an open-source vision-language foundation model and uses the expertise-enhanced dataset to fine-tune the vision-language foundation model to obtain the domain-specific video quality diagnosis model for the current video quality diagnosis process, it is specifically used for: The professional knowledge enhancement dataset is divided into a training sample set and a validation sample set. The training samples in the training sample set include historical video clips, corresponding professional questions, and corresponding expert answers. The open-source visual-language base model is iteratively trained by taking the historical video clips and the professional questions as inputs and the expert answers as the expected outputs. After training, the model is validated using the validation sample set, and the training parameters of the model are adjusted according to the validation results until the matching degree between the model's output on the validation sample set and the expert's answer reaches the preset requirements. The model that meets the preset requirements is determined as the domain-specific video quality diagnosis model for the current video quality diagnosis process.

[0009] In a preferred embodiment, the model fine-tuning module, after training is completed, uses the validation sample set to validate the currently trained model and adjusts the model's training parameters based on the validation results until the model's output on the validation sample set matches the expert answer to a preset requirement. Specifically, this is used to: Input the historical video clips and professional questions from the validation sample set into the currently trained model, and obtain the predicted answers output by the model; The predicted answer is compared with the corresponding expert answer in the verification sample set to determine the degree of matching between the predicted answer and the expert answer; If the matching degree does not meet the preset requirements, the training parameters of the model are adjusted according to the matching degree, and the adjusted model is trained in the next round using the training sample set. Repeat the training process until the matching degree reaches the preset requirement.

[0010] In a preferred embodiment, when the preliminary diagnosis module inputs the video frame sequence and the device metadata into the domain-specific video quality diagnosis model and outputs the preliminary diagnosis result of the current video quality diagnosis process, it is specifically used for: The video frame sequence and the device metadata are input into the domain-specific video quality diagnostic model; In the domain-specific video quality diagnostic model, visual analysis is performed on the video frame sequence to obtain a description of the fault phenomena related to video quality in the video frame sequence; Semantic parsing is performed on the device metadata to obtain the operating status information reflecting the operating status of the acquisition device from the device metadata; The fault phenomenon description is correlated with the operating condition information to determine the fault category and corresponding confidence level in the original video stream. The fault categories and their corresponding confidence levels are structurally integrated to obtain the preliminary diagnostic results of the current video quality diagnostic process.

[0011] In a preferred embodiment, when the preliminary diagnosis module performs correlation analysis between the fault phenomenon description and the operating condition information to determine the fault category and corresponding confidence level in the original video stream, it is specifically used for: Based on a preset set of fault categories and the corresponding visual anomaly weight coefficients and operating condition parameter weight coefficients, a mapping table between fault categories and weight coefficients is constructed. Semantic features are extracted from the description of the fault phenomenon to obtain an abnormal semantic vector of the description of the fault phenomenon, and the initial visual confidence level corresponding to the fault category is determined based on the degree of matching between the abnormal semantic vector and the fault category. An anomaly measurement is performed on the current state of the equipment operating parameters in the operating condition information to obtain an anomaly degree index for the equipment operating parameters. Based on the association mapping table, the initial visual confidence score and the anomaly index are weighted and fused to obtain the comprehensive confidence score of the fault category; The overall confidence level is compared with a preset threshold one by one to filter out the fault categories corresponding to the overall confidence level that is greater than the preset threshold. These are then used as the fault categories present in the original video stream, and the filtered overall confidence level is determined as the confidence level of the corresponding fault category.

[0012] In a preferred embodiment, the formula for calculating the overall confidence level is as follows: ; In the formula, Indicates the first The overall confidence level of each fault category Indicates the first Initial visual confidence for each fault category, Indicates the first The visual anomaly weight coefficients corresponding to each fault category Indicates the first The fault category corresponds to the first Weighting coefficients for each operating condition parameter Indicates the first Under the fault category, the first An indicator of the degree of abnormality of each equipment's operating parameters. This represents the total number of operating parameters of the equipment.

[0013] In a preferred embodiment, when the deep diagnostic module performs the inference of the underlying root cause of the fault phenomenon in the primary diagnostic results based on the fault phenomenon description and corresponding confidence level in the primary diagnostic results, in order to generate a structured comprehensive diagnostic report for the current video quality diagnostic process, it is specifically used for: The preliminary diagnostic results are subjected to structured analysis to obtain the fault phenomenon description and corresponding confidence level of the preliminary diagnostic results; Based on a preset fault cause mapping library, semantic association matching is performed on the fault phenomenon description to obtain a list of candidate root causes of the fault phenomenon description and the prior confidence of the candidate root causes. Based on the confidence level corresponding to the description of the fault phenomenon, a comprehensive confidence level assessment is performed on the candidate root causes in the candidate root cause list and their corresponding prior confidence levels to obtain the underlying root cause of the fault phenomenon in the primary diagnosis result. The underlying root causes and the descriptions of the fault phenomena are structured, integrated, and encapsulated to obtain a structured comprehensive diagnostic report for the current video quality diagnostic process.

[0014] To address the aforementioned problems, this invention also provides a video quality diagnosis method based on a multimodal large model, the method comprising: Step 1: Extract the video frame sequence containing timing information and the corresponding device metadata from the original video stream to be diagnosed; Step 2: Based on the historical database accumulated during the historical video quality diagnosis process, construct a professional knowledge-enhanced dataset for the current video quality diagnosis process; Step 3: Based on the open-source vision-language base model, the vision-language base model is fine-tuned using the aforementioned professional knowledge augmentation dataset to obtain the domain-specific video quality diagnosis model for the current video quality diagnosis process; Step 4: Input the video frame sequence and the device metadata into the domain-specific video quality diagnostic model, and output the preliminary diagnostic results of the current video quality diagnostic process; Step 5: Based on the description of the fault phenomena and the corresponding confidence levels in the primary diagnostic results, infer the underlying root causes of the fault phenomena in the primary diagnostic results to generate a structured comprehensive diagnostic report for the current video quality diagnostic process.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention performs joint diagnosis by fusing video frame timing information and device metadata, and improves the accuracy of fault identification by combining weighted confidence calculation, effectively reducing misjudgments and omissions, and achieving comprehensive and accurate detection of video quality anomalies.

[0016] 2. This invention achieves domain-specific fine-tuning of the model by enhancing it with a professional knowledge dataset, which can automatically infer the deep-seated root causes of faults and generate structured reports, significantly improving the overall efficiency of video quality diagnosis and problem tracing. Attached Figure Description

[0017] Figure 1 This is a system architecture diagram of a video quality diagnostic system based on a multimodal large model provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating a video quality diagnosis method based on a multimodal large model, provided in an embodiment of the present invention.

[0018] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments belong to some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “said” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0021] Depending on the context, the word "if" or "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0022] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0023] In practice, the server-side equipment deployed in a video quality diagnostic system based on a multimodal large model may consist of one or more devices. This system can be implemented as a business instance, a virtual machine, or hardware devices. For example, it can be implemented as a business instance deployed on one or more devices in a cloud node. Simply put, it can be understood as software deployed on a cloud node to provide video quality diagnostics based on a multimodal large model to various user terminals. Alternatively, it can be implemented as a virtual machine deployed on one or more devices in a cloud node, with application software installed to manage various user terminals. Or, it can be implemented as a server composed of numerous identical or different types of hardware devices, with one or more devices configured to provide video quality diagnostics based on a multimodal large model to various user terminals.

[0024] In terms of implementation, the video quality diagnostic system based on the multimodal large model and the user terminal are mutually adaptable. That is, if the video quality diagnostic system based on the multimodal large model is implemented as an application installed on a cloud service platform, then the user terminal is implemented as a client that establishes a communication connection with the application; or if the video quality diagnostic system based on the multimodal large model is implemented as a website, then the user terminal is implemented as a webpage; or if the video quality diagnostic system based on the multimodal large model is implemented as a cloud service platform, then the user terminal is implemented as a mini-program in an instant messaging application.

[0025] like Figure 1 The figure shown is a system architecture diagram of a video quality diagnosis system based on a multimodal large model provided in an embodiment of the present invention.

[0026] The video quality diagnostic system 100 based on a multimodal large model described in this invention can be located on a cloud server. In terms of implementation, it can be used as one or more service devices, or as an application installed in the cloud (e.g., a mobile service operator's server, server cluster, etc.), or it can be developed as a website. Depending on the functions implemented, the video quality diagnostic system 100 based on a multimodal large model may include a data acquisition module 101, a dataset construction module 102, a model fine-tuning module 103, a preliminary diagnosis module 104, and a deep diagnosis module 105. The modules described in this invention can also be called units, referring to a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, stored in the memory of the electronic device.

[0027] In this embodiment of the invention, in the video quality diagnostic system based on a multimodal large model, each of the above modules can be implemented independently and can call other modules. Here, "calling" can be understood as a module connecting to multiple modules of another type and providing corresponding services to those connected modules. In the video quality diagnostic system based on a multimodal large model provided by this embodiment of the invention, the applicability of the system architecture can be adjusted by adding modules and directly calling them without modifying the program code, achieving cluster-based horizontal expansion to quickly and flexibly expand the video quality diagnostic system based on a multimodal large model. In practical applications, the above modules can be set in the same device or different devices, or they can be set in virtual devices, such as service instances in a cloud server.

[0028] The following describes the components and workflow of a video quality diagnostic system based on a multimodal large model, using specific embodiments as examples: The data acquisition module 101 is used to extract the video frame sequence containing timing information and the corresponding device metadata from the original video stream to be diagnosed; In this embodiment of the invention, when the data acquisition module extracts the video frame sequence containing timing information and the corresponding device metadata from the original video stream to be diagnosed, it is specifically used for: The original video stream to be diagnosed is demultiplexed to obtain the video encoded stream and metadata stream of the original video stream; The video encoded stream is decoded to obtain the original video frame sequence of the original video stream; Record the playback timestamps of the original video frame sequence to obtain a video frame sequence containing timing information; Device metadata is parsed from the metadata stream. The device metadata includes device identifier, acquisition time, and video parameters.

[0029] The original video stream to be diagnosed is demultiplexed to obtain the video encoded stream and metadata stream. The demultiplexing process requires a demultiplexer module designed to parse the container format of the video. Taking MP4 as an example, the demultiplexer reads the binary data of the original video stream and identifies the `ftyp` box at the beginning of the file, which indicates the file type. Next, the demultiplexer locates the `moov` box, which contains the metadata of all media data. Inside the `moov` box, the demultiplexer finds the `trak` boxes, each corresponding to a media track. The demultiplexer distinguishes between the video track and the track containing metadata. Then, based on the information in the `moov` box, the demultiplexer locates the start position and length of the data corresponding to the video track and metadata track in the `mdat` box. Finally, the demultiplexer reads and copies these two parts of data completely, generating two independent data streams: the video encoded stream and the metadata stream.

[0030] The video encoded stream is decoded to obtain the original video frame sequence of the original video stream. The decoding process is performed by a video decoder. The video encoded stream is compressed data; assuming it uses the H.264 standard, the decoder first parses the sequence parameter set and image parameter set from the video encoded stream. These parameter sets contain the configuration information necessary for decoding the entire sequence and each frame. Then, the decoder begins processing the encoded data of one frame. It first performs entropy decoding, for example, using context-adaptive binary arithmetic decoding, to restore the compressed binary codewords into a series of syntax elements. These syntax elements include quantization coefficients and motion vectors, etc. Then, these quantization coefficients undergo an inverse quantization process, that is, each coefficient is multiplied by a specific scaling factor to recover the variable... The transformation coefficients are then subjected to an inverse transformation, typically an inverse discrete cosine transform, to be converted back to residual data in the pixel domain. If the current frame is an inter-coded frame, the decoder will retrieve the corresponding pixel block as the prediction value from the reference frame that has already been decoded and stored in the buffer, based on the motion vector parsed from the bitstream. If the current frame is an intra-coded frame, the decoder will use the adjacent pixels that have already been decoded in the current frame to make predictions in a specific direction and generate prediction blocks. Finally, the decoder adds the residual data obtained from the inverse transformation to the prediction data pixel by pixel to obtain the complete decoded image, i.e., the original video frame. All frames are arranged in the decoding order to form the original video frame sequence.

[0031] Record the playback timestamps of the original video frame sequence to obtain a video frame sequence containing timing information. The playback timestamps originate from the video encoded stream. In H.264 streams, timestamp information is usually contained in the display timestamp syntax element. When the decoder decodes the video encoded stream, it simultaneously parses the display timestamp corresponding to each frame. When the decoder outputs an original video frame, it binds the display timestamp value parsed from the bitstream at that moment to the pixel data of that frame. For example, the timestamp value is written as an attribute field into the header of the frame data structure. This timestamp value is a number representing the position of the frame relative to the start point on the playback timeline, usually in milliseconds. By appending its corresponding playback timestamp to each frame in the original video frame sequence, the original image sequence is transformed into a video frame sequence containing timing information. Each element in this new sequence clearly knows its position on the timeline.

[0032] Device metadata, including device identifier, acquisition time, and video parameters, is parsed from the metadata stream. The metadata stream is a structured binary data block. The parsing process is performed by a metadata parser. The parser first reads all data from the metadata stream into a memory buffer, and then reads it according to a preset byte offset order and field length. Starting from the zeroth byte of the buffer, the parser reads twenty bytes consecutively, interpreting these twenty bytes into a string using ASCII character encoding. This string is the device identifier, such as a device serial number. Next, the parser moves to the twentieth byte and reads eight bytes from there, interpreting these eight bytes into a 64-bit integer. This integer represents the number of milliseconds elapsed since the start of Coordinated Universal Time (UTC), i.e., the timestamp of the acquisition time. The parser then converts this timestamp to local time. The parser reads the date, time, and frame rate in year, month, day, hour, minute, and second format. Then, it moves to the 28th byte and begins reading the video parameters. The video parameters consist of multiple parameter pairs, each consisting of a parameter name and a parameter value. The parser first reads the parameter name length in one byte, then reads the corresponding number of bytes of the parameter name string based on this length, and then reads the parameter value in two bytes. The parameter value may be interpreted as an integer or a string. This process is repeated until all parameters are read. For example, if the parser finds a resolution of 1920 x 1080 and a frame rate of 30, all the parsed information is finally integrated into a structured data object. This object is the device metadata, which explicitly contains specific fields such as device identifier, acquisition time, and video parameters.

[0033] The dataset construction module 102 is used to construct a professional knowledge-enhanced dataset for the current video quality diagnosis process based on the historical database accumulated during the historical video quality diagnosis process. In this embodiment of the invention, when the dataset construction module executes the construction of a professional knowledge-enhanced dataset for the current video quality diagnosis process based on a historical database accumulated during historical video quality diagnosis, it is specifically used for: Historical diagnostic cases are obtained from the historical database. Each historical diagnostic case includes historical video clips, descriptions of historical fault phenomena corresponding to the historical video clips, and the historical root causes that led to the descriptions of historical fault phenomena. Based on the description of the historical fault phenomena, corresponding professional questions are generated, and based on the historical root causes, expert answers to the professional questions are generated. The analysis chain from the description of the historical fault phenomenon to the root cause is obtained from the historical diagnostic cases, and the analysis chain is used as the reasoning path information connecting the professional problem and the expert answer; The historical video clips, the professional questions, the expert answers, and the reasoning path information are associated and stored to obtain a professional knowledge-enhanced dataset for the current video quality diagnosis process.

[0034] Historical diagnostic cases are retrieved from the historical database. These cases include historical video clips, descriptions of the corresponding historical fault phenomena, and the root causes of those fault phenomena. The historical database is a relational database containing a table named "Diagnostic Cases." This table contains a case number, video clip storage path, fault phenomenon description text field, and root cause text field. The retrieval process is achieved by executing a Structured Query Language (SQL) statement. This statement instructs the database management system to select all records from the Diagnostic Cases table and retrieve the values ​​of the video clip storage path, fault phenomenon description, and root cause fields from each record. Based on the video clip storage path value, the system locates the corresponding video file from the specified file server or object storage and loads its binary data completely into memory. This video file is the historical video clip. The retrieved text field values ​​represent the historical fault phenomenon description and the historical root cause, respectively. These three elements, combined as a whole, constitute a complete historical diagnostic case object.

[0035] The system generates corresponding professional questions based on historical fault descriptions and expert answers based on the root causes of those faults. The professional question generation is achieved using a text conversion template, a pre-defined string containing the phrase "Please diagnose the possible causes of the following phenomenon." The system directly fills the specified placeholder positions in this template string with the text describing the historical fault phenomena, forming a new, complete interrogative sentence—the professional question. The expert answer generation is achieved using another text conversion template, containing the phrase "The root cause of this phenomenon is..." The system fills the specified placeholder positions in this template with the text describing the historical root causes, forming a declarative sentence—the expert answer. The entire generation process is a deterministic string concatenation operation, involving no inference or creation.

[0036] The system retrieves the analytical chain from historical diagnostic cases, tracing the historical fault phenomena to their root causes. This chain serves as a reasoning path connecting professional questions with expert answers. The analytical chain is stored in a separate table in the historical database, named the "Analysis Chain Table," which contains a case number and an analysis step text field. The system links the diagnostic case table and the analysis chain table using the case number, executing a join query. For each historical diagnostic case, the system finds the record with the same number in the analysis chain table and reads its analysis step text field. This field records the step-by-step logical process, in the form of "Observing phenomenon A, it is speculated that component B may be faulty; after checking, parameter C is abnormal, therefore the cause is located as D." This complete logical statement constitutes the analysis chain. The system extracts this text content verbatim and names it "Reasoning Path Information," thus completing the retrieval process.

[0037] Historical video clips, professional questions, expert answers, and reasoning path information are linked and stored to obtain the professional knowledge enhancement dataset for the current video quality diagnosis process. This linked storage is achieved by creating a new data table named the Professional Knowledge Enhancement Dataset Table. This table has four fields: video clip data, question text, answer text, and path text. The system constructs a new data record in memory, filling the first field with the binary data of the historical video clips, the second field with the generated professional question text string, the third field with the generated expert answer text string, and the fourth field with the obtained reasoning path information text string, thus forming a complete record. The system then performs a database insertion operation, writing this record containing the values ​​of all four fields into the newly created Professional Knowledge Enhancement Dataset Table. This database table, containing all the transformed records of historical cases, is the final professional knowledge enhancement dataset.

[0038] The model fine-tuning module 103 is used to fine-tune the visual-language base model based on the open-source visual-language base model using the professional knowledge augmentation dataset, so as to obtain the domain-specific video quality diagnosis model of the current video quality diagnosis process. In this embodiment of the invention, when the model fine-tuning module executes a domain-specific video quality diagnosis model based on an open-source vision-language foundation model and uses the professional knowledge augmentation dataset to fine-tune the vision-language foundation model to obtain the domain-specific video quality diagnosis model for the current video quality diagnosis process, it is specifically used for: The professional knowledge enhancement dataset is divided into a training sample set and a validation sample set. The training samples in the training sample set include historical video clips, corresponding professional questions, and corresponding expert answers. The open-source visual-language base model is iteratively trained by taking the historical video clips and the professional questions as inputs and the expert answers as the expected outputs. After training, the model is validated using the validation sample set, and the training parameters of the model are adjusted according to the validation results until the matching degree between the model's output on the validation sample set and the expert's answer reaches the preset requirements. The model that meets the preset requirements is determined as the domain-specific video quality diagnosis model for the current video quality diagnosis process.

[0039] The model fine-tuning module, after training, uses the validation sample set to validate the currently trained model and adjusts the model's training parameters based on the validation results until the model's output on the validation sample set matches the expert answer to a preset requirement. Specifically, this is used for: Input the historical video clips and professional questions from the validation sample set into the currently trained model, and obtain the predicted answers output by the model; The predicted answer is compared with the corresponding expert answer in the verification sample set to determine the degree of matching between the predicted answer and the expert answer; If the matching degree does not meet the preset requirements, the training parameters of the model are adjusted according to the matching degree, and the adjusted model is trained in the next round using the training sample set. Repeat the training process until the matching degree reaches the preset requirement.

[0040] The professional knowledge enhancement dataset is divided into a training sample set and a validation sample set. The training samples in the training sample set contain historical video clips, corresponding professional questions, and corresponding expert answers. The partitioning method is based on chronological order. The system reads all data records in the professional knowledge enhancement dataset and sorts them in ascending order according to the capture time field of the video clip in each record. Starting with the earliest sorted record, the system selects the first 80% of the records and assigns them to a set named the training sample set. Each record in this set contains all three parts: a historical video clip, a professional question, and an expert answer, constituting a training sample. The remaining 20% ​​of the records are assigned to another set, named the validation sample set, which has the same record structure as the training sample set.

[0041] This system iteratively trains an open-source vision-language foundation model, using historical video clips and expert questions as input and expert answers as the expected output. The training process begins with preparing the model input. The system reads a training sample from the training sample set, decodes the historical video clips within that sample into a series of image frames, and simultaneously reads the expert question text from the sample, converting both the image frame sequence and the question text into a vector representation acceptable to the model. Next, the model uses these vectors as input for computation, and its internal network layers output a vector sequence representing the answer. Simultaneously, the system also converts the expert answer text from the training sample into its corresponding vector representation, serving as the expected output. The core of the training is calculating the difference between the model's actual output vector and the expected output vector, measured using a cross-entropy function. Based on this difference, the system automatically calculates the adjustment direction and magnitude of each adjustable parameter within the model; this process is called backpropagation. Subsequently, the system updates all parameter values ​​of the model according to the calculated adjustment magnitude. Completing this process for one training sample is called one iteration; the system iterates through every sample in the training sample set, completing one round of training.

[0042] The system inputs historical video clips and professional questions from the validation sample set into the currently trained model to obtain the model's predicted answer. The system sequentially reads each sample from the validation sample set. For each sample, the system decodes the historical video clips it contains into an image frame sequence and converts the professional question text it contains into a text vector. These two sets of inputs are fed into the model, which has completed one round of training. The model processes this input, its internal network layers perform calculations, and finally, the output layer generates a probability distribution. This distribution corresponds to the probability that each word in its vocabulary will be the next output word. The model uses a greedy decoding strategy, selecting the word with the highest probability as the current output each time, and using that word as the input for the next round of decoding. This process is repeated to generate a word sequence until the model outputs a special symbol indicating the end. The final word sequence is converted into a text string, which is the model's predicted answer for that validation sample.

[0043] The predicted answer is compared with the corresponding expert answer in the validation sample set to determine the degree of match between the predicted and expert answers. The comparison method uses a complete string matching approach. The system reads the expert answer text for the current sample from the validation sample set, which is a standard answer string. Simultaneously, the system reads the predicted answer text generated by the model for the current sample. The degree of match is calculated by comparing the two strings character by character. The system sets a counter to an initial value of zero, and then starts comparing from the first character of each string. If the character is the same, the counter is incremented by one, and the comparison continues until either string ends. The final counter value represents the number of consecutive identical characters in the two strings. The system divides this number of identical characters by the total number of characters in the expert answer string to obtain a value between zero and one. This value is defined as the degree of match, with one representing a complete match and zero representing a complete mismatch. The system performs the above comparison process for every sample in the validation sample set and records the average degree of match for all samples.

[0044] If the matching degree does not meet the preset requirement, the model's training parameters are adjusted based on the matching degree, and the adjusted model is trained again using the training sample set. The preset requirement is a specific numerical threshold, such as 0.9. The system compares the calculated average matching degree with this threshold. If the average matching degree is less than 0.9, it is determined that the preset requirement has not been met. The specific method for adjusting the training parameters is to adjust the learning rate proportionally. The system presets an initial learning rate, such as 0.001. If the matching degree is lower than 0.9, the system multiplies the current learning rate by 0.9 to obtain a smaller new learning rate value. After adjustment, the system uses this new, smaller learning rate value as the step size control for parameter updates, re-traverses all samples in the training sample set, and executes the complete model forward computation, loss calculation, backpropagation, and parameter update process again. This is the next round of training.

[0045] The training process is repeated until the matching degree reaches the preset requirement. After each round of model training, the system automatically performs a complete validation process, which involves processing the entire validation sample set with the current model to obtain a new round of average matching degree, and comparing it with the preset threshold of 0.9. If the average matching degree is still less than 0.9, the system reduces the learning rate again according to the aforementioned rules and starts a new round of training. This "training-validation-comparison-adjustment" cycle will repeat continuously until the average matching degree calculated after a certain validation is equal to or greater than 0.9, at which point the cycle terminates.

[0046] The model that meets the preset requirements is selected as the domain-specific video quality diagnostic model for the current video quality diagnostic process. When the model's validation match reaches 0.9, the system stops the training loop. At this point, the system serializes and saves the parameter values ​​of all layers in the model, including weights and biases, as well as the complete structural definition of the model, into a separate model file. This model file is given a specific name, namely the domain-specific video quality diagnostic model. This model has the ability to receive video clips and question text as input and output high-quality diagnostic answer text, specifically for the current video quality diagnostic task.

[0047] The preliminary diagnosis module 104 is used to input the video frame sequence and the device metadata into the domain-specific video quality diagnosis model, and output the preliminary diagnosis results of the current video quality diagnosis process; In this embodiment of the invention, when the preliminary diagnosis module inputs the video frame sequence and the device metadata into the domain-specific video quality diagnosis model and outputs the preliminary diagnosis result of the current video quality diagnosis process, it is specifically used for: The video frame sequence and the device metadata are input into the domain-specific video quality diagnostic model; In the domain-specific video quality diagnostic model, visual analysis is performed on the video frame sequence to obtain a description of the fault phenomena related to video quality in the video frame sequence; Semantic parsing is performed on the device metadata to obtain the operating status information reflecting the operating status of the acquisition device from the device metadata; The fault phenomenon description is correlated with the operating condition information to determine the fault category and corresponding confidence level in the original video stream. The fault categories and their corresponding confidence levels are structurally integrated to obtain the preliminary diagnostic results of the current video quality diagnostic process.

[0048] When the preliminary diagnosis module performs correlation analysis between the fault phenomenon description and the operating condition information to determine the fault category and corresponding confidence level in the original video stream, it is specifically used for: Based on a preset set of fault categories and the corresponding visual anomaly weight coefficients and operating condition parameter weight coefficients, a mapping table between fault categories and weight coefficients is constructed. Semantic features are extracted from the description of the fault phenomenon to obtain an abnormal semantic vector of the description of the fault phenomenon, and the initial visual confidence level corresponding to the fault category is determined based on the degree of matching between the abnormal semantic vector and the fault category. An anomaly measurement is performed on the current state of the equipment operating parameters in the operating condition information to obtain an anomaly degree index for the equipment operating parameters. Based on the association mapping table, the initial visual confidence score and the anomaly index are weighted and fused to obtain the comprehensive confidence score of the fault category; The overall confidence level is compared with a preset threshold one by one to filter out the fault categories corresponding to the overall confidence level that is greater than the preset threshold. These are then used as the fault categories present in the original video stream, and the filtered overall confidence level is determined as the confidence level of the corresponding fault category.

[0049] The formula for calculating the overall confidence level is as follows: ; In the formula, Indicates the first The overall confidence level of each fault category Indicates the first Initial visual confidence for each fault category, Indicates the first The visual anomaly weight coefficients corresponding to each fault category Indicates the first The fault category corresponds to the first Weighting coefficients for each operating condition parameter Indicates the first Under the fault category, the first An indicator of the degree of abnormality of each equipment's operating parameters. This represents the total number of operating parameters of the equipment.

[0050] Derived from the category of fault The value is calculated by weighting and fusing the initial visual confidence level and the abnormality index of the equipment operating parameters. This value is defined as the comprehensive confidence level of the fault category. The abnormal semantic vector is derived from the semantic feature extraction of fault phenomenon descriptions by a domain-specific video diagnostic model. This vector corresponds to the fault category. The value obtained by calculating the similarity of the standard vectors and normalizing it is defined as the initial visual confidence score. Each fault category is derived from a preset set of fault categories. A fixed value is pre-specified and stored in the association mapping table; this value is defined as the visual anomaly weight coefficient. The fault category is derived from the preset fault category set. The corresponding number Each equipment operating condition parameter is pre-specified and stored in a fixed value in an association mapping table. This value is defined as the operating condition parameter weight coefficient. The operating condition information derived from parsing the device metadata is the first... The current value of each equipment operating parameter is used to measure its abnormality. The current value is compared with the preset normal range and the relative deviation is calculated. This value is defined as the abnormality index of the parameter. The total number of measurable parameters in the operating status information reflecting the equipment's operating status, obtained after semantic parsing of the equipment's metadata, is calculated by the system through counting.

[0051] The video frame sequence and device metadata are input into the domain-specific video quality diagnostic model. The system loads the image data of each frame of the video frame sequence as a three-dimensional tensor, and simultaneously loads the text information of the device metadata as a string. Through the data input interface defined by the model, these two parts of data are simultaneously passed to the processing module of the domain-specific video quality diagnostic model. After receiving these inputs, the model begins its internal calculation process.

[0052] In the domain-specific video quality diagnostic model, visual analysis is performed on the video frame sequence to obtain descriptions of fault phenomena related to video quality. The model uses its visual encoder component to process the video frame sequence. The visual encoder reads each image tensor frame by frame, extracts spatial features through convolutional layers, and then analyzes the temporal relationships between frames through recurrent neural network layers to identify abnormal visual patterns in the video, such as image blurring or color distortion. Finally, these patterns are summarized and output as a text description through a fully connected layer. This text description is the fault phenomenon description.

[0053] Semantic parsing of device metadata yields operational status information reflecting the data acquisition device's operating condition. The model uses its text encoder component to process the device metadata. The text encoder reads the string of device metadata, converts each word into a vector through a word embedding layer, and then uses an attention mechanism layer to understand the semantic relationships between fields. From this, it extracts specific values ​​such as the device type corresponding to the device identifier, the environmental timestamp corresponding to the acquisition time, and the resolution and frame rate corresponding to the video parameters. This information is then organized into a structured status report, which is the operational status information.

[0054] Based on a pre-defined set of fault categories and the corresponding visual anomaly weight coefficients and operating condition parameter weight coefficients, a mapping table is constructed between fault categories and their weight coefficients. The system pre-stores a list of fault categories, such as video stuttering and video snow, and assigns two numerical weight coefficients to each category. The system reads this data and creates a hash table data structure, storing each fault category name as the key and the corresponding visual anomaly weight coefficient and operating condition parameter weight coefficient as the value. This hash table is the mapping table.

[0055] Semantic features are extracted from the description of the fault phenomenon to obtain anomaly semantic vectors. Based on the degree of matching between the anomaly semantic vectors and the fault categories, the initial visual confidence level corresponding to each fault category is determined. The model uses a text encoder to convert the text describing the fault phenomenon into a 300-dimensional real-number vector, i.e., the anomaly semantic vector. Then, the cosine similarity between this vector and a preset standard vector for each fault category is calculated. The standard vectors are learned from the training data, and the cosine similarity value is calculated using a dot product and a modulus. The similarity value is then linearly scaled to the range of zero to one; this scaled value is the initial visual confidence level corresponding to each fault category.

[0056] The system measures the current state of equipment operating parameters in the operating condition information to obtain an anomaly index for the degree of anomaly. The system extracts the current value of each equipment operating parameter, such as temperature or voltage, from the operating condition information, compares the current value with a preset normal range, and if the current value is within the normal range, the anomaly degree is zero. If it exceeds the range, the system calculates the absolute difference between the current value and the normal boundary value, divides it by the width of the normal range to obtain the relative deviation, and takes the maximum relative deviation for all parameters. This maximum value is the anomaly degree index.

[0057] Based on the association mapping table, the initial visual confidence score and anomaly severity index are weighted and fused to obtain the comprehensive confidence score of the fault category. This is achieved by... The product of the initial visual confidence score and the visual anomaly weight coefficient for each fault category, and the product of the initial visual confidence score and the visual anomaly weight coefficient for the first fault category, and the product of the initial visual confidence score and the visual anomaly weight coefficient for the second fault category. The sum of the products of the abnormality index of each equipment operating condition parameter under each fault category and the corresponding weight coefficient of the operating condition parameter is used to achieve the analysis of the fault category. The calculation of the overall confidence level of each fault category integrates visual and operational parameter dimensional information to evaluate the confidence level of the fault category.

[0058] The system iterates through all fault categories and compares them against a preset threshold. Fault categories with overall confidence scores greater than the preset threshold are selected as the fault categories present in the original video stream, and the selected overall confidence scores are determined as the confidence scores for those fault categories. The system presets a threshold of 0.6. It iterates through all fault categories, comparing each overall confidence score with 0.6. If the overall confidence score is greater than 0.6, the fault category name is added to a result list, and the overall confidence score value is recorded as the confidence score for that category. If the overall confidence score is less than or equal to 0.6, it is skipped.

[0059] The system structurally integrates fault categories and their corresponding confidence levels to obtain the preliminary diagnostic results of the current video quality diagnosis process. The system organizes the selected fault categories and their corresponding confidence levels into a dictionary data structure, with each entry using the fault category name as the key and the confidence level value as the value. This dictionary is then serialized into a JSON string, which represents the preliminary diagnostic result.

[0060] The deep diagnostic module 105 is used to infer the underlying root cause of the fault phenomenon in the primary diagnostic result based on the fault phenomenon description and corresponding confidence level in the primary diagnostic result, so as to generate a structured comprehensive diagnostic report of the current video quality diagnostic process.

[0061] In this embodiment of the invention, when the deep diagnostic module performs the inference of the underlying root cause of the fault phenomenon in the primary diagnostic result based on the fault phenomenon description and corresponding confidence level in the primary diagnostic result, in order to generate a structured comprehensive diagnostic report for the current video quality diagnostic process, it is specifically used for: The preliminary diagnostic results are subjected to structured analysis to obtain the fault phenomenon description and corresponding confidence level of the preliminary diagnostic results; Based on a preset fault cause mapping library, semantic association matching is performed on the fault phenomenon description to obtain a list of candidate root causes of the fault phenomenon description and the prior confidence of the candidate root causes. Based on the confidence level corresponding to the description of the fault phenomenon, a comprehensive confidence level assessment is performed on the candidate root causes in the candidate root cause list and their corresponding prior confidence levels to obtain the underlying root cause of the fault phenomenon in the primary diagnosis result. The underlying root causes and the descriptions of the fault phenomena are structured, integrated, and encapsulated to obtain a structured comprehensive diagnostic report for the current video quality diagnostic process.

[0062] The preliminary diagnostic results are parsed in a structured manner to obtain the fault phenomenon description and corresponding confidence level. The preliminary diagnostic result is a JSON string, structured as a dictionary object containing key-value pairs, where the key is the fault category name and the value is the confidence level. The parsing process uses a JSON parser. The system reads the JSON string, converts it into an in-memory dictionary object, and then iterates through each entry. For each entry, the key is directly extracted as the fault phenomenon description, and the value is directly extracted as the corresponding confidence level. These two elements are then stored as a pair in a new list, where each element explicitly contains both the fault phenomenon description and the confidence level.

[0063] Based on a pre-defined fault cause mapping library, semantic association matching is performed on fault phenomenon descriptions to obtain a list of candidate root causes and their prior confidence levels. The fault cause mapping library is a relational database table containing three fields: fault phenomenon description, candidate root cause, and prior confidence level. The matching process uses an exact string matching method. The system takes the first description text from the parsed fault phenomenon description list and uses it as a query condition. A structured query is performed in the fault cause mapping library table to find records where the fault phenomenon description field value is exactly the same as the query text. Then, from these matching records, the values ​​of the candidate root cause field and the prior confidence level are extracted. Each candidate root cause and its corresponding prior confidence level are treated as a tuple. The set of all these tuples constitutes the list of candidate root causes and their prior confidence levels.

[0064] Based on the confidence level corresponding to the fault symptom description, a comprehensive confidence assessment is performed on the candidate root causes in the candidate root cause list and their corresponding prior confidence levels to obtain the underlying root cause of the fault symptom in the preliminary diagnosis results. For each candidate root cause in the candidate root cause list, the system reads its prior confidence level and the confidence level corresponding to the fault symptom description, multiplies these two values, and obtains a product, which is the comprehensive confidence level of the candidate root cause. The system calculates this comprehensive confidence level value for each candidate root cause in the list, then compares all calculated comprehensive confidence level values, selects the candidate root cause corresponding to the largest comprehensive confidence level value, and identifies this selected candidate root cause as the underlying root cause.

[0065] The system structurally integrates and encapsulates the descriptions of root causes and fault phenomena to obtain a structured comprehensive diagnostic report for the current video quality diagnosis process. The system creates a new dictionary object containing three key-value pairs: the first key is the fault phenomenon description, with the value being the text describing the fault phenomenon obtained from the parsing of the primary diagnostic results; the second key is the root cause, with the value being the text of the assessed root cause; and the third key is the overall confidence score, with the value being the overall confidence score corresponding to the root cause. The system then uses an XML serializer to convert this dictionary object into an XML-formatted string, which is the structured comprehensive diagnostic report.

[0066] Reference Figure 2 The diagram shown is a flowchart illustrating a video quality diagnosis method based on a multimodal large model according to an embodiment of the present invention. In this embodiment, the video quality diagnosis method based on a multimodal large model includes: Step 1: Extract the video frame sequence containing timing information and the corresponding device metadata from the original video stream to be diagnosed; Step 2: Based on the historical database accumulated during the historical video quality diagnosis process, construct a professional knowledge-enhanced dataset for the current video quality diagnosis process; Step 3: Based on the open-source vision-language base model, the vision-language base model is fine-tuned using the aforementioned professional knowledge augmentation dataset to obtain the domain-specific video quality diagnosis model for the current video quality diagnosis process; Step 4: Input the video frame sequence and the device metadata into the domain-specific video quality diagnostic model, and output the preliminary diagnostic results of the current video quality diagnostic process; Step 5: Based on the description of the fault phenomena and the corresponding confidence levels in the primary diagnostic results, infer the underlying root causes of the fault phenomena in the primary diagnostic results to generate a structured comprehensive diagnostic report for the current video quality diagnostic process.

[0067] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0068] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A video quality diagnostic system based on a multimodal large model, characterized in that, The system includes a data acquisition module, a dataset construction module, a model fine-tuning module, a preliminary diagnosis module, and a deep diagnosis module, wherein: The data acquisition module is used to extract the video frame sequence containing timing information and the corresponding device metadata from the original video stream to be diagnosed; The dataset construction module is used to construct a professional knowledge-enhanced dataset for the current video quality diagnosis process based on the historical database accumulated during the historical video quality diagnosis process. The model fine-tuning module is used to fine-tune the open-source vision-language base model using the professional knowledge augmentation dataset to obtain a domain-specific video quality diagnosis model for the current video quality diagnosis process. The preliminary diagnosis module is used to input the video frame sequence and the device metadata into the domain-specific video quality diagnosis model, and output the preliminary diagnosis results of the current video quality diagnosis process; The deep diagnostic module is used to infer the underlying root cause of the fault phenomenon in the primary diagnostic result based on the fault phenomenon description and corresponding confidence level in the primary diagnostic result, so as to generate a structured comprehensive diagnostic report of the current video quality diagnostic process.

2. The video quality diagnostic system based on a multimodal large model as described in claim 1, characterized in that, When the data acquisition module extracts the video frame sequence containing timing information and the corresponding device metadata from the original video stream to be diagnosed, it is specifically used for: The original video stream to be diagnosed is demultiplexed to obtain the video encoded stream and metadata stream of the original video stream; The video encoded stream is decoded to obtain the original video frame sequence of the original video stream; Record the playback timestamps of the original video frame sequence to obtain a video frame sequence containing timing information; Device metadata is parsed from the metadata stream. The device metadata includes device identifier, acquisition time, and video parameters.

3. The video quality diagnostic system based on a multimodal large model as described in claim 1, characterized in that, When the dataset construction module executes the process of constructing a professional knowledge-enhanced dataset for the current video quality diagnosis process based on the historical database accumulated during the historical video quality diagnosis process, it is specifically used for: Historical diagnostic cases are obtained from the historical database. Each historical diagnostic case includes historical video clips, descriptions of historical fault phenomena corresponding to the historical video clips, and the historical root causes that led to the descriptions of historical fault phenomena. Based on the description of the historical fault phenomena, corresponding professional questions are generated, and based on the historical root causes, expert answers to the professional questions are generated. The analysis chain from the description of the historical fault phenomenon to the root cause is obtained from the historical diagnostic cases, and the analysis chain is used as the reasoning path information connecting the professional problem and the expert answer; The historical video clips, the professional questions, the expert answers, and the reasoning path information are associated and stored to obtain a professional knowledge-enhanced dataset for the current video quality diagnosis process.

4. The video quality diagnostic system based on a multimodal large model as described in claim 1, characterized in that, When the model fine-tuning module executes a domain-specific video quality diagnosis model based on an open-source vision-language foundation model and uses the aforementioned expertise-enhanced dataset to fine-tune the vision-language foundation model to obtain the domain-specific video quality diagnosis model for the current video quality diagnosis process, it is specifically used for: The professional knowledge enhancement dataset is divided into a training sample set and a validation sample set. The training samples in the training sample set include historical video clips, corresponding professional questions, and corresponding expert answers. The open-source visual-language base model is iteratively trained by taking the historical video clips and the professional questions as inputs and the expert answers as the expected outputs. After training, the model is validated using the validation sample set, and the training parameters of the model are adjusted according to the validation results until the matching degree between the model's output on the validation sample set and the expert's answer reaches the preset requirements. The model that meets the preset requirements is determined as the domain-specific video quality diagnosis model for the current video quality diagnosis process.

5. The video quality diagnostic system based on a multimodal large model as described in claim 4, characterized in that, The model fine-tuning module, after training, uses the validation sample set to validate the currently trained model and adjusts the model's training parameters based on the validation results until the model's output on the validation sample set matches the expert answer to a preset requirement. Specifically, this is used for: Input the historical video clips and professional questions from the validation sample set into the currently trained model, and obtain the predicted answers output by the model; The predicted answer is compared with the corresponding expert answer in the verification sample set to determine the degree of matching between the predicted answer and the expert answer; If the matching degree does not meet the preset requirements, the training parameters of the model are adjusted according to the matching degree, and the adjusted model is trained in the next round using the training sample set. Repeat the training process until the matching degree reaches the preset requirement.

6. The video quality diagnostic system based on a multimodal large model as described in claim 1, characterized in that, When the preliminary diagnosis module inputs the video frame sequence and the device metadata into the domain-specific video quality diagnosis model and outputs the preliminary diagnosis result of the current video quality diagnosis process, it is specifically used for: The video frame sequence and the device metadata are input into the domain-specific video quality diagnostic model; In the domain-specific video quality diagnostic model, visual analysis is performed on the video frame sequence to obtain a description of the fault phenomena related to video quality in the video frame sequence; Semantic parsing is performed on the device metadata to obtain the operating status information reflecting the operating status of the acquisition device from the device metadata; The fault phenomenon description is correlated with the operating condition information to determine the fault category and corresponding confidence level in the original video stream. The fault categories and their corresponding confidence levels are structurally integrated to obtain the preliminary diagnostic results of the current video quality diagnostic process.

7. The video quality diagnostic system based on a multimodal large model as described in claim 6, characterized in that, When the preliminary diagnosis module performs correlation analysis between the fault phenomenon description and the operating condition information to determine the fault category and corresponding confidence level in the original video stream, it is specifically used for: Based on a preset set of fault categories and the corresponding visual anomaly weight coefficients and operating condition parameter weight coefficients, a mapping table between fault categories and weight coefficients is constructed. Semantic features are extracted from the description of the fault phenomenon to obtain an abnormal semantic vector of the description of the fault phenomenon, and the initial visual confidence level corresponding to the fault category is determined based on the degree of matching between the abnormal semantic vector and the fault category. An anomaly measurement is performed on the current state of the equipment operating parameters in the operating condition information to obtain an anomaly degree index for the equipment operating parameters. Based on the association mapping table, the initial visual confidence score and the anomaly index are weighted and fused to obtain the comprehensive confidence score of the fault category; The overall confidence level is compared with a preset threshold one by one to filter out the fault categories corresponding to the overall confidence level that is greater than the preset threshold. These are then used as the fault categories present in the original video stream, and the filtered overall confidence level is determined as the confidence level of the corresponding fault category.

8. The video quality diagnostic system based on a multimodal large model as described in claim 7, characterized in that, The formula for calculating the overall confidence level is as follows: ； In the formula, Indicates the first The overall confidence level of each fault category Indicates the first Initial visual confidence for each fault category, Indicates the first The visual anomaly weight coefficients corresponding to each fault category Indicates the first The fault category corresponds to the first Weighting coefficients for each operating condition parameter Indicates the first Under the fault category, the first An indicator of the degree of abnormality of each equipment's operating parameters. This represents the total number of operating parameters of the equipment.

9. The video quality diagnostic system based on a multimodal large model as described in claim 1, characterized in that, When the deep diagnostic module executes the inference of the underlying root causes of the fault phenomena in the primary diagnostic results based on the fault phenomenon descriptions and corresponding confidence levels in the primary diagnostic results, in order to generate a structured comprehensive diagnostic report for the current video quality diagnostic process, it is specifically used for: The preliminary diagnostic results are subjected to structured analysis to obtain the fault phenomenon description and corresponding confidence level of the preliminary diagnostic results; Based on a preset fault cause mapping library, semantic association matching is performed on the fault phenomenon description to obtain a list of candidate root causes of the fault phenomenon description and the prior confidence of the candidate root causes. Based on the confidence level corresponding to the description of the fault phenomenon, a comprehensive confidence level assessment is performed on the candidate root causes in the candidate root cause list and their corresponding prior confidence levels to obtain the underlying root cause of the fault phenomenon in the primary diagnosis result. The underlying root causes and the descriptions of the fault phenomena are structured, integrated, and encapsulated to obtain a structured comprehensive diagnostic report for the current video quality diagnostic process.

10. A video quality diagnosis method based on a multimodal large model, characterized in that, The method for using the video quality diagnostic system based on a multimodal large model as described in claim 1: Step 1: Extract the video frame sequence containing timing information and the corresponding device metadata from the original video stream to be diagnosed; Step 2: Based on the historical database accumulated during the historical video quality diagnosis process, construct a professional knowledge-enhanced dataset for the current video quality diagnosis process; Step 3: Based on the open-source vision-language base model, the vision-language base model is fine-tuned using the aforementioned professional knowledge augmentation dataset to obtain the domain-specific video quality diagnosis model for the current video quality diagnosis process; Step 4: Input the video frame sequence and the device metadata into the domain-specific video quality diagnostic model, and output the preliminary diagnostic results of the current video quality diagnostic process; Step 5: Based on the description of the fault phenomena and the corresponding confidence levels in the primary diagnostic results, infer the underlying root causes of the fault phenomena in the primary diagnostic results to generate a structured comprehensive diagnostic report for the current video quality diagnostic process.

Images