Video evaluation method and device based on artificial intelligence, equipment and medium

Abstract

The invention provides a video evaluation method and device based on artificial intelligence, an intelligent agent, electronic equipment, a storage medium and a program product, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes such as content generation based on artificial intelligence. The specific implementation scheme is as follows: in response to a received evaluation request, determining a to-be-evaluated video and an evaluation index of the to-be-evaluated video; under the condition that the evaluation index comprises a time sequence evaluation index, analyzing the to-be-evaluated video to obtain content information used for describing the to-be-evaluated video; based on the content information and the evaluation indexes, evaluation prompt information is generated; and inputting the assessment prompt information and the to-be-assessed video into the multi-modal large model to obtain a time sequence assessment result for the to-be-assessed video.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of computer vision, deep learning, and large models, and can be applied to scenarios such as content generation based on artificial intelligence. Specifically, it relates to video evaluation methods, devices, intelligent agents, electronic devices, storage media, and program products based on artificial intelligence. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, videos generated using AI-based content creation methods are becoming increasingly common, such as instructional videos, dance videos, and news broadcasts. However, the quality of videos generated automatically varies greatly. Therefore, video quality assessment before distribution has become a crucial technology. Summary of the Invention

[0003] This disclosure provides a video evaluation method, apparatus, intelligent agent, electronic device, storage medium, and program product based on artificial intelligence.

[0004] According to one aspect of this disclosure, an artificial intelligence-based video evaluation method is provided, comprising: in response to receiving an evaluation request, determining a video to be evaluated and evaluation metrics for the video to be evaluated; if the evaluation metrics include temporal evaluation metrics, analyzing the video to be evaluated to obtain content information describing the video to be evaluated; generating evaluation prompt information based on the content information and the evaluation metrics; and inputting the evaluation prompt information and the video to be evaluated into a multimodal large model to obtain a temporal evaluation result for the video to be evaluated.

[0005] According to another aspect of this disclosure, an artificial intelligence-based video evaluation apparatus is provided, comprising: a determination module, configured to determine a video to be evaluated and evaluation metrics for the video to be evaluated in response to receiving an evaluation request; an analysis module, configured to analyze the video to be evaluated, wherein the evaluation metrics include temporal evaluation metrics, to obtain content information describing the video to be evaluated; a generation module, configured to generate evaluation prompt information based on the content information and the evaluation metrics; and an evaluation module, configured to input the evaluation prompt information and the video to be evaluated into a multimodal large model to obtain a temporal evaluation result for the video to be evaluated.

[0006] According to another aspect of this disclosure, an artificial intelligence agent is provided, comprising: an input module for receiving input information; a processing module for determining a target task based on the input information received by the input module, determining a multimodal large model based on the target task, and obtaining output information by calling the multimodal large model to execute the method described above; and an output module for outputting the output information obtained by the processing module.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described above.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 This illustration schematically shows an exemplary system architecture for applying artificial intelligence-based video evaluation methods and apparatus according to embodiments of the present disclosure;

[0013] Figure 2 A flowchart illustrating an artificial intelligence-based video evaluation method according to an embodiment of the present disclosure is shown schematically.

[0014] Figure 3 A flowchart illustrating an artificial intelligence-based video evaluation method according to another embodiment of the present disclosure is shown schematically;

[0015] Figure 4 A flowchart illustrating the determination of content information of a video to be evaluated according to an embodiment of this disclosure is shown schematically.

[0016] Figure 5 A flowchart illustrating an artificial intelligence-based video evaluation method according to another embodiment of the present disclosure is shown schematically;

[0017] Figure 6 A block diagram of an artificial intelligence-based video evaluation apparatus according to an embodiment of the present disclosure is shown schematically.

[0018] Figure 7 A schematic diagram illustrating the structure of an intelligent agent of artificial intelligence according to embodiments of the present disclosure; and

[0019] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] Figure 1 The illustration schematically shows an exemplary system architecture for applying artificial intelligence-based video evaluation methods and apparatus according to embodiments of the present disclosure.

[0022] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to embodiments of this disclosure, intended to help those skilled in the art understand the technical content of this disclosure. However, they do not imply that embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For instance, in another embodiment, an exemplary system architecture for applying an AI-based video evaluation and apparatus may include a terminal device. However, the terminal device can implement the AI-based video evaluation and apparatus provided in the embodiments of this disclosure without interacting with a server.

[0023] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0024] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0025] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0026] Server 105 can be a server that provides various services, such as a backend management server that supports the content browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0027] It should be noted that the AI-based video evaluation method provided in this disclosure can generally be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103. Correspondingly, the AI-based video evaluation device provided in this disclosure can also be disposed in the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0028] Alternatively, the AI-based video evaluation method provided in this disclosure can generally be executed by server 105. Correspondingly, the AI-based video evaluation device provided in this disclosure can generally be located in server 105. The AI-based video evaluation method provided in this disclosure can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the AI-based video evaluation device provided in this disclosure can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0029] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0030] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0031] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0032] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0033] Figure 2 A flowchart illustrating an artificial intelligence-based video evaluation method according to an embodiment of the present disclosure is shown.

[0034] like Figure 2 As shown, the method includes operations S210~S240.

[0035] In operation S210, in response to receiving an evaluation request, the video to be evaluated and the evaluation metrics for the video to be evaluated are determined.

[0036] In operation S220, when the evaluation metrics include time-series evaluation metrics, the video to be evaluated is analyzed to obtain content information describing the video to be evaluated.

[0037] In operation S230, evaluation prompts are generated based on content information and evaluation indicators.

[0038] In operation S240, the evaluation prompt information and the video to be evaluated are input into the multimodal large model to obtain the time-series evaluation results for the video to be evaluated.

[0039] For example, the video to be evaluated can be generated using artificial intelligence-based content generation methods. For instance, the video to be evaluated can be generated using a large model, a diffusion model, a generative adversarial model, or other generative models.

[0040] The video to be evaluated can be a collection of key video frames, but it is not limited to this; it can also be a complete segment before frame splitting. Temporal evaluation metrics can be quantitative indicators used to assess whether the target object in the video to be evaluated achieves consistency, logic, and reasonableness over time.

[0041] For example, content information and evaluation metrics can be combined into structured information or instructions to obtain evaluation prompts. The content information indicates the evaluation content, and the evaluation metrics indicate the evaluation method. Different evaluation content can be generated based on different videos to be evaluated, and different evaluation prompts can be flexibly generated by combining the evaluation metrics.

[0042] For example, a multimodal large model refers to a deep learning model that is jointly trained with large-scale multimodal data and has the ability to perceive, understand, and generate across modalities. The multimodal large model can be a general large model or an expert large model obtained by fine-tuning based on requirements. The embodiments of this disclosure do not limit this, as long as it is a large model that can simultaneously process text-type evaluation prompts and image-type videos to be evaluated.

[0043] According to embodiments of this disclosure, when the evaluation metrics include time-series evaluation metrics, content information describing the video to be evaluated is obtained through automated analysis, and corresponding evaluation prompts are generated based on the content information and evaluation metrics. This can improve the scope and types of multimodal large model evaluation of the video to be evaluated, thereby improving the flexibility, breadth and refinement of video evaluation.

[0044] According to embodiments of this disclosure, for example, Figure 2 The operation S210 shown, which determines the evaluation index of the video to be evaluated based on the evaluation request for the video to be evaluated, may include: performing intent recognition on the evaluation request to obtain intent recognition results; determining initial evaluation index based on intent recognition results; and determining the initial evaluation index as the evaluation index if the initial evaluation index belongs to a predetermined evaluation index type.

[0045] For example, a pre-trained intent recognition model can be used to identify the intent of an evaluation request to obtain an intent recognition result. This disclosure is not limited to this; key phrases can also be extracted from the evaluation request, and intent recognition results can be obtained by matching these key phrases with predetermined keywords.

[0046] The predetermined evaluation indicator type can include multiple indicator types. These may include time-series evaluation indicators, image quality evaluation indicators, audio evaluation indicators, and so on. This disclosure is not limited to these; the predetermined evaluation indicator type may also include other types of evaluation indicators.

[0047] According to embodiments of this disclosure, by first identifying the intent of the evaluation request and then determining the evaluation index based on the intent identification result, the evaluation index can be flexibly determined according to different evaluation requests, thereby improving the flexibility of video evaluation.

[0048] According to embodiments of this disclosure, after performing intent recognition on the evaluation request and obtaining the intent recognition result, the video evaluation method may further include: in the case that the content information does not match the intent recognition result, determining an evaluation result that represents the content to be evaluated in the video to be evaluated as not matching the evaluation intent.

[0049] For example, the video to be evaluated might be an animation or a silent segment without audio. The intent recognition result is "Evaluate the consistency between audio and video content." Therefore, it can be determined that the content information does not match the intent recognition result, leading to an evaluation result that the content to be evaluated does not match the evaluation intent.

[0050] According to embodiments of this disclosure, by determining the evaluation result where the content to be evaluated in the video does not match the evaluation intention when the content information and the recognition result do not match, invalid evaluations can be avoided and evaluation efficiency can be improved.

[0051] Figure 3 A flowchart illustrating an artificial intelligence-based video evaluation method according to another embodiment of the present disclosure is shown.

[0052] like Figure 3 As shown, in response to receiving evaluation request 301, the video to be evaluated 302 is determined. Evaluation request 301 may carry the storage address of the video to be evaluated 302, and the video to be evaluated 302 can be obtained based on the storage address. Intent recognition is performed on evaluation request 301 to obtain intent recognition result 303. Based on intent recognition result 303, initial evaluation index 304 is determined. If it is determined that initial evaluation index 304 belongs to a predetermined evaluation index type 305, evaluation index 306 is obtained based on initial evaluation index 304. If evaluation index 306 includes a temporal evaluation index, the video to be evaluated 302 is analyzed to obtain content information 307 describing the video to be evaluated. Based on the content information 307 and evaluation index 306, evaluation prompt information 308 is generated. The evaluation prompt information 308 and the video to be evaluated 302 are input into the multimodal large model M309 to obtain a temporal evaluation result 310 for the video to be evaluated.

[0053] According to embodiments of this disclosure, for example, Figure 2 The operation S220 shown analyzes the video to be evaluated and obtains content information describing the video to be evaluated, which may include: the indicator type based on the time series evaluation indicator, and analyzes the video to be evaluated to obtain content information that matches the indicator type.

[0054] Timing evaluation metrics can include physical law metrics, temporal correlation metrics, frame skipping, and other similar metrics. Physical law metrics assess whether the motion of objects and scene changes in the video under evaluation follow real-world physical rules, such as gravity, collision, and light reflection. Temporal correlation metrics assess whether the causal relationships and temporal order of events in the video under evaluation are logically consistent. Frame skipping and frame locking are metrics used to evaluate the stability of the video's frame rate.

[0055] For example, the evaluation metrics may include one type of time-series evaluation metrics or multiple types of time-series evaluation metrics. According to the different types of time-series evaluation metrics, corresponding matching content information can be generated respectively.

[0056] According to embodiments of this disclosure, different content information is determined by using different types of time-series evaluation metrics, thereby further improving the flexibility of video evaluation.

[0057] According to embodiments of this disclosure, analyzing the video to be evaluated based on the index type of the time-series evaluation index to obtain content information matching the index type may include: when the time-series evaluation index includes physical law indicators, performing object recognition on the video to be evaluated to determine video objects. Video objects include at least one of foreground objects and background objects. Based on the video objects, performing scene recognition on the video to be evaluated to determine the scene recognition result. Based on the scene recognition result and the video objects, obtaining content information matching the physical law indicators.

[0058] By detecting physical laws, the physical plausibility of video content can be quantified, and whether the video "violates common sense." For example, detecting whether an object is not placed directly on a table, but has a gap between it and the table, thus violating the laws of gravity.

[0059] Determining video objects through object recognition in the video to be evaluated may include: breaking down the video into a continuous sequence of video frames; and then performing object recognition on the video frame sequence to identify the video objects.

[0060] For example, for each video frame, object recognition can be performed using methods such as object detection and image segmentation to identify the video object. For adjacent video frames, object tracking can be used to identify the same object in different video frames to determine the video object. The video object can be a person, animal, virtual object, or other object in the video.

[0061] For example, a large model can be used to process the video to be evaluated based on video objects to obtain scene recognition results describing the video. This disclosure is not limited thereto; a pre-trained classification model can also be used to process the video to be evaluated based on video objects to obtain scene recognition results describing the video.

[0062] Scene recognition results can include the environment in which the video object is located, such as indoors, streets, or natural environments. The content information obtained based on the scene recognition results and the video object can represent descriptive information about the video object's environment.

[0063] For example, content information could include a video subject being in a live stream room, sitting on a stool with a table in front of them and items on the table. Another example is a video subject running on the street.

[0064] According to embodiments of this disclosure, when the temporal evaluation metrics include physical law metrics, the content information obtained based on the video object and scene recognition results includes descriptive information about the video object's environment. This makes it easier for subsequent evaluations of the video's physical law metrics based on the content information to be targeted and instructive, thereby improving the comprehensiveness and effectiveness of the evaluation.

[0065] According to embodiments of this disclosure, analyzing the video to be evaluated based on the indicator type of the time-series evaluation metric to obtain content information matching the indicator type may further include: when the time-series evaluation metric includes a time-series correlation indicator, performing object recognition on the video to be evaluated to determine foreground objects. Based on the foreground objects, determining content information matching the time-series correlation indicator.

[0066] Foreground objects refer to dynamic subjects in an image that are closest to the observer, most prominent, or directly related to the subject, such as a moving car or a person speaking in a video.

[0067] For example, foreground objects in video frames can be extracted using a semantic segmentation model or an instance segmentation model. The position of the foreground object within the video frame sequence is then tracked to determine its motion information. Based on this motion information, content information is determined.

[0068] According to embodiments of this disclosure, since the content information determined by the foreground object includes the motion information of the foreground object, the motion information can be used to analyze whether there are problems such as objects disappearing or some actions being discontinuous, as well as to evaluate problems such as frame skipping and frame freezing.

[0069] According to embodiments of this disclosure, for example, Figure 2 The operation S230 shown, which generates evaluation prompt information based on content information and evaluation metrics, may include: extracting target phrases from the content information; adding the target phrases and evaluation metrics to a prompt template that matches the metric type of the time-series evaluation metrics; and generating evaluation prompt information.

[0070] For example, key entities, events, and attribute information can be extracted from content information to form target phrases, the target phrases can be associated with time-series evaluation indicators, and then populated into a prompt template to generate structured evaluation prompt information.

[0071] It can also refine content information and time-series evaluation metrics to generate evaluation prompts. For example, content information and time-series evaluation metrics can be input into a large model, which can then refine the content information and time-series evaluation metrics to generate evaluation prompts.

[0072] According to embodiments of this disclosure, by extracting target phrases and filling the target phrases and temporal evaluation metrics into a prompt template to generate evaluation prompt information, the standardization of the evaluation prompt information input into the multimodal large model can be guaranteed each time, thereby improving the stability and effectiveness of instructing the multimodal large model to perform video quality evaluation.

[0073] According to embodiments of this disclosure, for example, Figure 2 The operation S240 shown, which inputs the evaluation prompt information and the video to be evaluated into the multimodal large model, can obtain the time-series evaluation result for the video to be evaluated, including: extracting a key video frame sequence from the video to be evaluated and marking the time identifier of each key video frame in the key video frame sequence; inputting the key video frame sequence marked with time identifiers and the evaluation prompt information into the multimodal large model to obtain the time-series evaluation result.

[0074] Key video frame sequences can be sequences of representative video frames extracted from the video to be evaluated that reflect changes in video content or potential quality issues.

[0075] For example, key video frame sequences can be determined based on content differences between video frames, such as extracting key video frame sequences with differences in color or texture. Key video frame sequences can also be determined based on the motion state of foreground objects, such as prioritizing key video frame sequences with sudden changes in motion or that are stationary. Alternatively, key video frame sequences can be determined by combining content differences and foreground motion; for example, first extracting candidate key video frame sequences based on content changes, and then filtering from these candidate sequences for key video frame sequences with sudden changes in motion or that are stationary.

[0076] For example, the frame number of a video frame can be used as the time identifier for each key video frame, or the timestamp of a video frame can be used as the time identifier for each key video frame.

[0077] According to embodiments of this disclosure, by extracting key video frame sequences from the video to be evaluated and adding time markers to each key video frame, the temporal relationship of the video frame sequences is reflected, thereby improving the evaluation capability and accuracy of multimodal large models.

[0078] Figure 4 A flowchart illustrating the determination of content information of a video to be evaluated according to an embodiment of this disclosure is shown schematically.

[0079] like Figure 4As shown, in response to receiving the evaluation request 401, the video to be evaluated 402 and the evaluation index 403 of the video to be evaluated are determined. If the evaluation index 403 includes a temporal evaluation index 404, and the temporal evaluation index 404 includes a physical law index 405, object recognition is performed on the video to be evaluated 402 to determine the video object 407. Based on the video object 407, scene recognition is performed on the video to be evaluated to determine the scene recognition result 408. Based on the video object 407 and the scene recognition result 408, the content information 410 of the video to be evaluated is obtained.

[0080] When the temporal evaluation index 404 includes the temporal correlation index 406, object identification is performed on the video to be evaluated to determine the foreground object 409. Based on the foreground object 409, the content information 410 of the video to be evaluated is determined.

[0081] like Figure 4 As shown, multimodal large models can be used to evaluate physical properties or temporal correlation indicators of the video being evaluated. However, this is not the only approach. Multimodal large models can also be used to evaluate both physical properties and temporal correlation indicators of the video being evaluated simultaneously.

[0082] Specifically, target phrases matching the physical law indicators can be extracted from content information that matches the physical law indicators. These target phrases and the physical law indicators themselves are then added to a prompt template that matches the physical law indicators, generating evaluation prompt information for the physical law indicators.

[0083] Target phrases matching time-series correlation indicators can be extracted from content information that matches these indicators. These target phrases and the time-series correlation indicators themselves are then added to a prompt template that matches the time-series correlation indicators, generating evaluation prompts for these indicators.

[0084] The evaluation prompts for physical law indicators and temporal correlation indicators can be input into the multimodal large model simultaneously or sequentially to obtain the temporal evaluation results for the physical law indicators and the temporal correlation indicators. Based on the temporal evaluation results for the physical law indicators and the temporal correlation indicators, the temporal evaluation results for the video to be evaluated are obtained.

[0085] Both the time-series evaluation results of physical law indicators and the time-series evaluation results of time-series correlation indicators can be used as time-series evaluation results. If either the time-series evaluation results of physical law indicators or the time-series evaluation results of time-series correlation indicators indicate that the video to be evaluated does not meet the release conditions, feedback will be given indicating that the video to be evaluated does not meet the release conditions, so as to perform video repair on the video to be evaluated.

[0086] For example, physical law indicators are used to quantify the physical rationality of the video content being evaluated. The temporal evaluation results of these physical law indicators can characterize whether the video content is physically reasonable and whether it violates common sense. If the temporal evaluation results of the physical law indicators indicate that the video content is physically unreasonable, then the temporal evaluation results for the video indicate that it does not meet the release conditions. Video repair can be performed based on the temporal evaluation results of the physical law indicators.

[0087] For example, a temporal correlation index is used to assess whether the actions performed by foreground objects in a video are logically consistent in time. The temporal evaluation result of the temporal correlation index can characterize whether the actions performed by the foreground objects in the video are logically consistent in time. If the temporal evaluation result of the temporal correlation index indicates that the actions performed by the foreground objects in the video are not logically consistent in time, then the temporal evaluation result for the video indicates that the video does not meet the release conditions. Video repair can be performed on the video to be evaluated based on the temporal evaluation result of the temporal correlation index.

[0088] If both the time-series evaluation results of the physical law indicators and the time-series evaluation results of the time-series correlation indicators indicate that the video to be evaluated meets the release conditions, then the time-series evaluation results for the video to be evaluated indicate that the video to be evaluated meets the release conditions, and the video can be released after evaluation.

[0089] By combining physical law indicators and time-series correlation indicators in the time-series evaluation index, the analysis of the video to be evaluated can be covered in multiple dimensions, thereby further improving the comprehensiveness of the evaluation.

[0090] According to embodiments of this disclosure, after determining the video to be evaluated and its evaluation metrics in response to receiving an evaluation request, the video evaluation method may further include: when the evaluation metrics include image quality evaluation metrics, performing image quality evaluation on multiple video frames in the video to be evaluated respectively to obtain multiple evaluation values ​​characterizing image quality; and performing a comprehensive quality evaluation on the video to be evaluated based on the multiple evaluation values ​​to obtain a quality evaluation result characterizing video quality. The quality evaluation result includes at least one of the following data types: fluctuation coefficient, stability coefficient, mean, and variance.

[0091] For example, the image quality of each video frame can be evaluated from various dimensions such as the blur level, noise level, compression distortion level, generative adversarial network artifact coefficient, sharpness, contrast, and color saturation of the video to be evaluated, so as to obtain multiple evaluation values ​​for multiple video frames for the same dimension.

[0092] It can comprehensively evaluate multiple video frames. For example, by analyzing one or more of the fluctuation coefficient, stability coefficient, mean, variance, etc. of multiple evaluation values ​​of a video frame sequence, the quality of the video to be evaluated can be assessed in various dimensions, and the quality evaluation results representing the video quality in various dimensions can be obtained.

[0093] For example, a quality score for each dimension can be obtained based on the quality assessment results for each dimension, and the quality scores of multiple dimensions can be weighted and fused to obtain the quality assessment result of the video to be evaluated.

[0094] According to embodiments of this disclosure, by performing image quality assessment on multiple video frames in the video to be evaluated separately, and then combining the assessment with temporal evaluation metrics, the video quality can be evaluated more comprehensively, thereby improving the comprehensiveness of the video evaluation metrics.

[0095] According to embodiments of this disclosure, after determining the video to be evaluated and its evaluation metrics in response to receiving an evaluation request, the video evaluation method may further include: if the evaluation metrics include audio evaluation metrics, performing text conversion on the audio of the video to be evaluated to obtain audio text; and performing a consistency evaluation on the audio text and the video text of the video to be evaluated in chronological order to obtain an audio evaluation result.

[0096] For example, the timestamps of audio text and video text can be compared, and the audio evaluation result can be obtained based on the time difference between the audio text and video text.

[0097] It can also extract semantic information from audio text, evaluate the consistency between semantic information and video object actions, and obtain audio evaluation results.

[0098] According to embodiments of this disclosure, by combining temporal evaluation metrics, image quality evaluation metrics, and audio evaluation metrics, the evaluation of the video to be evaluated can be covered in multiple dimensions, further improving the comprehensiveness of the evaluation.

[0099] Figure 5 A flowchart illustrating an artificial intelligence-based video evaluation method according to another embodiment of the present disclosure is shown.

[0100] In response to receiving the evaluation request 501, the video to be evaluated 502 and the evaluation metrics 503 for the video to be evaluated are determined. If the evaluation metrics 503 include a temporal evaluation metric 504, the video to be evaluated is analyzed to obtain content information 507 describing the video to be evaluated. Based on the content information 507 and the temporal evaluation metric 504, evaluation prompt information 510 is generated. The evaluation prompt information 510 and the video to be evaluated 502 are input into the multimodal large model M513 to obtain the temporal evaluation result 514 for the video to be evaluated.

[0101] When evaluation metric 503 includes image quality evaluation metric 505, image quality is evaluated for each of the multiple video frames in the video to be evaluated 502, resulting in multiple evaluation values ​​508 characterizing image quality. Based on the multiple evaluation values ​​508, a comprehensive quality evaluation is performed on the video to be evaluated, resulting in a quality evaluation result 511 characterizing video quality.

[0102] When evaluation metric 503 includes audio evaluation metric 506, the audio of the video 502 to be evaluated is converted into text, resulting in audio text 509. Following the time sequence, a consistency evaluation is performed on the audio text 509 and the video text of the video to be evaluated, yielding the audio evaluation result 512.

[0103] Figure 6 A block diagram of an artificial intelligence-based video evaluation apparatus according to an embodiment of the present disclosure is shown schematically.

[0104] like Figure 6 As shown, the AI-based video evaluation device 600 includes a determination module 610, an analysis module 620, a generation module 630, and an evaluation module 640.

[0105] The determination module 610 is used to determine the video to be evaluated and the evaluation metrics of the video to be evaluated in response to receiving an evaluation request.

[0106] The analysis module 620 is used to analyze the video to be evaluated when the evaluation indicators include time-series evaluation indicators, and to obtain content information describing the video to be evaluated.

[0107] The generation module 630 is used to generate evaluation prompts based on content information and evaluation indicators.

[0108] The evaluation module 640 is used to input the evaluation prompt information and the video to be evaluated into the multimodal large model to obtain the time-series evaluation results for the video to be evaluated.

[0109] According to embodiments of this disclosure, the analysis module 620 includes an analysis submodule.

[0110] The analysis submodule is used to analyze the video to be evaluated based on the indicator type of the time-series evaluation indicators, and obtain content information that matches the indicator type.

[0111] According to embodiments of this disclosure, the analysis submodule includes an object recognition unit, a scene recognition unit, and a first content determination unit.

[0112] The object recognition unit is used to identify objects in the video to be evaluated when the temporal evaluation index includes physical law indexes, and to determine the video objects, which include at least one of foreground objects and background objects.

[0113] The scene recognition unit is used to perform scene recognition on the video to be evaluated based on video objects and determine the scene recognition result.

[0114] The first content determination unit is used to obtain content information that matches physical law indicators based on scene recognition results and video objects.

[0115] According to embodiments of this disclosure, the analysis submodule includes a foreground object recognition unit and a second content determination unit.

[0116] The foreground object recognition unit is used to identify foreground objects in the video to be evaluated, when the temporal evaluation metrics include temporal correlation metrics.

[0117] The second content determination unit is used to determine content information that matches the time-series correlation indicators based on the foreground object.

[0118] According to embodiments of this disclosure, the generation module 630 includes an extraction submodule, a prompt information generation submodule, and a prompt information generation submodule.

[0119] The extraction submodule is used to extract target phrases from content information.

[0120] The prompt message generation submodule is used to add target phrases and evaluation indicators to prompt templates that match the indicator types of time-series evaluation indicators, thereby generating evaluation prompt messages.

[0121] According to embodiments of this disclosure, the evaluation module 640 includes a tagging submodule and an evaluation submodule.

[0122] The tagging submodule is used to extract key video frame sequences from the video to be evaluated and tag each key video frame in the key video frame sequence with a time identifier.

[0123] The evaluation submodule is used to input the key video frame sequence marked with time markers and evaluation prompt information into the multimodal large model to obtain the time-series evaluation results.

[0124] According to embodiments of this disclosure, the AI-based video evaluation device 600 further includes an evaluation value module and an image quality evaluation module.

[0125] The evaluation value module is used to evaluate the image quality of multiple video frames in the video to be evaluated, when the evaluation metrics include image quality evaluation metrics, and obtain multiple evaluation values ​​that characterize the image quality.

[0126] The image quality assessment module is used to perform a comprehensive quality assessment of the video to be assessed based on multiple assessment values, and obtain a quality assessment result that characterizes the video quality. The quality assessment result includes at least one of the following data types: fluctuation coefficient, stability coefficient, mean, and variance.

[0127] According to embodiments of this disclosure, the AI-based video evaluation device 600 further includes an audio conversion module and an audio evaluation module.

[0128] The audio conversion module is used to convert the audio of the video to be evaluated into text when the evaluation metrics include audio evaluation metrics, thus obtaining audio text.

[0129] The audio evaluation module is used to perform consistency evaluation on the audio text and the video text of the video to be evaluated in chronological order, and obtain the audio evaluation result.

[0130] According to embodiments of this disclosure, the determination module includes an intent recognition submodule, an initial evaluation metric determination submodule, and an evaluation metric determination submodule.

[0131] The intent recognition submodule is used to recognize the intent of the evaluation request and obtain the intent recognition result.

[0132] The initial evaluation metric determination submodule is used to determine the initial evaluation metrics based on the intent recognition results.

[0133] The evaluation indicator determination submodule is used to determine the initial evaluation indicator as the evaluation indicator when the initial evaluation indicator belongs to a predetermined evaluation indicator type.

[0134] According to embodiments of this disclosure, the AI-based video evaluation device 600 further includes an evaluation result determination module.

[0135] The evaluation result determination module is used to determine the evaluation result when the content information and the intent recognition result do not match, indicating that the content to be evaluated in the video to be evaluated does not match the evaluation intent.

[0136] It should be noted that the video evaluation device based on artificial intelligence in the embodiments of this disclosure corresponds to the video evaluation method based on artificial intelligence in the embodiments of this disclosure. For a detailed description of the video evaluation device based on artificial intelligence, please refer to the video evaluation method based on artificial intelligence, which will not be repeated here.

[0137] Figure 7 A schematic block diagram of an artificial intelligence agent according to an embodiment of the present disclosure is shown.

[0138] In embodiments of this disclosure, such as Figure 7As shown, the AI ​​agent 700 may include an input module 710, a processing module 720, and an output module 730.

[0139] Input module 710 is used to receive input information.

[0140] The processing module 720 is used to determine the target task based on the input information received by the input module, determine the multimodal large model based on the target task, and execute the artificial intelligence-based video evaluation method provided according to the embodiments of this disclosure by calling the multimodal large model.

[0141] Output module 730 is used to output the output information obtained by the processing module.

[0142] According to embodiments of this disclosure, the input module 710 is responsible for receiving or sensing information such as queries, requests, instructions, signals, or data from the outside world (e.g., users or the external environment), and converting it into a format that the AI ​​agent 700 can understand and process. The input module 710 is the primary link for the AI ​​agent 700 to interact with the outside world, enabling the AI ​​agent 700 to efficiently and accurately obtain necessary "sensory" information from the outside world and respond to this information.

[0143] In the example, input module 710 can input the evaluation request and the video to be evaluated, as described above.

[0144] In the example, processing module 720 is the core support for the AI ​​agent 700's ability to handle complex tasks. Processing module 720 can execute the AI-based video evaluation method described above.

[0145] In the example, the performance of the processing module 720 is closely related to the multimodal large model on which the AI ​​agent 700 is based. To fully leverage the capabilities of the multimodal large model, the internal structure of the processing module 720 can be designed to be highly configurable and scalable to handle various types of tasks and requirements in real-world scenarios.

[0146] In the example, after the AI ​​agent 700 receives the evaluation request, the processing module 720 can process the evaluation prompt information and the video to be evaluated, obtain the time-series evaluation result for the video to be evaluated, and pass it to the output module 730.

[0147] Understandably, while multimodal models possess excellent language understanding and generation capabilities, like humans, their ability to solve tasks is limited without the aid of any tools. However, once the AI ​​agent 700 is given the ability to invoke tools, it can perform tasks such as using a calculator to complete mathematical calculations, using Python to perform data analysis, and using a search engine to generate weather forecasts.

[0148] In the example, output module 730 can output timing evaluation results.

[0149] The AI ​​agent 700 according to embodiments of this disclosure can simply and effectively improve the level of intelligence, as well as enhance flexibility and versatility.

[0150] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0151] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0152] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described above.

[0153] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0154] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0155] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0156] Multiple components in electronic device 800 are connected to input / output (I / O) interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0157] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as AI-based video evaluation methods. For example, in some embodiments, the AI-based video evaluation method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the AI-based video evaluation method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform AI-based video evaluation methods by any other suitable means (e.g., by means of firmware).

[0158] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0159] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0160] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0161] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0162] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0163] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0164] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0165] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An artificial intelligence-based video evaluation method, comprising: In response to receiving an evaluation request, determine the video to be evaluated and the evaluation metrics for the video to be evaluated; When the evaluation indicators include time-series evaluation indicators, the video to be evaluated is analyzed to obtain content information describing the video to be evaluated. Based on the content information and the evaluation indicators, an evaluation prompt message is generated; as well as The evaluation prompts and the video to be evaluated are input into a multimodal large model to obtain the time-series evaluation results for the video to be evaluated.

2. The method according to claim 1, wherein, The analysis of the video to be evaluated to obtain content information describing the video to be evaluated includes: Based on the indicator type of the time-series evaluation indicator, the video to be evaluated is analyzed to obtain content information that matches the indicator type.

3. The method according to claim 2, wherein, The method of analyzing the video to be evaluated based on the indicator type of the time-series evaluation index to obtain content information matching the indicator type includes: When the time-series evaluation index includes physical law indexes, object recognition is performed on the video to be evaluated to determine the video object, wherein the video object includes at least one of foreground object and background object; Based on the video object, scene recognition is performed on the video to be evaluated to determine the scene recognition result; and Based on the scene recognition results and the video object, content information matching the physical law indicators is obtained.

4. The method according to claim 2 or 3, wherein, The method of analyzing the video to be evaluated based on the indicator type of the time-series evaluation index to obtain content information matching the indicator type includes: When the temporal evaluation metrics include temporal correlation metrics, object recognition is performed on the video to be evaluated to determine foreground objects; and Based on the foreground object, determine the content information that matches the time-series correlation index.

5. The method according to any one of claims 1 to 4, wherein, The step of generating evaluation prompt information based on the content information and the evaluation indicators includes: Extracting target phrases from the content information; and The target phrase and the evaluation index are added to a prompt template that matches the index type of the time-series evaluation index to generate the evaluation prompt information.

6. The method according to any one of claims 1 to 5, wherein, The step of inputting the evaluation prompt information and the video to be evaluated into a multimodal large model to obtain the temporal evaluation result for the video to be evaluated includes: Extract a sequence of key video frames from the video to be evaluated, and label each key video frame in the sequence with a time identifier; and The key video frame sequence marked with time identifiers and the evaluation prompt information are input into the multimodal large model to obtain the temporal evaluation result.

7. The method according to any one of claims 1 to 6, further comprising: When the evaluation index includes an image quality evaluation index, the image quality of multiple video frames in the video to be evaluated is evaluated respectively to obtain multiple evaluation values ​​characterizing the image quality. as well as Based on multiple evaluation values, a comprehensive quality assessment is performed on the video to be evaluated to obtain a quality assessment result characterizing the video quality. The quality assessment result includes at least one of the following data types: fluctuation coefficient, stability coefficient, mean, and variance.

8. The method according to any one of claims 1 to 7, further comprising: When the evaluation metrics include audio evaluation metrics, the audio of the video to be evaluated is converted into text to obtain audio text; as well as The consistency between the audio text and the video text of the video to be evaluated is evaluated according to the time sequence to obtain the audio evaluation result.

9. The method according to any one of claims 1 to 8, wherein, Based on the evaluation request for the video to be evaluated, the evaluation metrics for the video to be evaluated are determined, including: The intent of the evaluation request is identified to obtain the intent identification result; Based on the intent recognition results, initial evaluation metrics are determined; If the initial evaluation indicator belongs to a predetermined evaluation indicator type, then the initial evaluation indicator is determined as the evaluation indicator.

10. The method of claim 9, further comprising: If the content information does not match the intent recognition result, an evaluation result is determined that indicates the content to be evaluated in the video to be evaluated does not match the evaluation intent.

11. A video evaluation device based on artificial intelligence, comprising: The determination module is used to determine the video to be evaluated and the evaluation index of the video to be evaluated in response to receiving an evaluation request; The analysis module is used to analyze the video to be evaluated when the evaluation indicators include time-series evaluation indicators, and to obtain content information describing the video to be evaluated. The generation module is used to generate evaluation prompt information based on the content information and the evaluation indicators; as well as The evaluation module is used to input the evaluation prompt information and the video to be evaluated into the multimodal large model to obtain the time-series evaluation result for the video to be evaluated.

12. An intelligent agent of artificial intelligence, comprising: The input module is used to receive input information; The processing module is configured to determine a target task based on the input information received by the input module, determine a multimodal large model based on the target task, and execute the method of any one of claims 1 to 10 by calling the multimodal large model to obtain output information; An output module is used to output the output information obtained by the processing module.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 10.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images