Feedback-based image quality diagnosis method and system, medium and equipment

Abstract

The invention relates to the field of computer vision, and discloses a feedback-based image quality diagnosis method and system, a medium and equipment, and the method comprises the steps: carrying out the training or fine tuning of a multi-modal large model, and obtaining a trained multi-modal large model; performing scene similarity or image similarity query on the to-be-diagnosed image in the dynamic quality diagnosis expert database, if the query result is similar, fusing the query result into the standard Prompt to generate an enhanced Prompt, asking questions for the multi-modal large model by the enhanced Prompt, outputting scene description, diagnosis scores and evaluation reasons, and obtaining a diagnosis result of the to-be-diagnosed image. And a diagnosis result of detailed description of image quality; if the query result is dissimilar, questioning the multi-modal large model by a standard Prompt, and outputting a packet diagnosis result; and determining whether the diagnosis result is stored in the dynamic quality diagnosis expert database or not according to user requirements so as to dynamically update the dynamic quality diagnosis expert database. According to the invention, the diagnosis processing result can be aligned with the cognition of the user, and the accuracy of diagnosis is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a feedback-based image quality diagnosis method, system, medium and device. Background Art

[0002] With the development of Skynet system and security system, the number of surveillance cameras is increasing, and video surveillance has penetrated into every field of people's daily life. In the event of camera failure, signal interference, or video quality degradation, the image quality collected by the camera will degrade and cannot complete the real-time monitoring task. How to diagnose the quality of a large number of cameras in the system is a problem that needs to be solved. It is unrealistic to rely solely on manual inspections, opening these tens of thousands of cameras one by one, and manually judging whether the cameras are abnormal.

[0003] The traditional image quality diagnosis method is to design different algorithms for different types of image quality degradation, extract features by artificial design, and then determine the fault type of the image. This diagnostic method requires designing different features for different fault types, but due to the inappropriate extracted features, the accuracy of diagnosis is not high. In the existing literature, comprehensive video quality abnormality data is collected to classify image quality into types such as signal loss, clarity abnormality, brightness abnormality, etc., build a deep learning model framework, read the adjusted deep learning model, and use the deep learning model to classify the input image. There is also a document that proposes a method for evaluating image quality, first obtaining an input image to be processed; the input image is combined with prompt information and input into a multimodal large model to obtain an image quality evaluation score of the input image; and the evaluation reason corresponding to the image quality evaluation score is output; wherein the multimodal large model is obtained by fine-tuning the multimodal large model using an image quality evaluation training data set. However, these methods generally require collecting sample data, calibrating the data, and then training the parameters of the deep learning model, or fine-tuning the large model. After training or fine-tuning, the parameters of these systems are usually fixed and cannot be adjusted according to the user's specific needs or sensitivities in actual applications.

[0004] However, in actual use, due to differences in camera deployment time at different sites or different users’ requirements for camera image quality, the diagnostic system’s diagnostic results must be able to follow the user’s judgment results, that is, the diagnostic system’s processing results must be aligned with the user’s judgment results. Or due to different camera deployment scenarios, the camera’s image quality may be reduced under the influence of ambient light, and users do not want to raise an alarm at this time.

[0005] Therefore, existing systems often fail to meet these personalized needs. In addition, existing systems are unable to modify the generation or diagnosis results of new images based on the user's processing results of previous images, resulting in a poor user experience. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide a feedback-based image quality diagnosis method, system, medium and device, which can align the diagnostic processing results with the user's cognition and improve the accuracy of diagnosis.

[0007] To achieve the above-mentioned purpose, in the first aspect, the technical solution adopted by the present invention is: a feedback-based image quality diagnosis method, which includes: using on-site rail transit data to train or fine-tune a multimodal large model to obtain a trained multimodal large model; when executing a new diagnostic task, performing a scene similarity or image similarity query on the image to be diagnosed in the dynamic quality diagnosis expert database, and if the query result is similar, the query result is integrated into the standard prompt to generate an enhanced prompt, and the enhanced prompt is used to ask questions to the multimodal large model, and output a diagnostic result including a scene description, a diagnostic score and an evaluation reason, and a detailed description of the image quality; if the query result is not similar, the standard prompt is used to ask questions to the multimodal large model, and output a diagnostic result including a scene description, a diagnostic score and an evaluation reason, and a detailed description of the image quality; determine whether the diagnostic result is stored in the dynamic quality diagnosis expert database according to user needs to dynamically update the dynamic quality diagnosis expert database.

[0008] Furthermore, the multimodal large model is trained or fine-tuned using field samples, including:

[0009] Construct image quality diagnosis sample sets, which are divided into single-image and multi-image sample sets;

[0010] The single-image and multi-image sample sets are used as input data for the multimodal large model, and the multimodal large model is trained or fine-tuned until the set conditions are met to obtain a trained large model.

[0011] Furthermore, a standard prompt is generated based on a preset structured question template, wherein the preset structured question template is a question about image quality, including video loss, clarity, contrast, noise, snow, brightness, color, and occlusion of the image.

[0012] Furthermore, the image to be diagnosed is queried for scene similarity or image similarity in the dynamic quality diagnosis database. If the query result is similar, the query result is integrated into the standard prompt to generate an enhanced prompt, which asks questions to the multimodal large model, including:

[0013] Compare the similarity between the image to be diagnosed and the images in the dynamic quality diagnosis expert database, and find the top-K similar images in the dynamic quality diagnosis expert database according to the set threshold, where K is a preset parameter;

[0014] Based on the user's standard prompt, similar images found, and the user's processing results on images of similar scenes, an enhanced prompt is generated, and the enhanced prompt asks questions to the large model;

[0015] The image to be diagnosed and the enhanced prompt are input into the multimodal large model to generate content and obtain the diagnosis result.

[0016] Furthermore, the image to be diagnosed is compared with the images in the quality diagnosis expert database for similarity, including:

[0017] The image to be diagnosed is encoded into a vector using an image encoder, and scene similarity comparison or image similarity comparison is performed with images in a dynamic quality diagnosis database.

[0018] Further, the multimodal large model is questioned, and the output includes a scene description, a diagnostic score and evaluation reasons, and a detailed description of the image quality, including:

[0019] Describe the scene of the image to be diagnosed, and describe the video loss, clarity, contrast, noise, snow, brightness, color, and occlusion of the image, score it, and give reasons for the evaluation;

[0020] The overall quality score of the image is obtained by combining the levels and index scores of each fault.

[0021] Further, whether the diagnosis result is stored in the dynamic quality diagnosis expert database is determined according to user needs, including:

[0022] If the user approves the diagnosis result of the large model, the image processing ends;

[0023] If the user does not agree with the diagnosis result of the large model, or the result is incorrect or biased, the scene description, evaluation score and quality description of the image will be modified;

[0024] The image, the modified scene description, the modified image score, and the modified image quality detailed description are added to the dynamic quality diagnosis expert database, the image, scene description, evaluation score and quality description are stored, and the dynamic quality diagnosis expert database is updated.

[0025] In the second aspect, the technical solution adopted by the present invention is: a feedback-based image quality diagnosis system, which includes: a training module, which uses on-site rail transit data to train or fine-tune the multimodal large model to obtain a trained multimodal large model; an enhanced prompt module, which performs scene similarity or image similarity query on the image to be diagnosed in the dynamic quality diagnosis expert database. If the query result is similar, the query result is integrated into the standard prompt to generate an enhanced prompt, and the enhanced prompt asks questions to the multimodal large model, and outputs a diagnosis result including a scene description, a diagnosis score and an evaluation reason, and a detailed description of the image quality; a standard prompt module, if the query result is not similar, the standard prompt asks questions to the multimodal large model, and outputs a diagnosis result including a scene description, a diagnosis score and an evaluation reason, and a detailed description of the image quality; a database dynamic update module, which determines whether the diagnosis result is stored in the dynamic quality diagnosis expert database according to user needs, so as to dynamically update the dynamic quality diagnosis expert database.

[0026] In a third aspect, the technical solution adopted by the present invention is: a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by a computing device, the computing device executes any one of the above methods.

[0027] In a fourth aspect, the technical solution adopted by the present invention is: a computing device, comprising: one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0028] The present invention adopts the above technical solution, which has the following advantages:

[0029] The present invention uses field samples to fine-tune the large model, inputs the image to be diagnosed into the large model for diagnosis, and gives a scene description, a diagnostic score, and an evaluation reason; the user processes the scene description, diagnostic score, or evaluation reason. If the user believes that the result is incorrect or biased, the generated result is modified, and the image and the processing result are stored in the quality diagnosis expert database; when performing subsequent diagnostic tasks, the new diagnostic image will refer to the processing results of similar images in the quality diagnosis database to generate a scene description, a diagnostic score, and an evaluation reason. Since the user participates in the evaluation of the diagnostic results, the system will become more and more aligned with the user's cognition over time, thereby improving the accuracy of the diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of the image quality diagnosis method based on feedback in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] When the existing image quality diagnosis system is applied to multiple stations in the railway or other industries, the image quality will vary due to the different construction times and camera installation times of each station. In addition, for different locations in the station, such as platforms, ticket offices, highway-rail crossings, tunnels, etc., these different locations have different image evaluation standards. If only a set of fixed parameter models is used, the diagnostic results will be unsatisfactory. Although using sample training or fine-tuning a large model can solve some problems, when this model is applied to other stations, it will still result in different diagnostic results and user experience. Therefore, the present invention proposes a feedback-based image quality diagnosis method, system, medium and device, firstly, the large model is fine-tuned using field samples; the image to be diagnosed is input into the large model for diagnosis, and a scene description, a diagnosis score and an evaluation reason are given; the user processes the scene description, the diagnosis score or the evaluation reason, and if the user believes that the result is incorrect or biased, the generated result is modified, and the image and the processing result are stored in the quality diagnosis expert database; when performing subsequent diagnostic tasks, the new diagnostic image will refer to the processing results of similar images in the quality diagnosis database to generate a scene description, a diagnosis score and an evaluation reason. The present invention can meet the needs of users more and more as the system is used.

[0032] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0033] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0034] In one embodiment of the present invention, a feedback-based image quality diagnosis method is provided. In this embodiment, Figure 1 As shown, the method comprises the following steps:

[0035] 1) Use on-site rail transit data to train or fine-tune the multimodal large model to obtain a trained multimodal large model;

[0036] 2) When executing a new diagnostic task, the image to be diagnosed is queried for scene similarity or image similarity in the dynamic quality diagnosis expert database. If the query result is similar, the query result is integrated into the standard prompt to generate an enhanced prompt. The enhanced prompt asks questions to the multimodal large model and outputs a diagnostic result including a scene description, a diagnostic score and evaluation reasons, and a detailed description of the image quality;

[0037] 3) If the query result is not similar, the multimodal large model is asked questions by the standard prompt, and the output includes a scene description, a diagnosis score and evaluation reasons, and a detailed description of the image quality.

[0038] 4) Determine whether the diagnosis result should be stored in the dynamic quality diagnosis expert database according to user needs, so as to dynamically update the dynamic quality diagnosis expert database.

[0039] In the above step 1), the multimodal large model is trained or fine-tuned using field samples, including the following steps:

[0040] 1.1) Construct an image quality diagnosis sample set, which is divided into single-image and multi-image sample sets;

[0041] In this embodiment, the single image sample format is:

[0042]

[0043] The fields are explained as follows:

[0044] image:

[0045] Type: string;

[0046] Description: Path to an image file. This field specifies the location of the image associated with the conversation.

[0047] Example value: "path / to / image.jpg";

[0048] conversations:

[0049] type: list;

[0050] Description: Contains every message in the conversation. Each message is a dictionary containing the following key-value pairs:

[0051] from: The source of the message, which can be "human" or "answer".

[0052] value: The specific content of the message. If the message contains , it means that the message references an image.

[0053] The multi-image sample format is as follows:

[0054]

[0055]

[0056] The fields are explained as follows:

[0057] image:

[0058] Type: string array;

[0059] Description: A list containing paths to multiple image files. These images may be referenced in different parts of the dialog.

[0060] Example value: ["path / to / image1.jpg","path / to / image2.jpg","path / to / image3.jpg"]

[0061] conversations:

[0062] type: list;

[0063] Description: Contains every message in the conversation. Each message is a dictionary containing the following key-value pairs:

[0064] from: The source of the message, which can be "human" or "answer".

[0065] value: The specific content of the message. If the message contains , it means that the message references an image. There can be multiple Tags to reference multiple images.

[0066] 1.2) Use the single-image and multi-image sample sets as input data for the multimodal large model, train or fine-tune the multimodal large model until the set conditions are met to obtain a trained large model.

[0067] In this embodiment, the multimodal large model is trained or fine-tuned, and the fine-tuning includes fine-tuning the multimodal large model using full model fine-tuning and Lora fine-tuning.

[0068] In the above step 2), a standard prompt is generated based on a preset structured question template, wherein the preset structured question template is a question about image quality, including video loss, clarity, contrast, noise, snow, brightness, color, and occlusion of the image.

[0069] In the above step 2), the image to be diagnosed is queried for scene similarity or image similarity in the dynamic quality diagnosis database. If the query result is similar, the query result is integrated into the standard prompt to generate an enhanced prompt, and the enhanced prompt asks questions to the multimodal large model, including the following steps:

[0070] 2.1.1) Compare the similarity between the image to be diagnosed and the images in the dynamic quality diagnosis expert database, and find the top-K similar images in the dynamic quality diagnosis expert database according to the set threshold, where K is a preset parameter;

[0071] In this embodiment, the new image to be diagnosed is compared with the images in the dynamic quality diagnosis expert database for similarity. Specifically, the image to be diagnosed I is encoded into a vector V using an image encoder, and the scene similarity or image similarity is compared with the images in the dynamic quality diagnosis database.

[0072] 2.1.2) Based on the user's standard prompt, the similar images found, and the user's processing results of similar scene images, an enhanced prompt is generated, and the enhanced prompt asks questions to the large model;

[0073] In this embodiment, an enhanced Prompt is generated based on the user standard Prompt, the similar images found, and the user's processing results of similar scene images, specifically:

[0074] The user standard prompt is combined with similar images and corresponding scene descriptions found in the dynamic quality diagnosis expert database, user-modified evaluation scores and quality descriptions, and the user's knowledge of processing results for similar images is injected into the user standard prompt to generate an enhanced prompt.

[0075] For example, evaluate the quality of image 1 and give a detailed description and score. Please refer to the generation result of reference image 2 in the database. Reference image 2 appears to have a color cast problem due to the influence of ambient light. This is caused by environmental influences. The camera itself does not have color anomalies.

[0076] Alternatively, evaluate the quality of image 3 and give a detailed description and score. Please refer to the generation result of reference image 4 in the database. Reference image 4 has some noise and the score of noise is 6 points.

[0077] Alternatively, the reference image 5 in the database appears to have a color cast due to the influence of ambient light, but it is actually a normal image. Please determine whether image 6 has a color cast problem.

[0078] 2.1.3) Input the image to be diagnosed and the enhanced prompt into the multimodal large model for content generation.

[0079] In the above steps 2) and 3), questions are asked to the multimodal large model, and the output includes a scene description, a diagnosis score and evaluation reasons, and a detailed description of the image quality. The following steps are included:

[0080] 2.2.1) Describe the scene of the image to be diagnosed, and describe the video loss, clarity, contrast, noise, snow, brightness, color, and occlusion of the image, and give a score and reason for the evaluation;

[0081] 2.2.2) Comprehensively evaluate the level and index scores of each fault to obtain the overall quality score of the image.

[0082] For example: As an image quality expert, describe the scene of the image, and rate the image quality according to video loss, clarity, contrast, noise, snow, brightness, color, and occlusion (with 10 points as the full score), and give a comprehensive score.

[0083] Generate results, scenario description:

[0084] This image is a screenshot of a surveillance video of a train platform. The rails, safety lines at the edge of the platform, and some pillar structures can be seen. The timestamp shows that it was around 5 pm on December 20, 2019.

[0085] Image quality assessment:

[0086] Video loss: 9 points

[0087] There is no noticeable frame loss in the image.

[0088] Clarity: 6 points

[0089] Overall, it's clear enough that you can make out major objects like tracks and platforms, but details aren't sharp enough.

[0090] Contrast: 4 points

[0091] There is not much difference in light and dark, and there is not enough sense of hierarchy to distinguish the relationship between different elements.

[0092] Noise: 7 points

[0093] There is a certain amount of graininess, especially noticeable in darker areas.

[0094] Snowflake: 8 points

[0095] No obvious signs of snowflake interference were observed.

[0096] Brightness: 5 points

[0097] The lighting conditions are average, neither too bright nor too dim, but somewhere in the middle.

[0098] Color: 5 points

[0099] This is a black and white image, so there is no color information to evaluate.

[0100] Occlusion: 9 points

[0101] No significant obstructions to vision were found.

[0102] Overall score: 6 points

[0103] Taking all the above indicators into consideration, the overall image quality is medium to low. Although the scene content can be basically recognized, the details need to be improved. If you want to improve its quality, you need to focus on improving contrast, reducing noise, and increasing color reproduction.

[0104] In the above step 4), determining whether the diagnosis result is stored in the dynamic quality diagnosis expert database according to user needs includes the following steps:

[0105] 4.1) If the user approves the diagnosis result of the large model, the image processing ends;

[0106] 4.2) If the user does not agree with the diagnosis result of the large model, or the result is incorrect or biased, the scene description, evaluation score and quality description of the image will be modified;

[0107] 4.3) The image, modified scene description, modified image score, and modified image quality detailed description are added to the dynamic quality diagnosis expert database (DB-IQD), the image, scene description, evaluation score, and quality description are stored, and the dynamic quality diagnosis expert database is updated.

[0108] In this embodiment, the user can decide whether to modify the generated result.

[0109] In one embodiment of the present invention, a feedback-based image quality diagnosis system is provided, comprising:

[0110] The training module uses on-site rail transit data to train or fine-tune the multimodal large model to obtain a trained multimodal large model;

[0111] The enhanced prompt module performs scene similarity or image similarity query on the image to be diagnosed in the dynamic quality diagnosis expert database. If the query result is similar, the query result is integrated into the standard prompt to generate an enhanced prompt. The enhanced prompt asks questions to the multimodal large model and outputs the diagnosis result including scene description, diagnosis score and evaluation reason, as well as a detailed description of the image quality.

[0112] Standard Prompt module: if the query result is not similar, the standard prompt will ask questions to the multimodal large model and output the diagnosis result including scene description, diagnosis score and evaluation reason, as well as a detailed description of image quality;

[0113] The database dynamic update module determines whether the diagnosis result should be stored in the dynamic quality diagnosis expert database according to user requirements, so as to dynamically update the dynamic quality diagnosis expert database.

[0114] In the above embodiment, the multimodal large model is trained or fine-tuned using field samples, including:

[0115] Construct image quality diagnosis sample sets, which are divided into single-image and multi-image sample sets;

[0116] The single-image and multi-image sample sets are used as input data for the multimodal large model, and the multimodal large model is trained or fine-tuned until the set conditions are met to obtain a trained large model.

[0117] In the above embodiment, a standard prompt is generated based on a preset structured question template, wherein the preset structured question template is a question about image quality, including video loss, clarity, contrast, noise, snow, brightness, color, and occlusion of the image.

[0118] In the above embodiment, the image to be diagnosed is queried for scene similarity or image similarity in the dynamic quality diagnosis database. If the query result is similar, the query result is integrated into the standard prompt to generate an enhanced prompt, and the enhanced prompt asks questions to the multimodal large model, including:

[0119] Compare the similarity between the image to be diagnosed and the images in the dynamic quality diagnosis expert database, and find the top-K similar images in the dynamic quality diagnosis expert database according to the set threshold, where K is a preset parameter;

[0120] Based on the user's standard prompt, similar images found, and the user's processing results on images of similar scenes, an enhanced prompt is generated, and the enhanced prompt asks questions to the large model;

[0121] The image to be diagnosed and the enhanced prompt are input into the multimodal large model to generate content and obtain the diagnosis result.

[0122] In this embodiment, the image to be diagnosed is compared with the images in the quality diagnosis expert database for similarity. Specifically, the image to be diagnosed is encoded into a vector using an image encoder, and a scene similarity comparison or image similarity comparison is performed with the images in the dynamic quality diagnosis database.

[0123] In the above embodiment, questions are asked to the multimodal large model, and the output includes a scene description, a diagnosis score and evaluation reasons, and a detailed description of the image quality, including:

[0124] Describe the scene of the image to be diagnosed, and describe the video loss, clarity, contrast, noise, snow, brightness, color, and occlusion of the image, score it, and give reasons for the evaluation;

[0125] The overall quality score of the image is obtained by combining the levels and index scores of each fault.

[0126] In the above embodiment, determining whether the diagnosis result is stored in the dynamic quality diagnosis expert database according to the user's needs includes:

[0127] If the user approves the diagnosis result of the large model, the image processing ends;

[0128] If the user does not agree with the diagnosis result of the large model, or the result is incorrect or biased, the scene description, evaluation score and quality description of the image will be modified;

[0129] The image, the modified scene description, the modified image score, and the modified image quality detailed description are added to the dynamic quality diagnosis expert database, the image, scene description, evaluation score and quality description are stored, and the dynamic quality diagnosis expert database is updated.

[0130] The system provided in this embodiment is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for specific processes and detailed contents, which will not be repeated here.

[0131] In one embodiment of the present invention, a computing device is provided, which may be a terminal, and may include: a processor, a communication interface, a memory, a display screen, and an input device. Among them, the processor, the communication interface, and the memory communicate with each other through a communication bus. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory, and the non-volatile storage medium stores an operating system and a computer program. When the computer program is executed by the processor, the method in each of the above embodiments is implemented; the internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a management network, NFC (near field communication) or other technologies. The display screen may be a liquid crystal display screen or an electronic ink display screen, and the input device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computing device, or an external keyboard, touchpad or mouse, etc. The processor may call the logic instructions in the memory.

[0132] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0133] In one embodiment of the present invention, a computer program product is provided, wherein the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments.

[0134] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores server instructions, and the computer instructions enable a computer to execute the methods provided in the above embodiments.

[0135] The above embodiment provides a computer-readable storage medium, whose implementation principle and technical effect are similar to those of the above method embodiment, and will not be repeated here.

[0136] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0137] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A feedback-based image quality diagnosis method, characterized in that: include: Use on-site rail transit data to train or fine-tune the multimodal large model to obtain a trained multimodal large model; When executing a new diagnostic task, the image to be diagnosed is queried for scene similarity or image similarity in the dynamic quality diagnosis expert database. If the query result is similar, the query result is integrated into the standard prompt to generate an enhanced prompt. The enhanced prompt asks questions to the multimodal large model and outputs a diagnostic result including scene description, diagnostic score and evaluation reason, as well as a detailed description of the image quality. If the query result is not similar, the multimodal large model is asked questions by the standard prompt, and the output includes a diagnosis result including a scene description, a diagnosis score and evaluation reason, and a detailed description of the image quality; Determine whether the diagnosis result is stored in the dynamic quality diagnosis expert database according to user needs, so as to dynamically update the dynamic quality diagnosis expert database.

2. The feedback-based image quality diagnosis method according to claim 1, characterized in that: Use field samples to train or fine-tune large multimodal models, including: Construct image quality diagnosis sample sets, which are divided into single-image and multi-image sample sets; The single-image and multi-image sample sets are used as input data for the multimodal large model, and the multimodal large model is trained or fine-tuned until the set conditions are met to obtain a trained large model.

3. The feedback-based image quality diagnosis method according to claim 1, characterized in that: A standard prompt is generated based on a preset structured question template, wherein the preset structured question template is a question about image quality, including video loss, clarity, contrast, noise, snow, brightness, color, and occlusion of the image.

4. The feedback-based image quality diagnosis method according to claim 1, characterized in that: The image to be diagnosed is queried for scene similarity or image similarity in the dynamic quality diagnosis database. If the query result is similar, the query result is integrated into the standard prompt to generate an enhanced prompt, which asks questions to the multimodal large model, including: Compare the similarity between the image to be diagnosed and the images in the dynamic quality diagnosis expert database, and find the top-K similar images in the dynamic quality diagnosis expert database according to the set threshold, where K is a preset parameter; Based on the user's standard prompt, similar images found, and the user's processing results on images of similar scenes, an enhanced prompt is generated, and the enhanced prompt asks questions to the large model; The image to be diagnosed and the enhanced prompt are input into the multimodal large model to generate content and obtain the diagnosis result.

5. The feedback-based image quality diagnosis method according to claim 4, characterized in that: Compare the image to be diagnosed with the images in the quality diagnosis expert database for similarity, including: The image to be diagnosed is encoded into a vector using an image encoder, and scene similarity comparison or image similarity comparison is performed with images in a dynamic quality diagnosis database.

6. The feedback-based image quality diagnosis method according to claim 1, characterized in that: Questions are asked to the multimodal large model, and the output includes a scene description, a diagnostic score and evaluation reasons, and a detailed description of the image quality, including: Describe the scene of the image to be diagnosed, and describe the video loss, clarity, contrast, noise, snow, brightness, color, and occlusion of the image, score it, and give reasons for the evaluation; The overall quality score of the image is obtained by combining the levels and index scores of each fault.

7. The feedback-based image quality diagnosis method according to claim 1, characterized in that: Determine whether the diagnosis results are stored in the dynamic quality diagnosis expert database based on user needs, including: If the user approves the diagnosis result of the large model, the image processing ends; If the user does not agree with the diagnosis result of the large model, or the result is incorrect or biased, the scene description, evaluation score and quality description of the image will be modified; The image, the modified scene description, the modified image score, and the modified image quality detailed description are added to the dynamic quality diagnosis expert database, the image, scene description, evaluation score and quality description are stored, and the dynamic quality diagnosis expert database is updated.

8. A feedback-based image quality diagnosis system, characterized in that: include: The training module uses on-site rail transit data to train or fine-tune the multimodal large model to obtain a trained multimodal large model; The enhanced prompt module performs scene similarity or image similarity query on the image to be diagnosed in the dynamic quality diagnosis expert database. If the query result is similar, the query result is integrated into the standard prompt to generate an enhanced prompt. The enhanced prompt asks questions to the multimodal large model and outputs the diagnosis result including scene description, diagnosis score and evaluation reason, as well as a detailed description of the image quality. Standard Prompt module: if the query result is not similar, the standard prompt will ask questions to the multimodal large model and output the diagnosis result including scene description, diagnosis score and evaluation reason, as well as a detailed description of image quality; The database dynamic update module determines whether the diagnosis result should be stored in the dynamic quality diagnosis expert database according to user requirements, so as to dynamically update the dynamic quality diagnosis expert database.

9. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any one of the methods of claims 1 to 7.

10. A computing device, characterized in that include: One or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the methods according to claims 1 to 7 instruction.

Images