Deep learning and image-text large model combined image diagnosis method, system, medium and equipment

Abstract

The invention relates to the field of computers, and discloses a deep learning and image-text large model combined image diagnosis method and system, a medium and equipment, and the method comprises the steps: carrying out the description and fault definition of an image diagnosis task, constructing image diagnosis data based on decoupling modeling, training a corresponding image diagnosis algorithm, and carrying out the preliminary image diagnosis of an image; inputting the preliminary image diagnosis result into a general image-text large model to generate a reply, and constructing an image diagnosis instruction data set through manual verification and data cleaning and enhancement; performing visual instruction fine adjustment on the image diagnosis instruction data set to obtain an adjustment and optimization image-text big model for enhancing the image diagnosis situation and IQA aspect capability, and correcting the preliminary image diagnosis result by the adjustment and optimization image-text big model according to the scene condition; and performing alarm logic processing based on the corrected image diagnosis result, judging whether to give an alarm or not, and transmitting an alarm signal to an operator. According to the method, the recognition effect on the interference category depending on high-level semantics is improved, and the interactivity and interpretability of graph diagnosis analysis are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to an image diagnosis method, system, medium and device combining deep learning with a large image and text model. Background Art

[0002] Image Quality Assessment (IQA) is an important task in the field of computer vision, which aims to identify the categories of visual interference in videos or still images and estimate the severity of these interferences. Existing image quality assessment methods usually use quality scores to describe image quality. These scores are not only used to compare the performance of different models, but are also widely used as evaluation indicators or training loss functions, which promotes the development of technologies in fields such as image generation and restoration.

[0003] However, the quality score as a description form has certain limitations. It can only provide a comprehensive assessment of image quality and cannot describe complex locality and content correlation in detail. In addition, traditional image quality assessment methods tend to focus on low-level visual features such as brightness, contrast, noise, blur, and color shift. These features are relatively easy to summarize and identify. However, in practical applications, such as railway monitoring image quality diagnosis, in addition to the above low-level features, it is also necessary to pay attention to high-level visual features such as no signal and occlusion. These features do not have obvious pattern characteristics and are difficult for traditional computer vision algorithms to effectively identify. Summary of the invention

[0004] In response to the above problems, the purpose of the present invention is to provide a method, system, medium and device for image diagnosis that combines deep learning with a large image and text model, which improves the recognition effect of interference categories that rely on high-level semantics and enhances the interactivity and interpretability of image diagnosis analysis.

[0005] To achieve the above-mentioned purpose, the present invention adopts the following technical scheme: a method for image diagnosis combining deep learning with a large image and text model, which includes: by describing the image diagnosis task and defining the fault, constructing image diagnosis data based on decoupled modeling and training the corresponding image diagnosis algorithm, and performing preliminary image diagnosis on the image; inputting the preliminary image diagnosis result into the general image and text large model to generate a reply, and constructing an image diagnosis instruction data set through manual verification, data cleaning and enhancement; fine-tuning the visual instructions based on the image diagnosis instruction data set to obtain an optimized image and text large model with enhanced image diagnosis context and IQA capabilities, and the optimized image and text large model corrects the preliminary image diagnosis result according to the scene conditions; performing alarm logic processing based on the corrected image diagnosis result, judging whether to alarm, and transmitting the alarm signal to the operator.

[0006] Furthermore, image diagnosis data based on decoupled modeling is constructed, including: based on the image diagnosis task description and fault definition, as well as specific business needs, image data and diagnostic information are independently processed through a decoupled image diagnosis data annotation method to construct image diagnosis data based on decoupled modeling.

[0007] Furthermore, we construct a data set of image diagnosis instructions, including:

[0008] Construct a pre-training data set in the railway field, including brief scene descriptions in the railway field, brief judgments on the quality of railway video surveillance images, management regulations related to railway surveillance, and news reports related to railway surveillance. All industry knowledge data exists in the form of picture-text pairs;

[0009] Based on the industry knowledge dataset in the pre-training stage, a downstream task image diagnosis instruction dataset is constructed.

[0010] Furthermore, the image diagnosis instruction data set includes a simple part and a complex part;

[0011] The simple part is responsible for identifying whether there is any abnormality in the image quality of the monitoring screen, what aspects of the quality abnormality exist, and briefly explaining the severity of the abnormality;

[0012] The complex part is deep logical reasoning. The reply needs to generate useful, harmless and concise information based on the input command prompt information; the command data is formed through multiple rounds of dialogue: self-instruct interacts with the SOTA's large graphic model, and the data is manually reviewed after generation; rethink the key aspects of the required reply and the corresponding capabilities of GPT-4V, and introduce prior supervision information through manual annotation prompts to enhance the accuracy of GPT-4V generation; high-quality detailed replies include image content, key distortions, the impact of distortion on content, and conclusions.

[0013] Furthermore, visual command fine-tuning is performed based on the image diagnosis command dataset, including: a pre-training stage and a fine-tuning stage;

[0014] In the pre-training stage, the industry expertise is collected, cleaned and organized into image-text pairs, and the industry knowledge is injected into the image diagnosis instruction dataset in an incremental pre-training manner;

[0015] In the fine-tuning phase, we construct question-answering data with different reasoning depths and covering all aspects of the task, and align them with task requirements in the form of visual instruction fine-tuning.

[0016] Furthermore, in the fine-tuning stage, the parameters of the image encoder are kept unchanged, and the mapping layer and the large language model are trained; at the same time, efficient fine-tuning and distributed training are introduced.

[0017] Furthermore, alarm logic processing is performed based on the corrected image diagnosis results, including:

[0018] According to the definition of specific business requirements, in the fault category that relies on low-level visual features, for the "unbearable" part, if the preliminary image diagnosis results have been reported and the severity exceeds the acceptable threshold, an alarm is directly issued;

[0019] The remaining images judged as "abnormal" by the preliminary image diagnosis results are analyzed by the large image and text model. The large image and text model is tuned to correct the preliminary image diagnosis results through the set instruction template and the preliminary image diagnosis results. The corrected image diagnosis structure is used to determine whether to send an alarm signal based on the fault level threshold.

[0020] The content of the instruction template includes context recognition, monitoring timestamp OCR recognition and OSD information.

[0021] A picture diagnosis system combining deep learning with a picture and text big model comprises: a deep learning initial diagnosis module, which constructs picture diagnosis data based on decoupled modeling and trains corresponding picture diagnosis algorithms through picture diagnosis task description and fault definition, and performs preliminary picture diagnosis on images; a data set construction module, which inputs preliminary picture diagnosis results into a general picture and text big model to generate a reply, and constructs a picture diagnosis instruction data set through manual verification, data cleaning and enhancement; a picture and text big model tuning module, which fine-tunes visual instructions based on the picture diagnosis instruction data set to obtain a tuned picture and text big model with enhanced picture diagnosis context and IQA capabilities, and the tuned picture and text big model corrects the preliminary picture diagnosis results according to scene conditions; an alarm logic judgment module, which performs alarm logic processing based on the corrected picture diagnosis results, judges whether to alarm, and transmits the alarm signal to the operator.

[0022] A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any of the above methods.

[0023] A computing device comprises: one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0024] The present invention adopts the above technical solution, which has the following advantages:

[0025] 1. This invention combines the traditional deep learning-based image diagnosis algorithm and the advanced large image and text model to form a new image diagnosis method, which not only improves the recognition effect of interference categories that rely on high-level semantics, but also enhances the interactivity and interpretability of image diagnosis analysis, so that it can be adaptively applied to different scenarios. At the same time, by cooperating with the traditional image diagnosis algorithm, it effectively avoids the cost consumption brought by the large image and text model in intensive reasoning tasks.

[0026] 2. The present invention uses a decoupled image diagnosis data annotation method to independently process image data and diagnostic information, and adopts a modular annotation process to separate and annotate image features from diagnostic results, thereby reducing manual intervention and improving the efficiency and accuracy of data annotation. This annotation method flexibly adapts to different task requirements, ensures the consistency of data quality, and provides high-quality basic data for subsequent model training.

[0027] 3. The present invention constructs data for industry knowledge injection and downstream task instruction fine-tuning for the large-scale image and text model: by collecting industry knowledge bases and injecting them into the large-scale image and text model, the model can deeply grasp the knowledge in specific fields. By using the constructed multi-dimensional data sources (images, texts, industry corpora, etc.), the knowledge is integrated with the large model through layered injection, and the downstream tasks are fine-tuned, so that the performance of the model in industry applications is more accurate and practical.

[0028] 4. The present invention combines traditional deep learning with the large image and text model: first, the traditional deep learning network is used to perform preliminary image processing, extract key feature information, and then input into the large image and text model to use its cross-modal understanding and reasoning capabilities for further diagnostic analysis. This combination not only solves the limitations of traditional deep learning in processing complex image tasks, but also enhances the model's multi-dimensional understanding of text, images, and industry knowledge, achieving more accurate diagnostic results.

[0029] In summary, the present invention organically integrates decoupled annotation, industry knowledge injection and multimodal diagnosis of large graphic models, constructs an intelligent diagnosis system that is adaptable to a variety of diagnosis scenarios and has a high degree of professional knowledge, and effectively improves the accuracy and efficiency of diagnosis. Through the innovation of these key technical points, the present invention not only solves the limitations of traditional deep learning in complex diagnosis scenarios, but also provides strong technical support for applications in different fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of the image diagnosis method combining deep learning with a large image and text model in an embodiment of the present invention;

[0031] Figure 2 It is a flowchart of vertical optimization of a general graphic large model in an embodiment of the present invention;

[0032] Figure 3is a flow chart of alarm logic decision in an embodiment of the present invention;

[0033] Figure 4 It is a schematic diagram of the image diagnosis effect when used in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In recent years, with the release of models such as ChatGPT and GPT-4-V by OpenAI, research on large language models, large image and text models, and even large multimodal models has made significant progress. These large models have demonstrated strong generalization capabilities through techniques such as prompt learning, contextual learning, and visual instruction fine-tuning. Especially in the open source field, large image and text models represented by BLIP-2 and LLaVA organically interact visual feature information with the language large model base through a bridge network, thus bridging the information gap between visual and text modalities, and have been gradually applied and promoted in fields such as medicine and finance.

[0035] The existing railway monitoring image quality has the following problems: 1) Identification and processing of low-level visual feature interference: Traditional image diagnosis algorithms perform well in identifying and processing low-level visual features (such as brightness, contrast, noise, etc.), but have limited recognition capabilities for abnormal categories that rely on high-level semantic information (such as no signal, occlusion, etc.). 2) Insufficient complex semantic discrimination: Existing image quality assessment methods have limitations when processing complex semantic information and cannot effectively identify and analyze high-level visual features such as no signal and occlusion. 3) Improvement of generalization ability and scene adaptability: Traditional methods have insufficient generalization capabilities in different scenarios and are difficult to deal with diverse image quality issues. 4) Optimization of computational costs: Simply relying on large models for reasoning will bring high computational costs, affecting the real-time and efficiency of the system.

[0036] In order to solve the above problems, the present invention provides an image diagnosis method, system, medium and equipment that combine deep learning with a large image and text model. By organically integrating the traditional image diagnosis algorithm with the large image and text model and adopting an efficient modular design, the fast execution advantages of the traditional image diagnosis algorithm are combined with the powerful reasoning ability of the large image and text model, thereby avoiding the high computing cost brought about by relying solely on large models for reasoning; ensuring the real-time and efficiency of the system, while improving the overall performance.

[0037] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0038] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0039] In one embodiment of the present invention, a method for image diagnosis is provided that combines deep learning with a large image and text model, which can identify the type of quality anomalies in the image, assess the severity of the anomalies, and perform reasoning analysis to ultimately decide whether to send an alarm signal to maintenance personnel. In this embodiment, the advantages of traditional deep learning and large image and text models are combined to effectively optimize the shortcomings of traditional image diagnosis. Through subsequent logical modules, the efficiency of image diagnosis of large models is improved, and adaptive adjustment space is provided for changes in business needs, making the entire system more flexible. Figure 1 As shown, the method comprises the following steps:

[0040] 1) By describing the image diagnosis task and defining the fault, constructing the image diagnosis data based on decoupling modeling and training the corresponding image diagnosis algorithm, the image is preliminarily diagnosed to complete the recognition and processing of low-level visual features;

[0041] In this embodiment, a traditional deep learning image diagnosis algorithm is used to perform basic quality assessment on the image (i.e., preliminary image diagnosis) to identify and process pixel-level interference that is easy to summarize. Preliminary image diagnosis can quickly and accurately identify and process fault categories of low-level visual features (e.g., brightness, color, blur, noise), providing a basis for subsequent advanced analysis.

[0042] However, errors still exist for faults that rely on high-level semantic information (such as no signal and occlusion), and because the algorithm is extremely sensitive to pixel-level features, false alarms caused by changes in light and shadow in the external environment often occur. As businesses migrate to different scenarios, alarm standards are constantly adjusted, and the lack of flexibility in traditional deep learning image diagnosis is gradually becoming apparent.

[0043] 2) Input the preliminary image diagnosis results into the general image and text model to generate responses, and build a clean, high-quality image diagnosis instruction dataset through manual verification, data cleaning and enhancement;

[0044] In this embodiment, the large image-text model further analyzes complex high-level interference categories through visual-language interaction, making up for the shortcomings of traditional image diagnosis algorithms in complex semantic discrimination. The large image-text model can recognize and process high-level visual features such as no signal and occlusion, improving the overall recognition ability of the system.

[0045] Moreover, through the prompt learning and context learning of the large image and text model, the system can better generalize to different scenarios and enhance the recognition accuracy of diverse abnormal categories. This makes the system not only suitable for railway monitoring images, but also can flexibly deal with image quality issues in other complex scenarios.

[0046] 3) Fine-tune the visual instructions based on the image diagnosis instruction dataset to obtain a tuned image-text large model that enhances the image diagnosis context and IQA capabilities. The tuned image-text large model corrects the preliminary image diagnosis results based on high-level semantic information such as scene conditions;

[0047] In this embodiment, benefiting from the semantic information understanding and deep reasoning capabilities of the large image and text model, the initial traditional image diagnosis output results can be corrected by combining the context of the monitoring screen and the input command prompts.

[0048] In actual use, the image to be tested is first processed by a traditional algorithm to obtain a preliminary image diagnosis result, and then the preliminary image diagnosis result is combined with the designed question instruction template and fed to the tuning model as input. The tuning model will correct the preliminary image diagnosis result based on high-level semantic information such as scene conditions.

[0049] 4) Perform alarm logic processing based on the corrected image diagnosis results, determine whether to alarm, and transmit the alarm signal to the operator for repair and maintenance.

[0050] When in use, the fast execution advantage of traditional image diagnosis algorithms is combined with the powerful reasoning ability of large image and text models, thus avoiding the high computational cost of relying solely on large models for reasoning. The real-time and efficiency of the system are ensured, while the overall performance is improved. The present invention not only improves the recognition accuracy of diverse abnormal categories, but also enhances the generalization ability and scene adaptability of the system. Finally, by introducing an adaptive diagnostic process, the system can flexibly respond to different types of image quality problems, and is particularly suitable for quality diagnosis tasks in complex scenes such as railway monitoring images.

[0051] In the above step 1), image diagnosis data based on decoupled modeling is constructed, specifically: based on the image diagnosis task description and fault definition, as well as specific business needs, the image data and diagnostic information are processed independently through a decoupled image diagnosis data annotation method to construct image diagnosis data based on decoupled modeling.

[0052] Among them, the image diagnosis data based on decoupled modeling constitutes a structured quality diagnosis sample database, which covers 9 major fault categories (including too bright, too dark, blurry, abnormal contrast, color shift, stripe noise, snowflake noise, no signal, and occlusion). Each category is further subdivided into 5 progressive severity levels (marked as none, slight, general, obvious, and severe).

[0053] Since previous anomaly classification algorithms are often implemented based on one-hot labeling, this will bring about label sparsity problems, and the lack of modeling of relationships between anomalies will make the model less interpretable. The decoupled classification system of the present invention ensures that the diagnosis results can accurately reflect the actual situation of the fault, whether it is a minor initial sign or a serious damage, it can be properly evaluated and classified.

[0054] In the above step 2), constructing the image diagnosis instruction data set includes the following steps:

[0055] 2.1) Construct a pre-training data set in the railway field, including brief scene descriptions in the railway field, brief judgments on the quality of railway video surveillance images, management regulations related to railway surveillance, and news reports related to railway surveillance. All industry knowledge data exists in the form of picture-text pairs;

[0056] In this embodiment, the length of the text sequence of the image and text is in the range of 20-128, which roughly satisfies the normal distribution.

[0057] 2.2) Based on the industry knowledge dataset in the pre-training stage, a downstream task image diagnosis instruction dataset is constructed.

[0058] In this embodiment, the image diagnosis instruction data set includes a simple part and a complex part.

[0059] The simple part is responsible for identifying whether there are abnormal image quality in the monitoring screen, what aspects of the quality abnormality exist, and briefly describing the severity of the abnormal problem (top2). The question-answer generation method is through cloze-style template generation. First, it interacts with SOTA LLM such as GPT to generate a preliminary question-answer template pool (about 50), and then manually verifies it to remove duplicate and unclear templates. Finally, 20 question-answer templates are selected to form a template pool. When forming the question-answer data, for each caption sample, one is randomly selected from the template pool for application.

[0060] The complex part is deep logical reasoning. The reply needs to generate useful, harmless and concise information based on the input command prompt information; the command data is formed through multiple rounds of dialogue: self-instruct interacts with the SOTA's large graphic model, and the data is manually reviewed after generation; rethink the key aspects of the required reply and the corresponding capabilities of GPT-4V, and introduce prior supervision information through manual annotation prompts to enhance the accuracy of GPT-4V generation; high-quality detailed replies include image content, key distortions, the impact of distortion on content, and conclusions.

[0061] In the above step 3), since the general large image and text model usually performs poorly on specific downstream tasks, it is necessary to perform vertical tuning of the general model in specific fields.

[0062] Specifically, fine-tuning of visual instructions is performed based on the image diagnosis instruction dataset, including: pre-training stage and fine-tuning stage, such as Figure 2 shown.

[0063] During the pre-training phase, industry expertise is collected, cleaned, and organized into image-text pairs, and industry knowledge is injected into the image diagnosis instruction dataset in an incremental pre-training manner. During this phase, the parameters of the image encoder and the large language model are fixed, and only the intermediate image-text mapping module is trained.

[0064] In the fine-tuning phase, we construct question-answering data with different reasoning depths and covering all aspects of the task, and align them with task requirements in the form of visual instruction fine-tuning.

[0065] Among them, in the fine-tuning stage, the parameters of the image encoder are kept unchanged, and the mapping layer and the language large model are trained; at the same time, efficient fine-tuning (such as LoRA, Q-LoRA) and distributed training (such as data parallelism, tensor parallelism, pipeline parallelism, and sequence parallelism) are introduced.

[0066] In the above step 4), the alarm logic processing is performed based on the corrected image diagnosis results, such as Figure 3 As shown, the following steps are included:

[0067] 4.1) According to the definition of specific business requirements, in the fault category that relies on low-level visual features, for the "unbearable" part, if the preliminary image diagnosis results have been reported and the severity exceeds the acceptable threshold, an alarm is directly issued;

[0068] 4.2) For the remaining images that are judged as "abnormal" by the preliminary image diagnosis results, they enter the image and text large model for analysis. The image and text large model is adjusted to correct the preliminary image diagnosis results through the set instruction template and the preliminary image diagnosis results, and the corrected image diagnosis results are obtained. The corrected image diagnosis structure is used to decide whether to send an alarm signal based on the fault level threshold; among which, the content of the instruction template includes situation recognition, monitoring timestamp OCR recognition and OSD information.

[0069] In summary, if Figure 4 As shown, the present invention can effectively improve the efficiency and accuracy of annotation when used: by adopting a decoupled image diagnosis data annotation method, the present invention effectively reduces manual intervention, avoids subjective bias, and significantly improves the efficiency and accuracy of data annotation. Moreover, the modular annotation process also improves the consistency and scalability of the annotation data, ensuring the stability and reliability of the data quality.

[0070] Furthermore, the present invention uses an industry knowledge injection solution to deeply integrate the knowledge base of professional fields with the large graphic model, so that the model has the ability to understand and apply industry-specific knowledge. This process not only broadens the scope of application of the model, but also improves its performance in specific fields, enabling it to better cope with diagnostic tasks in complex scenarios.

[0071] Furthermore, the present invention improves the accuracy of the model in downstream tasks: through the mechanism of fine-tuning downstream task instructions, the present invention can match industry knowledge with the performance of the model in actual application scenarios. The output of the model is highly consistent with the task requirements, ensuring its efficient execution and accurate response to specific tasks, and improving the accuracy and reliability of task completion.

[0072] Furthermore, the present invention realizes the fusion of multimodal information and improves the diagnostic effect: the multimodal diagnostic system combining traditional deep learning and large-scale image and text models can make comprehensive use of images, texts and industry knowledge to achieve efficient fusion of cross-modal information. This multimodal diagnostic method effectively makes up for the limitations of a single mode and improves the diagnostic effect in complex scenarios, especially when dealing with faults that rely on high-level semantic information.

[0073] Furthermore, the self-learning ability and diagnostic robustness of the present invention are enhanced: the present invention has a self-learning mechanism, which can continuously improve the diagnostic accuracy and robustness of the model through continuous feedback and optimization. This self-learning feature ensures that the system can still maintain high adaptability and diagnostic effect when facing new tasks and data, and enhances the long-term stability and reliability of the system.

[0074] Furthermore, the invention is applicable to a wide range of scenarios: through multimodal fusion and industry knowledge injection, the invention can adapt to the needs of different fields and scenarios, especially in the fields of medical diagnosis and industrial detection, and has broad application prospects. The flexibility and scalability of the system enable it to quickly adapt to new application scenarios and provide customized solutions for different industries.

[0075] In one embodiment of the present invention, a graphic diagnosis system combining deep learning with a graphic and text model is provided, which includes:

[0076] The deep learning initial diagnosis module describes the image diagnosis task and defines the fault, constructs image diagnosis data based on decoupled modeling, and trains the corresponding image diagnosis algorithm to perform preliminary image diagnosis on the image;

[0077] The data set construction module inputs the preliminary image diagnosis results into the general image and text model to generate responses, and constructs the image diagnosis instruction data set through manual verification, data cleaning and enhancement;

[0078] The image-text large model tuning module fine-tunes the visual instructions based on the image diagnosis instruction dataset to obtain a tuned image-text large model that enhances the image diagnosis context and IQA capabilities. The tuned image-text large model corrects the preliminary image diagnosis results according to the scene conditions;

[0079] The alarm logic judgment module performs alarm logic processing based on the corrected image diagnosis results, judges whether to alarm, and transmits the alarm signal to the operator.

[0080] In the above embodiment, image diagnosis data based on decoupled modeling is constructed, specifically: based on the image diagnosis task description and fault definition, as well as specific business needs, the image data and diagnostic information are processed independently through a decoupled image diagnosis data labeling method to construct image diagnosis data based on decoupled modeling.

[0081] In the above embodiment, constructing the image diagnosis instruction data set includes:

[0082] Construct a pre-training data set in the railway field, including brief scene descriptions in the railway field, brief judgments on the quality of railway video surveillance images, management regulations related to railway surveillance, and news reports related to railway surveillance. All industry knowledge data exists in the form of picture-text pairs;

[0083] Based on the industry knowledge dataset in the pre-training stage, a downstream task image diagnosis instruction dataset is constructed.

[0084] In this embodiment, the image diagnosis instruction data set includes a simple part and a complex part;

[0085] The simple part is responsible for identifying whether there is any abnormality in the image quality of the monitoring screen, what aspects of the quality abnormality exist, and briefly explaining the severity of the abnormality;

[0086] The complex part is deep logical reasoning. The reply needs to generate useful, harmless and concise information based on the input command prompt information; the command data is formed through multiple rounds of dialogue: self-instruct interacts with the SOTA's large graphic model, and the data is manually reviewed after generation; rethink the key aspects of the required reply and the corresponding capabilities of GPT-4V, and introduce prior supervision information through manual annotation prompts to enhance the accuracy of GPT-4V generation; high-quality detailed replies include image content, key distortions, the impact of distortion on content, and conclusions.

[0087] In the above embodiment, fine-tuning of visual instructions based on the image diagnosis instruction data set includes: a pre-training stage and a fine-tuning stage;

[0088] In the pre-training stage, the industry expertise is collected, cleaned and organized into image-text pairs, and the industry knowledge is injected into the image diagnosis instruction dataset in an incremental pre-training manner;

[0089] In the fine-tuning phase, we construct question-answering data with different reasoning depths and covering all aspects of the task, and align them with task requirements in the form of visual instruction fine-tuning.

[0090] In this embodiment, during the fine-tuning stage, the parameters of the image encoder are kept unchanged, and the mapping layer and the language large model are trained; at the same time, efficient fine-tuning and distributed training are introduced.

[0091] In the above embodiment, the alarm logic processing is performed based on the corrected image diagnosis result, including:

[0092] According to the definition of specific business requirements, in the fault category that relies on low-level visual features, for the "unbearable" part, if the preliminary image diagnosis results have been reported and the severity exceeds the acceptable threshold, an alarm is directly issued;

[0093] The remaining images judged as "abnormal" by the preliminary image diagnosis results are analyzed by the large image and text model. The large image and text model is tuned to correct the preliminary image diagnosis results through the set instruction template and the preliminary image diagnosis results. The corrected image diagnosis structure is used to determine whether to send an alarm signal based on the fault level threshold.

[0094] The content of the instruction template includes context recognition, monitoring timestamp OCR recognition and OSD information.

[0095] The system provided in this embodiment is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for specific processes and detailed contents, which will not be repeated here.

[0096] In one embodiment of the present invention, a computing device is provided, which may be a terminal, and may include: a processor, a communication interface, a memory, a display screen, and an input device. Among them, the processor, the communication interface, and the memory communicate with each other through a communication bus. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory, and the non-volatile storage medium stores an operating system and a computer program. When the computer program is executed by the processor, the method in each of the above embodiments is implemented; the internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a management network, NFC (near field communication) or other technologies. The display screen may be a liquid crystal display screen or an electronic ink display screen, and the input device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computing device, or an external keyboard, touchpad or mouse, etc. The processor may call the logic instructions in the memory.

[0097] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0098] In one embodiment of the present invention, a computer program product is provided, wherein the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments.

[0099] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores server instructions, and the computer instructions enable a computer to execute the methods provided in the above embodiments.

[0100] The above embodiment provides a computer-readable storage medium, whose implementation principle and technical effect are similar to those of the above method embodiment, and will not be repeated here.

[0101] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0102] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0103] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for image diagnosis combining deep learning with a large image and text model, characterized in that: include: By describing the image diagnosis task and defining the fault, we construct image diagnosis data based on decoupled modeling and train the corresponding image diagnosis algorithm to perform preliminary image diagnosis on the image. The preliminary image diagnosis results are input into the general image and text model to generate responses, and the image diagnosis instruction dataset is constructed through manual verification, data cleaning and enhancement; Fine-tune the visual instructions based on the image diagnosis instruction dataset to obtain a large-scale image and text model that enhances the image diagnosis context and IQA capabilities. The large-scale image and text model is used to calibrate the preliminary image diagnosis results according to the scene conditions. Based on the corrected image diagnosis results, alarm logic processing is performed to determine whether to issue an alarm, and the alarm signal is transmitted to the operator.

2. The image diagnosis method combining deep learning with a large image and text model as claimed in claim 1, characterized in that: Construct image diagnosis data based on decoupled modeling, including: Based on the description of image diagnosis tasks, fault definition, and specific business needs, a decoupled image diagnosis data annotation method is used to independently process image data and diagnostic information, and construct image diagnosis data based on decoupled modeling.

3. The image diagnosis method combining deep learning with a large image and text model as claimed in claim 1, characterized in that: Construct a data set of image diagnosis instructions, including: Construct a pre-training data set in the railway field, including brief scene descriptions in the railway field, brief judgments on the quality of railway video surveillance images, management regulations related to railway surveillance, and news reports related to railway surveillance. All industry knowledge data exists in the form of picture-text pairs; Based on the industry knowledge dataset in the pre-training stage, a downstream task image diagnosis instruction dataset is constructed.

4. The image diagnosis method combining deep learning with a large image and text model as claimed in claim 1, characterized in that: The image diagnosis instruction data set includes simple and complex parts; The simple part is responsible for identifying whether there is any abnormality in the image quality of the monitoring screen, what aspects of the quality abnormality exist, and briefly explaining the severity of the abnormality; The complex part is deep logical reasoning. The reply needs to generate useful, harmless and concise information based on the input command prompt information; the command data is formed through multiple rounds of dialogue: self-instruct interacts with the SOTA's large graphic model, and the data is manually reviewed after generation; rethink the key aspects of the required reply and the corresponding capabilities of GPT-4V, and introduce prior supervision information through manual annotation prompts to enhance the accuracy of GPT-4V generation; high-quality detailed replies include image content, key distortions, the impact of distortion on content, and conclusions.

5. The image diagnosis method combining deep learning with a large image and text model as claimed in claim 1, characterized in that: Fine-tune visual instructions based on the image diagnosis instruction dataset, including: pre-training stage and fine-tuning stage; In the pre-training stage, the industry expertise is collected, cleaned and organized into image-text pairs, and the industry knowledge is injected into the image diagnosis instruction dataset in an incremental pre-training manner; In the fine-tuning phase, we construct question-answering data with different reasoning depths and covering all aspects of the task, and align them with task requirements in the form of visual instruction fine-tuning.

6. The image diagnosis method combining deep learning with a large image and text model as claimed in claim 5, characterized in that: During the fine-tuning phase, the parameters of the image encoder are kept unchanged, and the mapping layer and the large language model are trained; at the same time, efficient fine-tuning and distributed training are introduced.

7. The image diagnosis method combining deep learning with a large image and text model as claimed in claim 1, characterized in that: Perform alarm logic processing based on the corrected image diagnosis results, including: According to the definition of specific business requirements, in the fault category that relies on low-level visual features, for the "unbearable" part, if the preliminary image diagnosis results have been reported and the severity exceeds the acceptable threshold, an alarm will be directly issued; For the remaining images judged as "abnormal" by the preliminary image diagnosis results, they are analyzed by the image-text large model. The image-text large model is tuned to correct the preliminary image diagnosis results through the set instruction template and the preliminary image diagnosis results to obtain the corrected image diagnosis results. The corrected image diagnosis structure is used to decide whether to send an alarm signal according to the fault level threshold. The content of the instruction template includes context recognition, monitoring timestamp OCR recognition and OSD information.

8. A picture diagnosis system combining deep learning with a large picture and text model, characterized in that: include: The deep learning initial diagnosis module describes the image diagnosis task and defines the fault, constructs image diagnosis data based on decoupled modeling, and trains the corresponding image diagnosis algorithm to perform preliminary image diagnosis on the image; The data set construction module inputs the preliminary image diagnosis results into the general image and text model to generate responses, and constructs the image diagnosis instruction data set through manual verification, data cleaning and enhancement; The image-text large model tuning module fine-tunes the visual instructions based on the image diagnosis instruction dataset to obtain a tuned image-text large model that enhances the image diagnosis context and IQA capabilities. The tuned image-text large model corrects the preliminary image diagnosis results according to the scene conditions; The alarm logic judgment module performs alarm logic processing based on the corrected image diagnosis results, judges whether to alarm, and transmits the alarm signal to the operator.

9. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any one of the methods of claims 1 to 7.

10. A computing device, characterized in that include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any one of the methods described in claims 1 to 7.

Images