Image quality classification processing method and system based on multi-modal large model

Abstract

The invention discloses an image quality classification processing method and system based on a multi-modal large model, and relates to the technical field of electric power inspection image processing, and the method comprises the steps: determining a modal data matching strategy of an inspection image; obtaining various associated modal data based on a modal data matching strategy; performing matching judgment on each type of associated modal data and the inspection image to obtain primarily screened modal data; and determining effective information and fuzzy information of the inspection image, respectively pairing each type of preliminary screening modal data with the effective information and the fuzzy information, judging whether the preliminary screening modal data is fusion modal data based on the first pairing result and the second pairing result, and performing association labeling processing on the fusion modal data and the inspection image. According to the method and the system, matching judgment is carried out on the inspection images and the associated modal data, matching can be carried out according to the recognizable range of the flaw inspection images, and associated labeling processing is carried out according to the final matching result, so that the quality of synthetic data after fusion of different modal information is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of power inspection image processing, and in particular to an image quality classification processing method and system based on a multi-modal large model. Background Art

[0002] A multimodal large model refers to a machine learning model that can process and understand multiple types of data or data of the same type in different forms. By using a multimodal large model, power inspection can be highly automated and intelligent, and the combination of multimodal data helps to locate and classify problems more accurately, and can provide more accurate diagnostic results than a single mode, which is of great significance in power inspection applications. In the process of using a multimodal large model to fuse data of different modes for output, it depends on the correlation matching and synchronization consistency of these data. If the matching and synchronization cannot be guaranteed, it is possible that the multimodal large model will deviate from the training and learning method, or cause the accuracy of the results to be greatly reduced during judgment and analysis. Therefore, how to ensure that various modal data have complementary correlation characteristics, time synchronization characteristics, and spatial consistency characteristics when applying a multimodal large model has become a key point.

[0003] In the existing technology, multimodal large models are mostly used for image data enhancement. They use image data combined with input data from different shooting sources or angles to improve and enrich image processing tasks, that is, they use information from other modalities to provide a deeper understanding and support, and standardize other modal information (normalization, resizing, encoding conversion) for fusion analysis. During the whole process, simple matching and alignment operations are performed for different modal information only based on timestamps, spatial coordinates and other information. No further consideration is given to the matching degree between different modal information, how to more reliably classify and process defective information, and how to effectively label different modal information. As a result, the quality of the fused image synthesis data is not high, and there is a lot of noise, distortion or other forms of interference, which affects the accuracy of the final analysis output results.

[0004] In view of this, this application is hereby filed. Summary of the invention

[0005] The purpose of the present invention is to provide an image quality classification processing method and system based on a multimodal large model. The processing method and system extract other modal data sets associated with the inspection image, and then make a matching judgment according to each associated modal data. It can also pair them according to the identifiable range of the defective inspection image, and perform associated annotation processing according to the final pairing result to ensure the quality of the synthesized data after the fusion of different modal information.

[0006] The embodiment of the present invention is achieved as follows: In a first aspect, a method for image quality classification based on a multimodal large model comprises the following steps: identifying a target attribute of an inspection image, performing a preliminary analysis on the inspection image based on the target attribute, and determining a modal data matching strategy for the inspection image; screening an associated modal data set from an inspection database based on the modal data matching strategy, and classifying the associated modal data set into types to obtain a plurality of associated modal data; performing a matching judgment on each of the associated modal data with the inspection image to obtain a matching degree of the associated modal data, and using the associated modal data with a matching degree higher than a preset matching value as preliminary screening modal data, wherein the matching judgment includes: Inspection time judgment, inspection object judgment and inspection location judgment; delineate the identifiable range and unidentifiable range of the inspection image, extract valid information according to the identifiable range, extract fuzzy information according to the unidentifiable range, pair each type of preliminary screening modal data with the valid information and the fuzzy information respectively to obtain a first pairing result and a second pairing result; based on the first pairing result and the second pairing result, determine whether the preliminary screening modal data is used as fused modal data, and associate and annotate the fused modal data with the inspection image, wherein the associated annotation processing includes annotating associated areas, annotating associated objects and annotating associated forms.

[0007] In some optional implementations, the delineation of the recognizable range and the unrecognizable range of the inspection image, extracting valid information based on the recognizable range, and extracting fuzzy information based on the unrecognizable range includes the following steps: using a training model to analyze and determine the pixel quality of the inspection image, and identifying normal pixel coordinates and abnormal pixel coordinates; using all the normal pixel coordinates to construct the recognizable range, and using all the abnormal pixel coordinates to construct the unrecognizable range; performing preliminary identification of pixels within the recognizable range and obtaining covered target content, and obtaining the valid information based on the covered target content; performing preliminary identification of pixels within the unrecognizable range to obtain abnormal type content, and obtaining the fuzzy information based on the abnormal type content.

[0008] In some optional embodiments, the use of all the normal pixel coordinates to construct the recognizable range and the use of all the abnormal pixel coordinates to construct the unrecognizable range include the following steps: assigning morphological mask values ​​to each normal pixel coordinate and each abnormal pixel coordinate of the inspection image; constructing a first positive mask value distribution map and a first negative mask value distribution map according to the morphological mask values ​​of all normal pixel coordinates, and constructing a second positive mask value distribution map and a second negative mask value distribution map according to the morphological mask values ​​of all abnormal pixel coordinates; determining the recognizable range using the first positive mask value distribution map and the second negative mask value distribution map, and determining the unrecognizable range using the first negative mask value distribution map and the second positive mask value distribution map.

[0009] In some optional embodiments, the preliminary identification of pixels within the identifiable range and obtaining the covered target content includes the following steps: determining a first coverage area of ​​the identifiable range, performing feature extraction on a mask value distribution map of the first coverage area, detecting and identifying the extracted features, and obtaining the covered target content; the preliminary identification of pixels within the unidentifiable range and obtaining abnormal type content includes the following steps: determining a second coverage area of ​​the unidentifiable range and an intersection area where the unidentifiable range overlaps with the identifiable range; quantifying fuzziness for the mask value distribution within the second coverage area, performing cluster analysis on all quantified fuzziness information, and generating a fuzziness visualization distribution map; performing preliminary screening of abnormal types on the fuzziness visualization distribution map to obtain abnormal preliminary screening results; determining a first coverage area associated with the intersection area, and re-screening the abnormal preliminary screening results according to the associated first coverage area to obtain the abnormal type content.

[0010] In some optional embodiments, pairing each type of the preliminary screening modal data with the valid information and the fuzzy information respectively to obtain a first pairing result and a second pairing result includes the following steps: obtaining format conversion information of the preliminary screening modal data; performing a correlation analysis between the format conversion information and the mask value features within the first coverage area to obtain a first feature matching correlation degree, and calculating the first pairing result based on the first feature matching correlation degree; performing a correlation analysis between the format conversion information and the mask value features within the second coverage area to obtain a second feature matching correlation degree, and calculating the second pairing result based on the second feature matching correlation degree; wherein, after obtaining the second feature matching correlation degree, it also includes: performing a correlation analysis between the format conversion information and the mask value features within the intersection area to obtain a third feature matching correlation degree, and calculating the second pairing result based on the second feature matching correlation degree and the third feature matching correlation degree.

[0011] In some optional embodiments, the calculation of the second pairing result based on the second feature matching correlation degree and the third feature matching correlation degree includes the following steps: determining a first weight and a second weight; comparing the difference between the second feature matching correlation degree and the third feature matching correlation degree, and adjusting the first weight and / or the second weight based on the difference; assigning the adjusted first weight to the second feature matching correlation degree and / or the second weight to the third feature matching correlation degree.

[0012] In some optional embodiments, the associating and annotating processing of the fused modal data and the inspection image includes the following steps: determining the associated area position information of the inspection image and the fused modal data; locking the associated objects of the inspection image and the fused modal data respectively based on the associated area position information; judging the type of association form based on the associated objects of the inspection image and the fused modal data; wherein the type of association form includes one of full association, strong association and weak association.

[0013] In some optional embodiments, the method of determining the type of association form based on the associated objects of the inspection image and the fused modal data includes the following steps: calculating the similarity between the inspection image and any associated object of the fused modal data, merging the similarities between the inspection image and all associated objects of the fused modal data to obtain a total similarity, performing interval comparison based on the total similarity, and determining the type of association form.

[0014] In some optional implementations, the method further includes the step of adjusting the type of the association form: tracing back a second pairing result of the fused modal data, calculating the second pairing result to determine an association adjustment parameter, assigning the association adjustment parameter to the total similarity to obtain an adjusted similarity, and redetermining the type of the association form based on the adjusted similarity.

[0015] In a second aspect, an image quality classification processing system based on a multimodal large model comprises: A first recognition unit, which is used to recognize the target attribute of the inspection image, perform a preliminary analysis on the inspection image based on the target attribute, and determine a modal data matching strategy for the inspection image; A second identification unit, which is used to screen a related modal data set from the inspection database based on the modal data matching strategy, classify the related modal data set into types, and obtain a plurality of related modal data; A first processing unit, which is used to perform matching judgment between each type of the associated modal data and the inspection image, obtain the matching degree of the associated modal data, and use the associated modal data with a matching degree higher than a preset matching value as the primary screening modal data, wherein the matching judgment includes inspection time judgment, inspection object judgment and inspection location judgment; a second processing unit, which is used to define a recognizable range and an unrecognizable range of the inspection image, extract valid information according to the recognizable range, extract fuzzy information according to the unrecognizable range, and pair each of the primary screening modal data with the valid information and the fuzzy information, respectively, to obtain a first pairing result and a second pairing result; A third processing unit is used to determine whether the initial screening modal data is used as fused modal data based on the first pairing result and the second pairing result, and to associate and annotate the fused modal data with the inspection image, wherein the association annotation processing includes annotating associated areas, annotating associated objects, and annotating associated forms.

[0016] The beneficial effects of the embodiments of the present invention are: The image quality classification processing method and system based on the multimodal large model provided by the embodiment of the present invention can determine the modal data type that can be matched based on the target attribute by preliminarily identifying the target attribute of the inspection image, and directly extract different categories of related modal data from the inspection database according to the matching strategy corresponding to the modal data type that can be matched. This related modal data has the basis for data fusion analysis with the inspection image; then, by judging the matching of each related modal data with the inspection image, the matching degree judgment operation between different modal information is performed from multiple aspects such as whether the inspection time is consistent, whether the inspection object is consistent, and whether the inspection location is consistent. On this basis, further analysis can be performed on whether there are any defect types in the unrecognizable range in the inspection image, that is, by matching the associated modal data with the recognizable range and the unrecognizable range respectively to deal with the unrecognizable problems such as occlusion, exposure, and blurred shooting in the inspection image, so that the inspection image in the defective situation can be more accurately matched with the screened preliminary screening modal data, and the preliminary screening modal data with a higher final matching degree can be used as the fused modal data, so that the fused modal data and the inspection image are more conducive to the subsequent training or analysis and judgment of the multimodal large model, thereby ensuring the quality of the input image data of the multimodal large model and obtaining more accurate output results.

[0017] In general, the image quality classification processing method and system based on the multimodal large model provided in the embodiments of the present invention can analyze and judge whether the associated modal data is preliminarily matched and whether it can be highly matched with the defect inspection image, so as to associate and annotate the finally screened fused modal data with the inspection image, so as to achieve different levels of classification and judgment processing from the initial associated modal data to the final fused modal data, so as to ensure the data quality of the final multimodal large model input image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of the main steps of the processing method provided by an embodiment of the present invention; Figure 2 for Figure 1 A flowchart of the first part of steps of step S400, one of the main steps shown; Figure 3 for Figure 2 The sub-step flow chart of S420 in the first part of the steps is shown; Figure 4 for Figure 2 The sub-step flow chart of S430 and S440 in the first part of the steps is shown; Figure 5 for Figure 1 A flowchart of the second part of steps of step S400, one of the main steps shown; Figure 6 for Figure 1 A flow chart of one of the main steps, step S500; Figure 7 for Figure 6 The flowchart of sub-steps of step S530 in step S500 is shown; Figure 8 A modular schematic diagram of a processing system provided by an embodiment of the present invention.

[0020] Icon: 600 - processing system; 610 - first identification unit; 620 - second identification unit; 630 - first processing unit; 640 - second processing unit; 650 - third processing unit. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be understood that the "system", "device" and / or "module" used in the present invention is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0024] As shown in the present invention and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular, but also include the plural. In general, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list, and the method or device may also include other steps or elements.

[0025] Flowcharts are used in the present invention to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0026] Embodiment: When preprocessing the input data of a multimodal large model, the data of different modes are classified and screened, and then the data is fused as the model input. In this process, how to classify and screen the data of different modes becomes a key issue. Among them, data of different modes can be different forms of the same data (such as image data of different shooting angles and shooting conditions), or different types of data (such as image data, voice data, and sensor data, etc.). At present, the most widely used is image data. Multimodal learning or analysis and judgment is performed based on the image data sent back by the inspection robot (ground robot or aerial drone), which means that the image data of different modes are classified and fused as the training data set or analysis and judgment data set of the multimodal large model.

[0027] In the above operation mode, we classify and process the captured image data according to the source of the image data and the device identification number of the captured target, and then form fused data of the same type as the model input training set and test set, which can ensure a certain degree of judgment accuracy, but there will still be some misjudgments. After in-depth analysis, it is found that the quality of the input fused data is the basis for guaranteeing the accuracy of the final analysis results, that is, it is necessary to consider how to ensure the quality of the fused data with higher matching. Therefore, how to achieve the classification and screening of multimodal data in the early stage and the fusion and annotation in the later stage needs to be further optimized, that is, considering the matching degree, consistency, and annotation relevance between different modal data. On this basis, we also found that once the fused data formed when there are defects in the captured image data (such as target occlusion, unstable lighting, blurred shooting, etc.), the accuracy of the subsequent analysis results will be greatly reduced. In view of this situation, it is also necessary to consider how to ensure that the key defective image data can also be classified and screened with higher matching degree, consistency, and annotation relevance, which is also one of the key issues. In response to the related issues discussed above, this embodiment provides an image quality classification and processing method based on a multimodal large model. This processing method can perform strong correlation matching between the image data of the main application and other different modal data, and also considers the correlation matching of defective image data during the matching process, so that the screened out other modal data can be effectively labeled with the image data, thereby ensuring the quality of the fused synthetic data.

[0028] Please refer to Figure 1 The present embodiment provides an image quality classification processing method based on a multimodal large model, comprising the following steps: S100: Identify the target attributes of the inspection image, perform preliminary analysis on the inspection image based on the target attributes, and determine the modal data matching strategy of the inspection image; this step represents the target attribute recognition of the inspection image to be used as one of the basis of the training set or the test set. The target attribute recognition is mainly for preliminary recognition of the attributes (type, address, power-on status, etc.) of the target device to be monitored in the inspection image. The identified target attribute data can be uploaded to the background server by the inspection device when uploading the image data, so as to facilitate the background server to extract strategies or plans based on the target attributes, so as to determine the modal data matching strategy of the inspection image. The modal data matching strategy mainly refers to the modal data type that can be matched, the data source address, and the strategy plan formed for the target and other contents. The strategy plan can be configured in advance or formed by combining pre-configuration and subsequent dynamic adjustment. It is not limited here.

[0029] S200: Based on the modal data matching strategy, an associated modal data set is screened from the inspection database, and the associated modal data set is classified into types to obtain a plurality of associated modal data. This step indicates that the modal data matching strategy obtained through the above steps screens out all data that can be associated with the target attribute of the inspection image from the inspection database received from the background server as an associated modal data set, and then the associated modal data set is classified into types. Specifically, taking the associated modal data set as an image data set (the same below) as an example, due to the differences in shooting time, shooting position, shooting angle, shooting target coverage and shooting method of different inspection devices, the same power equipment may have different modal image data categories, and these different modal image data categories need to be classified, for example, according to the same shooting time, shooting position, shooting angle, shooting target coverage and shooting method, so as to obtain a plurality of associated modal data. Different associated modal data may differ in at least one of the shooting time, shooting position, shooting angle, shooting target coverage and shooting method.

[0030] S300: Perform a matching judgment on each of the associated modal data and the inspection image to obtain the matching degree of the associated modal data, and use the associated modal data with a matching degree higher than a preset matching value as the initial screening modal data, wherein the matching judgment includes inspection time judgment, inspection object judgment and inspection location judgment; this step means that all the different types of associated modal data initially extracted are respectively matched with the inspection image to mainly judge whether the inspection time is consistent, whether the inspection object is consistent and whether the inspection location is consistent (in different implementations, it can be further subdivided into whether the inspection angle is consistent, whether the inspection method is consistent, etc.). After matching and judging the inspection time, inspection object and inspection location, the matching degree of each associated modal data is obtained, and the matching degree is compared with the preset matching value. The associated modal data with a matching degree higher than the preset matching value is used as the initial screening modal data, that is, as the object of initial matching, the subsequent classification processing step can be performed.

[0031] S400: Delineate the recognizable range and unrecognizable range of the inspection image, extract valid information according to the recognizable range, extract fuzzy information according to the unrecognizable range, pair each of the primary screening modal data with the valid information and the fuzzy information, respectively, to obtain a first pairing result and a second pairing result; this step indicates that by judging whether the inspection image has defects, it is considered in advance whether it will have a significant impact when matching and judging with the associated modal data, so as to cope with the problem of relatively reduced quality of later fusion data, that is, the recognizable range and unrecognizable range on the inspection image are delineated respectively. If there is an unrecognizable range, it is considered that the inspection image has defects (occlusion, exposure, blur, etc.), then when pairing with the corresponding primary screening modal data, it is necessary to consider whether the recognizable range is highly matched and whether the unrecognizable range is highly matched, so as to ensure that the two can serve as the basis of fusion modal data with high matching and consistency.

[0032] Specifically, it is necessary to extract valid information from the recognizable range of the inspection image and extract fuzzy information from the unrecognizable range, and then pair each of the preliminary screening modal data with the valid information and the fuzzy information respectively to determine whether the shooting content is highly consistent, thereby obtaining a first pairing result corresponding to the valid information and a second pairing result corresponding to the fuzzy information respectively.

[0033] S500: Based on the first pairing result and the second pairing result, determine whether the initial screening modal data is used as fused modal data, and associate and annotate the fused modal data with the inspection image, wherein the association annotation processing includes annotating the associated area, annotating the associated object, and annotating the associated form; this step means that the first pairing result and the second pairing result are comprehensively considered, and then the result of the combination of the two is combined to determine whether the initial screening modal data is used as fused modal data. The fused modal data refers to the basis that can enter the same classification as the inspection image and can be used for fusion data synthesis (with high consistency), so that the obtained fused modal data can ensure that the synthesized image data has high quality, so as to serve as input data for training or analysis and judgment of the multimodal large model. It should be noted that before being used as input data, the fused modal data and the inspection image need to be associated and annotated, mainly including the information annotation of the associated area (coordinate area in the image), the associated object (target object in the image) and the associated form (degree of association), combined with the image data and accurate annotation information, so as to improve the accuracy of the output result of the multimodal large model.

[0034] Through the above technical solution, in order to ensure the quality and relevance of image data in the multimodal large model, so as to improve the accuracy of multimodal large model training and analysis judgment. First, the target attributes of the inspection image are identified, and the modal data matching strategy is determined based on this to ensure the pertinence of the initial screening. Then, the modal data sets related to the target attributes are screened out from the database and divided into types to ensure the effective separation of different modal data. Through accurate matching judgment (including consistency check of time, object and location), the modal data with high matching degree are screened out as the initial screening result. On this basis, in view of possible image defects, the recognizable range and unrecognizable range of the inspection images are delineated, and the initial screening modal data are paired with them to ensure that a high degree of matching and consistency can be maintained even if there are image quality problems. Finally, the pairing results are comprehensively considered to decide which initial screening modal data can be used as fused modal data, and they are associated and annotated to ensure the quality and availability of the final synthesized data. This not only optimizes the classification and screening of multimodal data, but also effectively solves the problem of data quality degradation caused by image defects, providing high-quality training and test data for large multimodal models, thereby improving the reliability of model output.

[0035] On the basis of the above technical solution, the process of defining the recognizable range and the unrecognizable range of the inspection image to reflect whether the inspection image has defects is mainly achieved by processing the pixels. For details, please refer to Figure 2 The step of defining the recognizable range and the unrecognizable range of the inspection image, extracting valid information according to the recognizable range, and extracting fuzzy information according to the unrecognizable range comprises the following steps: S410: Analyze and judge the pixel quality of the inspection image using the training model, and identify the coordinates of normal pixel points and abnormal pixel points. This step means using a preliminary judgment model to perform a preliminary analysis on the quality of the pixel points of the inspection image, such as using a convolutional neural network model to analyze each inspection image pixel by pixel or small block of pixels by means of a sliding window, and obtain a feature vector for each position. Based on feature extraction, a classification layer is added to distinguish between normal pixels and abnormal pixels, and which pixels are marked as abnormal and which are marked as normal are determined by a set threshold, thereby obtaining the coordinates of normal pixel points and abnormal pixel points.

[0036] S420: Use all the normal pixel coordinates to construct the recognizable range, and use all the abnormal pixel coordinates to construct the unrecognizable range; this step means that all the normal pixel coordinates obtained are connected in series to construct the pixel area of ​​the recognizable range, and all the abnormal pixel coordinates are connected in series to construct the pixel area of ​​the unrecognizable range. It should be noted that the recognizable range and the unrecognizable range can be an integrally connected area or multiple segmented areas. The following analysis will be taken as an example of one area of ​​each recognizable range or unrecognizable range (the same applies to the remaining areas).

[0037] S430: Perform preliminary identification on the pixels within the identifiable range and obtain the covered target content, and obtain the valid information based on the covered target content; this step indicates obtaining the covered target content within the identifiable range based on target detection (using the model to output the location information and category label of each detected target) and semantic segmentation technology (obtaining the outline of each detected target and its category information), and the target type, outline and position of the target content are used as the valid information.

[0038] S440: Perform preliminary recognition on the pixels within the unrecognizable range to obtain abnormal type content, and obtain the blurred information based on the abnormal type content; this step means that the target detection technology is also used to extract the feature contour and position of the unrecognizable range, and then the possible target type (blurred body, occluded body, exposed body, etc.) can be output as the abnormal type content through field judgment (context judgment).

[0039] Through the above technical solution, the training model is used to conduct a detailed analysis of the pixel quality of the inspection image, distinguish the coordinates of normal pixels and abnormal pixels, separate the clear part of the image from the defective part, and provide a basis for targeted analysis. Then, for the pixels within the identifiable range, target detection and semantic segmentation technology are used to accurately identify and obtain information covering the target content, ensuring a detailed understanding of the key elements in the image; and for the pixels within the unidentifiable range, the abnormal type content is determined through preliminary identification combined with contextual judgment, and the information of the blurred or occluded area is analyzed as much as possible, thereby providing a more detailed pairing comparison basis for the subsequent initial screening modal data.

[0040] Considering that there may be three judgment situations when analyzing and judging the pixel quality of the inspection image using the training model, that is, the pixel coordinates are normal pixel coordinates, abnormal pixel coordinates or pixel coordinates that cannot be judged, there will be a blind spot in the process of delineating the recognizable range and the unrecognizable range (neither belonging to the recognizable range nor the unrecognizable range). In order to cope with this situation, it is ensured that the information (texture features) contained in each pixel can be utilized, especially for the subsequent preliminary screening modal data pairing process. Therefore, in this embodiment, the recognizable range and the unrecognizable range are delineated by the positive and negative union division method. For details, please refer to Figure 3 The steps of using all the normal pixel coordinates to construct the recognizable range and using all the abnormal pixel coordinates to construct the unrecognizable range include the following steps: S421: Assign morphological mask values ​​to each normal pixel coordinate and each abnormal pixel coordinate of the inspection image respectively; this step means assigning recognizable numerical values ​​to each normal pixel coordinate and each abnormal pixel coordinate as morphological mask values. The morphological mask values ​​are generally of two positive and negative types, which can be 0 or 1, or 1 and -1, or other numerical combinations, etc. The purpose is to distinguish between the types of normal pixel coordinates and abnormal pixel coordinates.

[0041] S422: construct a first positive mask value distribution map and a first negative mask value distribution map according to the morphological mask values ​​of all normal pixel coordinates, and construct a second positive mask value distribution map and a second negative mask value distribution map according to the morphological mask values ​​of all abnormal pixel coordinates; this step means constructing a distribution map of the morphological mask values ​​of all pixel coordinates on the inspection image, if it is a normal pixel coordinate, the first positive mask value (1) is assigned, and the remaining pixel coordinates are assigned the first negative mask value (-1), thereby constructing a first positive mask value distribution map and a first negative mask value distribution map, which are opposite and complementary morphological mask value distribution maps.

[0042] Similarly, if it is determined that the pixel coordinates are abnormal, the second positive mask value (-1) is assigned, and the other pixel coordinates are assigned the second negative mask value (1), thereby constructing a second positive mask value distribution map and a second negative mask value distribution map, both of which are also opposite and complementary morphological mask value distribution maps.

[0043] S423: Determine the recognizable range using the first positive mask value distribution map and the second reverse mask value distribution map, and determine the unrecognizable range using the first reverse mask value distribution map and the second positive mask value distribution map; this step indicates that the recognizable range is determined by taking the union of the first positive mask value distribution map (distribution map of the value 1) and the second reverse mask value distribution map (distribution map of the value 1). Similarly, the unrecognizable range is determined by taking the union of the first reverse mask value distribution map (distribution map of the value -1) and the second positive mask value distribution map (distribution map of the value -1). This positive and reverse union method can eliminate the situation where there are mask values ​​of pixel coordinates that cannot be determined, thereby facilitating subsequent pixel feature matching.

[0044] Through the above technical solution, the identifiable range and the unidentifiable range can be combined to completely cover the combined area of ​​the entire inspection image, and the boundary between the identifiable range and the unidentifiable range is the coordinate position of the above-mentioned undeterminable pixel type. These pixel points cannot accurately determine their texture features individually, but combined with the surrounding neighborhood pixels, their texture features can be further inferred, so as to more reliably determine the abnormal type implied by the fuzzy texture features of the pixels in the unidentifiable range. Therefore, when extracting fuzzy information, it is necessary to consider the available information contained in the boundary between the identifiable range and the unidentifiable range. For details, please refer to Figure 4 The preliminary identification of the pixels within the identifiable range and obtaining the covered target content includes the following steps: S431: Determine the first coverage area of ​​the identifiable range, perform feature extraction on the mask value distribution map of the first coverage area, detect and identify the extracted features, and obtain the covered target content; this step means determining the mask values ​​of all pixels in the pixel coverage area of ​​the identifiable range (boundary box determination), and then performing feature extraction, detection and identification (such as target detection model and semantic segmentation model) according to the distribution of the mask values. The extracted feature information and category labels are used as the covered target content.

[0045] Please refer again Figure 4 The preliminary identification of the pixels within the unrecognizable range to obtain abnormal type content includes the following steps: S441: Determine the second coverage area of ​​the unrecognizable range and the intersection area where the unrecognizable range overlaps with the recognizable range; this step means separately determining the pixel coverage area of ​​the unrecognizable range (as the second coverage area) and the pixel coverage area where the recognizable range overlaps with the recognizable range (as the intersection area). Since the recognizable range and the unrecognizable range are constructed by the aforementioned positive and negative union method, there are intersection ranges between the two, and these intersection ranges are composed of pixel coordinates whose types cannot be accurately determined.

[0046] S442: quantify the fuzziness of the mask value distribution in the second coverage area, perform cluster analysis on all quantified fuzziness information, and generate a fuzziness visualization distribution map; this step represents the numerical quantization of the fuzziness of the mask value distribution map in the second coverage area. The fuzziness value can be any value between 0 and 1, or other range data. The purpose is to convert and represent it according to the confidence value in the pixel type judgment process, and obtain the specific confidence value of each pixel type that cannot be accurately judged, so as to convert and represent the fuzziness value of the pixel. Then, cluster analysis is performed on the fuzziness information of all pixel coordinates to generate a fuzziness visualization distribution map, so that the fuzzy distribution of all pixels in the second coverage area can be intuitively obtained.

[0047] S443: Perform a preliminary screening of abnormal types on the fuzziness visualization distribution map to obtain abnormal preliminary screening results; this step indicates that a preliminary screening operation of possible abnormal types is performed based on the obtained visualization distribution map. For example, based on the concentration of pixel fuzziness and the range of fuzziness values, it is judged whether there is occlusion (the overall fuzziness value is concentrated and there is occlusion texture), exposure (the overall value is concentrated and there is no obvious texture) or blur (the fuzziness value is dispersed and there are multiple textures), etc., so as to obtain possible situations of abnormal preliminary screening results.

[0048] S444: Determine the first coverage area associated with the intersection area, and re-screen the abnormal initial screening results according to the associated first coverage area to obtain the abnormal type content. This step means that the intersection area is associated with the (directly adjacent) first coverage area and its analyzed coverage target content is used to assist the re-screening and judgment of the abnormal initial screening results, further excluding the possibility of occlusion, exposure, and blur, and finally obtaining relatively certain abnormal type content.

[0049] Through the above technical scheme, combined with the analysis of the recognizable range and the unrecognizable range, for the clearly identifiable target area (the first coverage area), feature extraction, detection and recognition are used to obtain the covered target content; and for the fuzzy area that cannot be directly identified (the second coverage area) and the junction of the two, fuzzy quantification, cluster analysis and visualization distribution map are introduced to evaluate the abnormal type of pixels in these areas. By associating the first coverage area information in the intersection area, the judgment accuracy of the abnormal type in the unrecognizable range is further optimized, which not only ensures the effective analysis of the entire inspection image, but also uses the neighborhood information of difficult-to-judge pixels to assist decision-making, thereby improving the recognition ability of occlusion, poor exposure or real blur.

[0050] By obtaining the recognizable range and its effective information and the unrecognizable range and its fuzzy information of the inspection image, it can be further paired with the preliminary screening modal data, thereby avoiding the situation where the preliminary screening modal data cannot have a high match with the defective area due to the defective area in the inspection image, thereby reducing the quality of the subsequent fusion data. Therefore, it is necessary to consider pairing the preliminary screening modal data with effective information and fuzzy information respectively, and only after the pairing results of both meet the requirements can we further consider whether to use it as the supporting fusion data of the inspection image. For details, please refer to Figure 5 The step of pairing each of the primary screening modal data with the valid information and the fuzzy information to obtain a first pairing result and a second pairing result comprises the following steps: S450: Obtain the format conversion information of the initial screening modal data; this step means that the initial screening modal data is first format converted. Taking the initial screening modal data as image data as an example, the format conversion includes steps such as image format standard conversion, size adjustment (scaling, cropping), space conversion processing (color, pixel value standardization), metadata processing, etc., so as to obtain the format conversion information through the above-mentioned format conversion processing. If the initial screening modal data is voice data, the format conversion includes steps such as file format standardization, encoding quality adjustment, feature generation and normalization, and voice file (with image data) time and space synchronization processing to obtain the format conversion information. This embodiment mainly takes the initial screening modal data as image data as an example to illustrate, and classifies and analyzes other modal image data with different shooting angles, shooting methods and shooting times with the inspection images.

[0051] S460: Perform correlation analysis on the format conversion information and the mask value features in the first coverage area to obtain a first feature matching correlation degree, and calculate the first pairing result based on the first feature matching correlation degree; this step means performing feature extraction (including metadata records) on the obtained format conversion information and performing correlation analysis with the mask value features in the first coverage area (local feature extraction based on the bounding box), calculating the similarities of all aligned features using feature alignment (ensuring that the image features after format conversion and the original mask value features are compared in the same coordinate system) and similarity measurement, thereby obtaining a first feature matching correlation degree (for example, a weighted score calculation method), and finally calculating the first pairing result based on the first feature matching correlation degree.

[0052] S470: Perform correlation analysis on the format conversion information and the mask value features in the second coverage area to obtain a second feature matching correlation degree, and calculate the second pairing result according to the second feature matching correlation degree; this step means that the obtained format conversion information is also subjected to feature extraction (including metadata recording) and correlation analysis with the mask value features in the second coverage area (local feature extraction based on the bounding box), and the mask value features that can be extracted in the second coverage area are also subjected to feature alignment and similarity measurement with the extracted features of the format conversion information, so as to obtain a second feature matching correlation degree, and then the second pairing result is calculated according to the second feature matching correlation degree. Since the mask value features that can be extracted in the second coverage area may be limited, the second pairing result may be insufficiently judged if it is calculated only based on the mask value features in the second coverage area. Therefore, on this basis, it is necessary to further consider correlation analysis with the mask value features in the intersection area, that is, to perform the following steps.

[0053] S480: After obtaining the second feature matching correlation, it also includes: performing a correlation analysis on the format conversion information and the mask value feature in the intersection area to obtain a third feature matching correlation, and calculating the second pairing result according to the second feature matching correlation and the third feature matching correlation; this step indicates that the obtained format conversion information is also subjected to feature extraction (including metadata records) and a correlation analysis with the mask value feature in the intersection area to obtain the third feature matching correlation. It should be noted that the third feature matching correlation covers clearer texture features, which can assist in judging whether the mask value features in the second coverage area are highly correlated and paired. Therefore, the second pairing result is calculated according to the second feature matching correlation and the third feature matching correlation, so that the second pairing result has a more reasonable basis for judgment, and can avoid the problem that the pairing result cannot be accurately calculated due to unclear mask value features in the two coverage areas.

[0054] Through the above technical solution, the format conversion processing is performed on the initial screening modal data to obtain its format conversion information, and then the format conversion information is respectively analyzed for correlation with the mask value features in the first coverage area (valid information) and the second coverage area (fuzzy information) defined in the inspection image to calculate the first and second feature matching correlation, and then obtain the corresponding pairing results. Considering that the second coverage area may have unclear features, the mask value features of the cross area are further introduced for auxiliary analysis to improve the accuracy of the second pairing result, which not only enhances the matching accuracy between different modal data, but also ensures that reliable data fusion quality can be obtained even in the presence of defects.

[0055] On this basis, if the mask value feature texture of the intersection area is clearer, it can assist the high-precision calculation of the second pairing result. Therefore, when calculating the second pairing result based on the second feature matching correlation degree and the third feature matching correlation degree, it is necessary to consider the numerical data represented by the second feature matching correlation degree and the third feature matching correlation degree. The one with the greater similarity can be given a greater weight in the calculation. Specifically, the calculation of the second pairing result based on the second feature matching correlation degree and the third feature matching correlation degree includes the following steps: Determine a first weight and a second weight, which can be initial assigned values ​​and can be obtained based on experience; then compare the difference between the second feature matching correlation degree and the third feature matching correlation degree to clarify the size of the difference between the two, and then adjust the first weight and / or the second weight according to the difference, for example, the larger the value of whichever is larger, the larger the upward adjustment range, and this goal can be achieved by adjusting one of them, or by adjusting both at the same time; finally, assign the adjusted first weight to the second feature matching correlation degree and / or the second weight to the third feature matching correlation degree, that is, whichever is adjusted, the adjusted first weight is assigned to the adjuster, so as to calculate the final second pairing result.

[0056] Through the above series of operation steps, the first pairing result and the second pairing result of the initial screening modal data and the recognizable range and unrecognizable range of the inspection image can be obtained, and then the first pairing result and the second pairing result are used to comprehensively judge whether the initial screening modal data is used as fused modal data. If the first pairing result and the second pairing result both meet the requirements, the initial screening modal data can be used as fused modal data. It should be noted that due to the defects (occlusion, exposure, blur, etc.) in the unrecognizable range of the inspection image, if the initial screening modal data (such as image data) also covers the same area, but the features of the initial screening modal data in this area are clearer, and can also be highly matched with the features that may exist in the unrecognizable range (using the features of the above-mentioned intersection area for auxiliary judgment), it means that the corresponding second pairing result is more likely to meet the requirements, and the initial screening modal data can be used as fused modal data.

[0057] After the fused modal data is obtained, it means that the original associated modal data and the inspection image are highly matched in terms of matching and consistency. In order to facilitate the multimodal large model to identify the semantic label of the feature when performing feature detection, so as to obtain a more accurate output analysis result, it is necessary to associate and annotate the two before data synthesis. For details, please refer to Figure 6 The process of associating and labeling the fused modal data with the inspection image comprises the following steps: S510: Determine the location information of the associated area between the inspection image and the fused modal data; this step means first locating the specific location of the inspection image in the associated area of ​​the fused modal data, then the location information of the area covering the same detection target is located, and then further target association annotation is performed, that is, step S520: Based on the location information of the associated area, the associated objects of the inspection image and the fused modal data are locked respectively; this step means locking the object targets that are mutually associated with the two in the associated area position respectively, which can be one or more, and the associated objects are respectively annotated for association. Finally, step S530 is performed: based on the associated objects of the inspection image and the fused modal data, the type of association form is determined; this step means further annotating the association form on each group of associated objects to indicate the degree of association. The type of association form includes one of full association, strong association and weak association. Different types of association forms indicate the strength of the association degree between the corresponding associated objects, so as to facilitate the subsequent distinction in the multi-modal large model analysis.

[0058] Through the above technical solution, after determining that the initial screening modal data can be used as fused modal data, further association annotation processing is performed to ensure that the synthesized data has clear semantic labels, and by locating the location information of the associated area, locking the associated objects, and judging and annotating the type of association form (such as full association, strong association or weak association), it provides structured data that is easy to parse and process for the multimodal large model. On the basis of the above technical solution, if there are multiple groups of associated objects in the same associated area, it is necessary to consider the degree of association of each group separately, so as to finally judge the type of association form between the fused modal data and the inspection image regarding the associated area. For details, please refer to Figure 7 The method of determining the type of association form based on the associated object between the inspection image and the fusion modality data comprises the following steps: S531: Calculate the similarity between the inspection image and any associated object of the fused modal data, merge the similarities between the inspection image and all associated objects of the fused modal data, and obtain the total similarity; this step means judging the similarity between the inspection image and each group of associated objects of the fused modal data (such as using a feature similarity measurement method), and then merge the similarities of all groups of associated objects (such as averaging or weighted averaging), and finally obtain the total similarity between the inspection image and the fused modal data with respect to the associated objects in the corresponding associated area. Then proceed to step S532: perform interval comparison based on the total similarity (configure an interval comparison table), and use the total similarity to determine the type of association form of the corresponding associated area, whether it is full association, strong association or weak association.

[0059] Considering that when determining the associated area of ​​the inspection image, the associated area may exist in the unrecognizable range of the inspection image, there are certain obstacles in judging the similarity of the associated objects in the area, and it is impossible to accurately judge the corresponding type of the associated form. At this time, the second pairing result can be used to assist in judging the degree of similarity, that is, after the step of judging the type of the associated form, the step of adjusting the type of the associated form is also included: S533: tracing back the second pairing result of the fused modal data, calculating the second pairing result to determine the associated adjustment parameter, assigning the associated adjustment parameter to the total similarity to obtain the adjusted similarity, and redetermining the type of the associated form based on the adjusted similarity; this step represents tracing back the second pairing result calculated in the aforementioned judgment step, determining (such as empirically) an associated adjustment parameter according to the numerical value of the second pairing result, and then assigning (such as by summing) the associated adjustment parameter to the total similarity, and finally obtaining the adjusted similarity. The adjusted similarity can assist the judgment result of the similarity of the associated objects within the unrecognizable range, so that it is closer to the real analysis result, avoiding the problem of inaccurate calculation, so that the final adjusted similarity is used to redetermine the type of the associated form with more rationality and reliability.

[0060] Through the above technical scheme, in the case where there are multiple groups of associated objects in the same associated area, the similarity of each group of associated objects can be calculated and combined to obtain the total similarity, and then the type of association form (such as full association, strong association or weak association) can be determined based on the total similarity, thereby providing structured and easy-to-parse data for the multimodal large model. Secondly, for associated objects within the unrecognizable range in the inspection image, the second pairing result is introduced as an auxiliary judgment means. By calculating the association adjustment parameter and applying it to the total similarity, the adjusted similarity is obtained, and the type of association form is re-determined. This not only overcomes the difficulty of similarity judgment caused by image defects, but also improves the accuracy and reliability of association form judgment, ensuring the authenticity and rationality of the final analysis result.

[0061] This embodiment also provides an image quality classification processing system based on a multi-modal large model (hereinafter referred to as the processing system 600), see Figure 8 The modular schematic diagram of the processing system 600 is mainly used to divide the processing system 600 into functional modules according to the embodiment of the above method. For example, each functional module can be divided, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the present invention is schematic and is only a logical functional division. There may be other division methods in actual implementation. For example, in the case of dividing each functional module corresponding to each function, Figure 8 What is shown is only a schematic diagram of a system / device, wherein the processing system 600 may include a first identification unit 610, a second identification unit 620, a first processing unit 630, a second processing unit 640 and a third processing unit 650. The functions of each unit module are described below.

[0062] A first recognition unit 610, which is used to recognize the target attribute of the inspection image, perform a preliminary analysis on the inspection image based on the target attribute, and determine a modal data matching strategy for the inspection image; A second identification unit 620 is used to filter the associated modal data set from the inspection database based on the modal data matching strategy, classify the associated modal data set into types, and obtain multiple types of associated modal data; A first processing unit 630 is used to perform matching judgment between each of the associated modal data and the inspection image, obtain the matching degree of the associated modal data, and use the associated modal data with a matching degree higher than a preset matching value as the primary screening modal data, wherein the matching judgment includes inspection time judgment, inspection object judgment and inspection location judgment; The second processing unit 640 is used to define the recognizable range and the unrecognizable range of the inspection image, extract valid information according to the recognizable range, extract fuzzy information according to the unrecognizable range, and pair each of the primary screening modal data with the valid information and the fuzzy information, respectively, to obtain a first pairing result and a second pairing result; in some embodiments, the second processing unit 640 is also used to use the training model to analyze and determine the pixel quality of the inspection image, identify the normal pixel point coordinates and the abnormal pixel point coordinates; use all the normal pixel point coordinates to construct the recognizable range, and use all the abnormal pixel point coordinates to construct the unrecognizable range; for the The pixel points within the identifiable range are preliminarily identified and the target content is obtained, and the valid information is obtained based on the target content; the pixel points within the unidentifiable range are preliminarily identified to obtain the abnormal type content, and the fuzzy information is obtained based on the abnormal type content; and a morphological mask value is assigned to each normal pixel coordinate and each abnormal pixel coordinate of the inspection image; a first positive mask value distribution map and a first reverse mask value distribution map are respectively constructed according to the morphological mask values ​​of all normal pixel coordinates, and a second positive mask value distribution map and a second reverse mask value distribution map are respectively constructed according to the morphological mask values ​​of all abnormal pixel coordinates; the first positive mask value distribution map and the second reverse mask value distribution map are respectively constructed using the first positive mask value distribution map and the The second inverse mask value distribution map is used to determine the recognizable range, and the first inverse mask value distribution map and the second positive mask value distribution map are used to determine the unrecognizable range; and a first coverage area for determining the recognizable range is performed, and features are extracted from the mask value distribution map of the first coverage area, and the extracted features are detected and recognized to obtain the covered target content; a second coverage area of ​​the unrecognizable range and an overlapping area of ​​the unrecognizable range and the recognizable range are determined; fuzziness is quantified for the mask value distribution in the second coverage area, and all quantified fuzziness information is clustered to generate a fuzziness visualization distribution map; the fuzziness visualization distribution map is analyzed. The layout performs a preliminary screening of abnormal types to obtain a preliminary screening result of the abnormalities; determines a first coverage area associated with the intersection area, and re-screens the preliminary screening result of the abnormalities according to the associated first coverage area to obtain the abnormal type content; is also used to obtain format conversion information of the preliminary screening modal data; performs a correlation analysis between the format conversion information and the mask value feature in the first coverage area to obtain a first feature matching correlation degree, and calculates the first pairing result according to the first feature matching correlation degree; performs a correlation analysis between the format conversion information and the mask value feature in the second coverage area to obtain a second feature matching correlation degree, and calculates the second pairing result according to the second feature matching correlation degree;Wherein, after obtaining the second feature matching correlation, the method further includes: performing correlation analysis between the format conversion information and the mask value feature in the intersection area to obtain a third feature matching correlation, and calculating the second pairing result according to the second feature matching correlation and the third feature matching correlation; and determining a first weight and a second weight; comparing the difference between the second feature matching correlation and the third feature matching correlation, and adjusting the first weight and / or the second weight according to the difference; assigning the adjusted first weight to the second feature matching correlation and / or the second weight to the third feature matching correlation. ;

[0063] The third processing unit 650 is used to determine whether the preliminary screening modal data is used as fused modal data based on the first pairing result and the second pairing result, and to perform association annotation processing on the fused modal data and the inspection image, wherein the association annotation processing includes annotating an associated area, annotating an associated object, and annotating an associated form; in some embodiments, the third processing unit 650 is also used to determine the location information of the associated area between the inspection image and the fused modal data; based on the location information of the associated area, respectively lock the associated objects of the inspection image and the fused modal data; based on the associated object position information, determine the type of the associated form based on the associated objects of the inspection image and the fused modal data; and calculate the similarity between the inspection image and any associated object of the fused modal data, merge the similarities between the inspection image and all associated objects of the fused modal data to obtain a total similarity, perform interval comparison based on the total similarity, and determine the type of the associated form; and trace the second pairing result of the fused modal data, calculate the second pairing result to determine the association adjustment parameter, assign the association adjustment parameter to the total similarity to obtain an adjusted similarity, and re-determine the type of the association form based on the adjusted similarity.

[0064] In the above embodiments, the more specific working process of each functional unit can refer to the corresponding content disclosed in the aforementioned method embodiment. In addition, each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk (sol-identified state disk, SSD)), etc.

[0065] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0066] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0067] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0068] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for image quality classification based on a multimodal large model, characterized in that: The steps include: Identify the target attributes of the inspection image, perform preliminary analysis on the inspection image based on the target attributes, and determine a modal data matching strategy for the inspection image; Based on the modal data matching strategy, a related modal data set is selected from the inspection database, and the related modal data set is classified into types to obtain a plurality of related modal data; Performing a matching judgment between each of the associated modal data and the inspection image to obtain a matching degree of the associated modal data, and using the associated modal data with a matching degree higher than a preset matching value as the primary screening modal data, wherein the matching judgment includes inspection time judgment, inspection object judgment and inspection location judgment; Delineating a recognizable range and an unrecognizable range of the inspection image, extracting valid information according to the recognizable range, extracting fuzzy information according to the unrecognizable range, and pairing each of the primary screening modal data with the valid information and the fuzzy information, respectively, to obtain a first pairing result and a second pairing result; Based on the first pairing result and the second pairing result, it is determined whether the initial screening modal data is used as fused modal data, and the fused modal data is associated with the inspection image and annotated, wherein the associated annotation processing includes annotating associated areas, annotating associated objects, and annotating associated forms.

2. The image quality classification processing method based on multimodal large model according to claim 1 is characterized in that: Defining the recognizable range and the unrecognizable range of the inspection image, extracting valid information according to the recognizable range, and extracting fuzzy information according to the unrecognizable range includes the following steps: The training model is used to analyze and judge the pixel quality of the inspection image, and the normal pixel coordinates and the abnormal pixel coordinates are identified; the recognizable range is constructed using all the normal pixel coordinates, and the unrecognizable range is constructed using all the abnormal pixel coordinates; the pixels within the recognizable range are preliminarily identified and the covered target content is obtained, and the valid information is obtained based on the covered target content; the pixels within the unrecognizable range are preliminarily identified to obtain the abnormal type content, and the fuzzy information is obtained based on the abnormal type content.

3. The image quality classification processing method based on multimodal large model according to claim 2 is characterized in that: The steps of constructing the recognizable range by using the coordinates of all the normal pixel points and constructing the unrecognizable range by using the coordinates of all the abnormal pixel points include: Assigning morphological mask values ​​to each normal pixel coordinate and each abnormal pixel coordinate of the inspection image; constructing a first positive mask value distribution map and a first reverse mask value distribution map according to the morphological mask values ​​of all normal pixel coordinates, and constructing a second positive mask value distribution map and a second reverse mask value distribution map according to the morphological mask values ​​of all abnormal pixel coordinates; The recognizable range is determined using the first positive mask value distribution map and the second inverse mask value distribution map, and the unrecognizable range is determined using the first inverse mask value distribution map and the second positive mask value distribution map.

4. The image quality classification processing method based on multimodal large model according to claim 3 is characterized in that: The preliminary identification of the pixels within the identifiable range and obtaining the covered target content comprises the following steps: determining a first coverage area of ​​the identifiable range, performing feature extraction on a mask value distribution map of the first coverage area, detecting and identifying the extracted features, and obtaining the covered target content; The step of performing preliminary identification on the pixels within the unrecognizable range to obtain abnormal type content comprises the following steps: determining a second coverage area of ​​the unrecognizable range and an intersection area where the unrecognizable range overlaps with the recognizable range; Fuzzy quantification is performed on the mask value distribution within the second coverage area, and cluster analysis is performed on all quantified fuzziness information to generate a fuzziness visualization distribution map; the fuzziness visualization distribution map is preliminarily screened for abnormal types to obtain a preliminarily screened abnormality result; A first coverage area associated with the intersection area is determined, and the abnormal initial screening result is rescreened according to the associated first coverage area to obtain the abnormal type content.

5. The image quality classification processing method based on multimodal large model according to claim 4 is characterized in that: The step of pairing each of the primary screening modal data with the valid information and the fuzzy information to obtain a first pairing result and a second pairing result comprises the following steps: Acquire format conversion information of the primary screening modal data; perform correlation analysis on the format conversion information and the mask value feature in the first coverage area to obtain a first feature matching correlation degree, and calculate the first pairing result according to the first feature matching correlation degree; perform correlation analysis on the format conversion information and the mask value feature in the second coverage area to obtain a second feature matching correlation degree, and calculate the second pairing result according to the second feature matching correlation degree; Among them, after obtaining the second feature matching correlation degree, it also includes: performing correlation analysis on the format conversion information and the mask value feature in the intersection area to obtain a third feature matching correlation degree, and calculating the second pairing result according to the second feature matching correlation degree and the third feature matching correlation degree.

6. The image quality classification processing method based on multimodal large model according to claim 5 is characterized in that: The step of calculating the second pairing result based on the second feature matching correlation degree and the third feature matching correlation degree includes the following steps: determining a first weight and a second weight; comparing the difference between the second feature matching correlation degree and the third feature matching correlation degree, and adjusting the first weight and / or the second weight based on the difference; assigning the adjusted first weight to the second feature matching correlation degree and / or assigning the second weight to the third feature matching correlation degree.

7. The image quality classification processing method based on multimodal large model according to claim 1 is characterized in that: The process of associating and annotating the fused modality data with the inspection image comprises the following steps: Determine the associated area position information of the inspection image and the fused modal data; based on the associated area position information, respectively lock the associated objects of the inspection image and the fused modal data; based on the associated objects of the inspection image and the fused modal data, determine the type of the associated form; wherein the type of the associated form includes one of full association, strong association and weak association.

8. The image quality classification processing method based on multimodal large model according to claim 7 is characterized in that: The method of determining the type of association form based on the associated objects of the inspection image and the fused modal data includes the following steps: calculating the similarity between the inspection image and any associated object of the fused modal data, merging the similarities between the inspection image and all associated objects of the fused modal data to obtain a total similarity, performing interval comparison based on the total similarity, and determining the type of association form.

9. The image quality classification processing method based on multimodal large model according to claim 8 is characterized in that: The method also includes the step of adjusting the type of the association form: tracing back the second pairing result of the fused modal data, calculating the second pairing result to determine an association adjustment parameter, assigning the association adjustment parameter to the total similarity to obtain an adjusted similarity, and redetermining the type of the association form based on the adjusted similarity.

10. An image quality classification processing system based on a multimodal large model, characterized in that: include: A first recognition unit, which is used to recognize the target attribute of the inspection image, perform a preliminary analysis on the inspection image based on the target attribute, and determine a modal data matching strategy for the inspection image; A second identification unit, which is used to screen a related modal data set from the inspection database based on the modal data matching strategy, classify the related modal data set into types, and obtain a plurality of related modal data; A first processing unit, which is used to perform matching judgment between each type of the associated modal data and the inspection image, obtain the matching degree of the associated modal data, and use the associated modal data with a matching degree higher than a preset matching value as the primary screening modal data, wherein the matching judgment includes inspection time judgment, inspection object judgment and inspection location judgment; a second processing unit, which is used to define a recognizable range and an unrecognizable range of the inspection image, extract valid information according to the recognizable range, extract fuzzy information according to the unrecognizable range, and pair each of the primary screening modal data with the valid information and the fuzzy information, respectively, to obtain a first pairing result and a second pairing result; A third processing unit is used to determine whether the initial screening modal data is used as fused modal data based on the first pairing result and the second pairing result, and to associate and annotate the fused modal data with the inspection image, wherein the association annotation processing includes annotating associated areas, annotating associated objects, and annotating associated forms.

Images