Image defect analysis method, device, equipment, storage medium and program product

Abstract

The application provides a defect analysis method, device and equipment of image, storage medium and program product. It relates to the field of image processing. The method comprises: constructing an analysis strategy for defect analysis, the analysis strategy comprising at least one of a rating dimension, a defect level, a description word corresponding to the defect level, and a preset condition; encoding the analysis strategy to obtain a structured prompt word template; in response to receiving image data to be analyzed sent by an imaging device, inputting the image data and the prompt word template into a preset large model to analyze and process the image data and obtain a pre-analysis result of the image data; and performing confidence evaluation on the pre-analysis result, and determining an analysis result of the image data according to the evaluation result of the confidence evaluation and the pre-analysis result. The defect analysis effect of the image is improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a method, apparatus, device, storage medium, and program product for image defect analysis. Background Technology

[0002] In the context of the rapid iteration of smart terminal devices, the evaluation of the shooting quality of imaging equipment has become a core part of product development, testing and quality control.

[0003] In related technologies, traditional image quality defect analysis relies on manual evaluation. Test engineers need to visually inspect images and comprehensively judge defects from multiple dimensions such as exposure (AE), white balance (AWB), image signal processing (ISP), and focus (AF), and classify the severity of defects based on subjective experience.

[0004] However, in the above methods, the manual evaluation method relies heavily on expert experience, which can lead to poor results in defect analysis. Summary of the Invention

[0005] This application provides a method, apparatus, device, storage medium, and program product for image defect analysis, in order to solve the problem of poor defect analysis results.

[0006] In a first aspect, this application provides a method for defect analysis of images, comprising:

[0007] The analysis strategy for defect analysis is constructed, which includes at least one of the following: rating dimensions, defect level, descriptive terms corresponding to the defect level, and preset conditions.

[0008] The analysis strategy is encoded to obtain a structured prompt word template;

[0009] In response to receiving image data to be analyzed from the imaging device, the image data and prompt word template are input into a preset large model to analyze and process the image data and obtain the pre-analysis results of the image data;

[0010] The confidence level of the pre-analysis results is assessed, and the analysis results of the image data are determined based on the confidence level assessment results and the pre-analysis results.

[0011] In one possible design, a defect analysis strategy is constructed, which includes at least one of the following: rating dimensions, defect level, descriptive terms corresponding to the defect level, and preset conditions.

[0012] Multiple rating dimensions are obtained, including at least one of the following: exposure dimension, white balance dimension, image processing dimension, and focus dimension;

[0013] Based on multiple rating dimensions, determine at least one sub-dimension corresponding to each rating dimension;

[0014] Obtain preset conditions, defect levels, and defect judgment conditions corresponding to the defect levels;

[0015] An analysis strategy for defect analysis is constructed based on multiple rating dimensions, at least one sub-dimensional corresponding to the multiple rating dimensions, preset conditions, defect levels, and defect judgment conditions corresponding to the defect levels.

[0016] In one possible design, a defect analysis strategy is constructed based on multiple rating dimensions, at least one sub-dimensional corresponding to each rating dimension, preset conditions, defect levels, and defect judgment conditions corresponding to each defect level. This strategy includes:

[0017] Determine the descriptive terms corresponding to the defect level based on the defect level;

[0018] Generate a preset defect description based on the descriptive terms corresponding to the defect level and the defect judgment conditions;

[0019] Based on the preset defect description, multiple rating dimensions, at least one sub-dimension corresponding to the multiple rating dimensions, preset conditions, defect level, and defect judgment conditions corresponding to the defect level, a defect analysis strategy is constructed.

[0020] In one possible design, the image data is analyzed and processed based on the prompt word template to obtain pre-analysis results of the image data, including:

[0021] Based on the image data, determine at least one feature label of the image data;

[0022] Determine the scene type of the image data based on at least one feature label;

[0023] Based on the scene type, determine the scene strategy corresponding to the scene type. The scene strategy is used to adapt the image data to the actual scene during analysis and processing.

[0024] The scene strategy, image data, and prompt word template are input into a preset large model to obtain the pre-analysis results of the image data.

[0025] In one possible design, the confidence level of the pre-analysis results is assessed, including:

[0026] Based on the pre-analysis results, visual consistency is judged on the image data to obtain a first value, which is used to indicate the accuracy of the image defects identified by the pre-analysis results.

[0027] Based on the pre-analysis results, the image data is judged for scene consistency to obtain a second value, which is used to indicate the degree of logical consistency of the scene determined by the pre-analysis results.

[0028] Based on the pre-analysis results, the image data is output to determine consistency, and a third value is obtained. The third value is used to indicate the degree of consistency of the image data in multiple pre-analysis results.

[0029] The confidence level assessment result is determined based on the first, second, and third values.

[0030] In one possible design, based on the confidence level assessment results and the pre-analysis results, the analysis results of the image data are determined, including:

[0031] When the evaluation result is greater than or equal to the first threshold, the pre-analysis result is determined as the analysis result of the image data;

[0032] When the evaluation result is less than the first threshold and greater than or equal to the second threshold, the analysis result of the image data is determined based on the image data, the prompt word template, the first value, the second value, and the third value.

[0033] If the evaluation result is less than the second threshold, the image data and pre-analysis results are sent to the reviewer's user device so that the reviewer can perform a manual review. If the first threshold is greater than the second threshold, the image data and pre-analysis results are sent to the reviewer's user device.

[0034] Secondly, this application provides an image defect analysis apparatus, comprising: a construction module, an encoding processing module, an analysis processing module, and a determination module, wherein,

[0035] The building module is used to construct the analysis strategy for defect analysis. The analysis strategy includes at least one of the following: rating dimensions, defect level, descriptive terms corresponding to the defect level, and preset conditions.

[0036] The encoding processing module is used to encode the analysis strategy to obtain a structured prompt word template;

[0037] The analysis and processing module is used to, in response to receiving image data to be analyzed sent by the imaging device, input the image data and prompt word template into a preset large model to analyze and process the image data and obtain the pre-analysis results of the image data;

[0038] The determination module is used to assess the confidence level of the pre-analysis results and, based on the assessment results of the confidence level and the pre-analysis results, determine the analysis results of the image data.

[0039] In one possible design, building blocks are specifically used for,

[0040] Multiple rating dimensions are obtained, including at least one of the following: exposure dimension, white balance dimension, image processing dimension, and focus dimension;

[0041] Based on multiple rating dimensions, determine at least one sub-dimension corresponding to each rating dimension;

[0042] Obtain preset conditions, defect levels, and defect judgment conditions corresponding to the defect levels;

[0043] An analysis strategy for defect analysis is constructed based on multiple rating dimensions, at least one sub-dimensional corresponding to the multiple rating dimensions, preset conditions, defect levels, and defect judgment conditions corresponding to the defect levels.

[0044] In one possible design, building blocks are specifically used for,

[0045] Determine the descriptive terms corresponding to the defect level based on the defect level;

[0046] Generate a preset defect description based on the descriptive terms corresponding to the defect level and the defect judgment conditions;

[0047] Based on the preset defect description, multiple rating dimensions, at least one sub-dimension corresponding to the multiple rating dimensions, preset conditions, defect level, and defect judgment conditions corresponding to the defect level, a defect analysis strategy is constructed.

[0048] In one possible design, the analysis and processing module is specifically used for,

[0049] Based on the image data, determine at least one feature label of the image data;

[0050] Determine the scene type of the image data based on at least one feature label;

[0051] Based on the scene type, determine the scene strategy corresponding to the scene type. The scene strategy is used to adapt the image data to the actual scene during analysis and processing.

[0052] The scene strategy, image data, and prompt word template are input into a preset large model to obtain the pre-analysis results of the image data.

[0053] In one possible design, the module is specifically used for,

[0054] Based on the pre-analysis results, visual consistency is judged on the image data to obtain a first value, which is used to indicate the accuracy of the image defects identified by the pre-analysis results.

[0055] Based on the pre-analysis results, the image data is judged for scene consistency to obtain a second value, which is used to indicate the degree of logical consistency of the scene determined by the pre-analysis results.

[0056] Based on the pre-analysis results, the image data is output to determine consistency, and a third value is obtained. The third value is used to indicate the degree of consistency of the image data in multiple pre-analysis results.

[0057] The confidence level assessment result is determined based on the first, second, and third values.

[0058] In one possible design, the module is specifically used for,

[0059] When the evaluation result is greater than or equal to the first threshold, the pre-analysis result is determined as the analysis result of the image data;

[0060] When the evaluation result is less than the first threshold and greater than or equal to the second threshold, the analysis result of the image data is determined based on the image data, the prompt word template, the first value, the second value, and the third value.

[0061] If the evaluation result is less than the second threshold, the image data and pre-analysis results are sent to the reviewer's user device so that the reviewer can perform a manual review. If the first threshold is greater than the second threshold, the image data and pre-analysis results are sent to the reviewer's user device.

[0062] Thirdly, this application provides an electronic device, comprising: at least one processor and a memory; the memory storing computer-executable instructions; the at least one processor executing the computer-executable instructions stored in the memory, causing the at least one processor to perform the defect analysis method of the image as described in the first aspect above and various possible designs of the first aspect.

[0063] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the image defect analysis method of the first aspect and various possible designs of the first aspect.

[0064] Fifthly, this application provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned first aspect and a method for analyzing defects in images of various possible designs of the first aspect.

[0065] The image defect analysis method, apparatus, device, storage medium, and program product provided in this application can construct an analysis strategy for defect analysis when it is necessary to analyze image defects. The analysis strategy includes at least one of the following: rating dimensions, defect level, descriptive words corresponding to the defect level, and preset conditions. The analysis strategy is encoded to obtain a structured prompt word template. In response to receiving image data to be analyzed sent by the imaging device, the image data and the prompt word template are input into a preset large model to analyze and process the image data to obtain a pre-analysis result of the image data. The confidence level of the pre-analysis result is evaluated, and the analysis result of the image data is determined based on the confidence level evaluation result and the pre-analysis result. In this way, by constructing a multi-dimensional analysis strategy and transforming it into executable logical rules, the subjective judgment differences of different evaluators on the same defect can be avoided, thereby improving the effectiveness of the analysis. At the same time, by combining the semantic understanding capabilities of the prompt word template and the preset large model, the preset large model can perform comprehensive reasoning based on image content and rule logic, and complete the defect classification without human intervention. Furthermore, since the preset large model may exhibit hallucination phenomena when analyzing image defects, such as fabricating non-existent defects or exaggerating the severity of defects, the confidence level assessment is used to unify the image defect judgment criteria, effectively ensuring the validity of the image defect analysis results. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0067] Figure 1 This is a schematic diagram of the system architecture provided for an embodiment of this application;

[0068] Figure 2 A flowchart illustrating an image defect analysis method provided in an embodiment of this application;

[0069] Figure 3 A schematic diagram illustrating the process of constructing a defect analysis strategy provided in this application embodiment;

[0070] Figure 4 This is a schematic diagram illustrating the process of determining the analysis results of image data provided in an embodiment of this application;

[0071] Figure 5 A schematic diagram of the structure of an image defect analysis device provided in an embodiment of this application;

[0072] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0073] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0074] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0075] The collection, storage, use, processing, transmission, provision, and disclosure of user data and other information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0076] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0077] To facilitate understanding, the following will be combined with... Figure 1 The system architecture applicable to the embodiments of this application will be described.

[0078] Figure 1 This is a schematic diagram of the system architecture provided for an embodiment of this application. Please refer to [link / reference]. Figure 1 This includes imaging equipment and electronic equipment. Imaging equipment refers to devices capable of acquiring or generating images, and can be used to output image materials for defect analysis. Examples of imaging equipment include cameras, industrial cameras, digital cameras, scanners, and X-ray machines. Electronic equipment can be devices with on-device computing capabilities, such as terminal devices and servers. Imaging equipment can send acquired images to electronic equipment, which can then convert the images into image processing data and perform defect analysis based on defect analysis strategies.

[0079] Image quality assessment using existing imaging equipment mainly includes manual visual inspection, objective detection based on traditional image processing, and deep learning classification models trained for single-class defects. Manual methods typically involve testers making subjective judgments about image defects based on experience; traditional algorithms often identify overexposure, underexposure, blurring, and other problems through brightness distribution, edge information, or frequency domain features; classification models output labels or probability results for specific defects.

[0080] The aforementioned solutions are insufficient to meet the unified analysis requirements across multiple dimensions and scenarios in practical applications. While manual evaluation offers some flexibility, it relies on human experience. When faced with defects involving subjective semantics, such as blurry images or unnatural colors, inconsistencies in ratings can easily arise among different personnel, resulting in low evaluation efficiency. Traditional algorithms, while capable of handling some quantifiable problems, have limited ability to identify complex defects requiring contextual understanding. Classification models typically rely on predefined defect categories, lack generalization ability, and their outputs are often simple labels, making it difficult to directly form standardized and unified analytical conclusions.

[0081] Due to the lack of unified rating dimensions, defect levels, and description standards, existing output results often suffer from inconsistent descriptions and insufficient reproducibility, posing difficulties for subsequent defect archiving, statistical analysis, and quality tracking. Especially when R&D testing is carried out in batches, if analytical efficiency, consistency of conclusions, and standardization of results cannot be balanced, it will be difficult to support the automated implementation of the imaging equipment quality assessment process.

[0082] To address the aforementioned technical problems, in this embodiment, when it is necessary to analyze image defects, an analysis strategy for defect analysis can be constructed. This strategy includes at least one of the following: rating dimensions, defect level, descriptive words corresponding to the defect level, and preset conditions. The analysis strategy is encoded to obtain a structured prompt word template. In response to receiving image data to be analyzed from the imaging device, the image data is analyzed according to the prompt word template to obtain a pre-analysis result. The pre-analysis result is then evaluated for confidence. Based on the confidence evaluation result and the pre-analysis result, the analysis result of the image data is determined. Thus, by constructing a multi-dimensional analysis strategy and transforming it into executable logical rules, the subjective judgment differences of different evaluators regarding the same defect are avoided, improving the analysis effect. Furthermore, by combining the prompt word template and the semantic understanding capabilities of the preset large model, the preset large model can perform comprehensive reasoning based on image content and rule logic, completing defect rating without manual intervention. Moreover, by unifying the image defect judgment standard through confidence evaluation, the validity of the image defect analysis results is effectively guaranteed.

[0083] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0084] Figure 2 This is a flowchart illustrating an image defect analysis method provided in an embodiment of this application. Please refer to... Figure 2 As shown, the method may include the following steps:

[0085] S201. Analytical strategies for constructing defect analysis.

[0086] The execution subject of this application embodiment can be an electronic device or an image defect analysis device installed in an electronic device. The image defect analysis device can be implemented by software or by a combination of software and hardware. The electronic device can be a device with on-device computing capabilities, such as a server or terminal device.

[0087] The analysis strategy for defect analysis can refer to the set of rules used to carry out image defect analysis. Its function is to organize the rating dimensions, defect levels, descriptive words and preset conditions required for subsequent analysis into a unified rule structure for subsequent encoding processing and image analysis to call.

[0088] Defect analysis strategies can be used to perform defect analysis on images sent by imaging equipment.

[0089] In one possible implementation, the analysis strategy may include at least one of the following: rating dimensions, defect levels, descriptive terms corresponding to the defect levels, and preset conditions.

[0090] The rating dimensions can be used to measure image defects from different perspectives. Specifically, as analytical dimensions for defect analysis, they break down image quality issues into multiple independently assessable aspects. For example, rating dimensions may include at least one of the following: exposure dimension, white balance dimension, image processing dimension, and focus dimension.

[0091] Defect levels can be used to classify the severity of defects, so that different defect behaviors can fall into a unified level system.

[0092] For example, defect levels can include fatal defects, major defects, obvious defects, and minor flaws. Specifically, when the defect level is fatal, the current function can be considered completely ineffective and has no value for normal shooting use; when the defect level is major, the core function can be considered severely abnormal, with key subject information lost or severely distorted; when the defect level is obvious, the defect can be considered clearly perceptible, but the core function is still usable; when the defect level is minor flaw, only minor flaws exist, and it can be used normally without being easily noticed.

[0093] In one possible implementation, when the defect level is critical, the current defect can be considered to need to be fixed immediately; when the defect level is major, the current defect can be considered to need to be prioritized; when the defect level is obvious, the current defect can be considered to potentially affect the user experience of the image and needs to be rectified; and when the defect level is minor flaw, the current defect can be considered to be able to be optimized later.

[0094] Descriptive terms can refer to standardized text used to express the corresponding defect levels, so that the final output uses a consistent language template.

[0095] For example, when the defect level is fatal, the corresponding descriptive terms can be "complete" or "extreme," such as "completely overexposed" or "extremely blurry." When the defect level is major, the corresponding descriptive terms can be "severe" or "abnormal," such as "severe color cast" or "severe noise." When the defect level is obvious, the corresponding descriptive terms can be "obvious" or "manifest," such as "obviously yellowish" or "manifestly hazy." When the defect level is minor, the corresponding descriptive terms can be "slight" or "minor," such as "slight color cast" or "minor exposure."

[0096] Preset conditions can be used to define red line conditions and domain-specific conditions.

[0097] The red line condition refers to a special judgment rule that automatically raises the defect level to no less than the preset minimum level when the condition is met. Its function is to set a judgment bottom line that cannot be downgraded for critical defects.

[0098] For example, red line conditions may include overexposure red lines, blurry red lines, automatic exposure flashing red lines, quantity upgrade rules, etc.

[0099] Specifically, the overexposure red line refers to an image being overexposed to a red line. Regardless of other dimensions, the exposure defect level is at least a major defect. In one exemplary implementation, this condition can be determined by: the proportion of overexposed area in the main image region exceeding a preset threshold (e.g., exceeding 50%), and the complete loss of key textures. Specifically, this can be quantified by detecting the percentage of pixels with brightness values ​​greater than or equal to a preset threshold (e.g., brightness value ≥ 240) using a histogram analysis algorithm.

[0100] The "blurred red line" refers to a problem where the image appears blurry, and regardless of the performance in other dimensions, the defect level in the exposure dimension is at least a major defect. In one exemplary implementation, the criteria for determining this condition can be: the global contrast of the image is lower than a preset contrast threshold, and the high-pass component energy of the image is lower than a preset threshold, indicating that the image detail and clarity are severely insufficient.

[0101] The automatic exposure flicker red line can refer to an automatic exposure flickering problem that is red line-level. Regardless of the performance of other dimensions, the defect level of the exposure dimension is at least a major defect. In one exemplary implementation, the criterion for determining this condition can be: the average brightness change between consecutive frames exceeds a preset flicker threshold (e.g., the brightness difference between adjacent frames > 15%).

[0102] The quantity upgrade rule can refer to the fact that when any one defect level exceeds the first numerical number of sub-dimensions, the overall defect level of the image needs to be upgraded by one level.

[0103] Domain-specific conditions refer to differentiated judgment rules formulated based on specific application areas (such as portrait photography, specific market regions, etc.) or specific scenario types. Their function is to adapt to specific scenarios based on general classification rules.

[0104] For example, domain-specific conditions may include skin color rules, scene priority rules, and project priority rules.

[0105] Specifically, the skin tone rule can refer to the white balance dimension's defect level being at least a significant defect when the skin tone of a portrait shows a color cast.

[0106] Scene priority rule can refer to the fact that in portrait scenes, when there is a problem of the brightness of the face being too dark or overexposed, the defect level is higher than the defect level of other scenes. In one exemplary implementation, the average brightness of the face region can be calculated after the face detection algorithm locates the face region, and then compared with the preset reasonable range of face brightness to determine the defect level.

[0107] Project prioritization rules refer to determining the defect level according to a preset priority order when there are conflicts between different rating dimensions based on different project needs. For example, in low-to-mid-range equipment projects, when there is a conflict between color noise control and white balance saturation, the defect level is determined first based on the performance of white balance saturation.

[0108] In one possible implementation, the defect analysis strategy can be constructed as follows: obtain multiple rating dimensions; determine at least one sub-dimension corresponding to the multiple rating dimensions; obtain preset conditions, defect levels, and defect judgment conditions corresponding to the defect levels; and construct the defect analysis strategy based on the multiple rating dimensions, at least one sub-dimension corresponding to the multiple rating dimensions, preset conditions, defect levels, and defect judgment conditions corresponding to the defect levels.

[0109] The rating dimensions include exposure, white balance, image processing, and focus. The exposure dimension can include sub-dimensions such as evaluating exposure accuracy, dynamic range, contrast and clarity, and performance in special scenes. The white balance dimension can include sub-dimensions such as evaluating white balance accuracy, color reproduction, and skin tone representation. The image processing dimension can include sub-dimensions such as evaluating noise control, texture and detail, sharpening effect, and artifact suppression. The focus dimension can include sub-dimensions such as evaluating focus accuracy.

[0110] The defect judgment criteria corresponding to the defect level can be quantified into the judgment criteria for the defect level. For example, the judgment criteria for major defects in the exposure dimension can include an average screen brightness of less than 30.

[0111] Understandably, multiple rating dimensions, at least one sub-dimension corresponding to multiple rating dimensions, preset conditions, defect levels, and defect judgment conditions corresponding to defect levels can be preset values.

[0112] In this way, vague human judgment experience can be transformed into standardized and quantifiable rules, avoiding discrepancies in analysis. At the same time, the four-layer structure of rating dimensions, sub-dimensions, defect levels, and judgment conditions makes the analysis strategy more detailed.

[0113] S202. Encode the analysis strategy to obtain a structured prompt word template.

[0114] Encoding processing can refer to transforming the constructed text and numerical analysis strategies into a fixed, machine-recognizable, and structured standardized text format, removing colloquial content, and unifying word order, parameters, and decision logic.

[0115] The structured prompt word template refers to a standardized text template with fixed sentence structure and fixed parameter placeholders. It has a built-in framework of all defect judgment rules and reserves space for image features, scenes and equipment parameters. When inputting images later, you can directly fill in the corresponding information to complete the automatic analysis without messy free text.

[0116] For example, assuming the analysis strategy is "a blur radius greater than 5 for a focus defect is a serious defect", the prompt word template after encoding processing would be "Dimension: Focus Dimension, Sub-dimension: Focus Accuracy, Level: Major Defect, Defect Judgment Condition: Blur Radius > 5".

[0117] In one possible implementation, the electronic device reads relevant strategy content from the analysis strategy and encodes it according to a preset template to form a structured prompt template that can be used by the model. This prompt template can be used to indicate the analysis task, analysis basis, and output requirements. During the encoding process, different contents within the analysis strategy can be organized according to a unified structure, enabling the model to infer from image data based on the analysis strategy during analysis. For preset conditions, corresponding applicable prerequisites or limitations can be written into the prompt template.

[0118] In this way, the structured prompt word template formed after encoding retains the logical relationship in the analysis strategy and provides unified constraints on the output of subsequent processing. It can reason along the same analysis framework when facing different images, thereby improving the consistency and reproducibility of subsequent pre-analysis results.

[0119] S203. In response to receiving the image data to be analyzed sent by the imaging device, input the image data and prompt word template into a preset large model to obtain the pre-analysis results of the image data.

[0120] The preset large model can be a pre-trained multimodal large model, that is, a large neural network model with image and text multimodal understanding capabilities based on the self-attention transformer architecture.

[0121] The pre-defined large model may include a visual encoding layer, a text encoding layer, a multimodal fusion layer, and an output generation layer.

[0122] The visual encoding layer transforms input image data into visual feature representations. It segments the input image into fixed-size patches, each patch is mapped to a fixed-dimensional vector representation via linear projection, and positional encoding is added before feeding it into a multi-layer Transformer encoder. The final output is a sequence of visual features of the image. In one possible implementation, the visual encoding layer comprises 12 to 24 Transformer encoder layers, each containing a multi-head self-attention mechanism and a feedforward neural network. The hidden layer dimension is 768 to 1024, and the number of attention heads is 12 to 16. The image input resolution supports various specifications from 224×224 to 1500×1500 pixels, maintaining the aspect ratio through adaptive scaling and padding techniques.

[0123] The text encoding layer is used to convert input text prompts into text feature representations, and it consists of multiple stacked self-attention layers and feedforward neural network layers. In one possible implementation, the text encoding layer contains 24 to 96 Transformer layers, with hidden layer dimensions ranging from 2048 to 8192. The text input is segmented into words using byte-by-byte encoding, with a vocabulary size ranging from 100,000 to 320,000.

[0124] Multimodal fusion layers can be used to fuse visual features output from visual encoding layers with text features output from text encoding layers across modalities, enabling joint understanding of image content and text instructions. The multimodal fusion layer employs a cross-attention mechanism, using text features as queries and visual features as keys and values, allowing the model to focus on visual regions in the image that are semantically relevant to the current text when generating text output. In one possible implementation, the multimodal fusion layer comprises 4 to 8 cross-attention layers, alternating with the self-attention layers of the text encoding layer.

[0125] The output generation layer can be used to autoregressively generate text output based on the feature representation after multimodal fusion. The output generation layer uses a linear classification head to map the hidden layer representation to the vocabulary space, calculates the generation probability of each word through a function, and determines the final text output sequence through beam search or sampling strategies.

[0126] The training process of a pre-defined large-scale model can include a pre-training phase, an instruction fine-tuning phase, and a human feedback reinforcement learning phase. The pre-training phase involves pre-training on a large-scale image-text pairing dataset, enabling the model to learn the semantic alignment relationships between images and text. The instruction fine-tuning phase involves fine-tuning on a dataset containing diverse task instructions and corresponding outputs, allowing the model to complete specific tasks following natural language instructions. The human feedback reinforcement learning phase involves training a reward model by ranking the model's output preferences of human annotators, and then using a proximal policy optimization algorithm to perform reinforcement learning training on the model, making the analysis results generated by the model more consistent with the preferences and standards of human image evaluation experts.

[0127] The training process of a pre-defined large model can be categorized using loss function convergence, downstream task performance saturation, and a maximum training epoch limit. Loss function convergence refers to the model's loss being considered convergent when the change in the loss function value on the validation set is less than a preset threshold (e.g., relative change < 1%) over N consecutive training epochs (e.g., N=3). Downstream task performance saturation refers to evaluating the model's performance metrics on a specified downstream task (e.g., image description quality, defect classification accuracy) on the validation set. When the improvement in these metrics is less than a preset threshold (e.g., improvement < 0.5%) over N consecutive training epochs, the model's performance is considered saturated. The maximum training epoch limit is set to prevent overfitting (e.g., no more than 3 epochs in the pre-training phase and no more than 5 epochs in the fine-tuning phase). Training is forcibly terminated when the maximum number of training epochs is reached.

[0128] After acquiring image data, the imaging equipment can send the image data to an electronic device so that the electronic device can perform defect analysis on the imaging equipment.

[0129] Image data can refer to a single image, a frame from a series of images, or a representative image selected in the testing process.

[0130] Input the prompt word template and image data into a pre-trained large model, and use the output of the pre-trained large model as the pre-analysis result of the image data.

[0131] The pre-analysis results may include a natural language analysis report containing defect level determinations for each dimension, specification descriptions, and optimization suggestions.

[0132] In one possible implementation, the pre-analysis results of the image data can also be obtained by: determining at least one feature label of the image data based on the image data; determining the scene type of the image data based on the at least one feature label; determining the scene strategy corresponding to the scene type based on the scene type; and inputting the scene strategy, image data, and prompt word template into a preset large model to obtain the pre-analysis results of the image data.

[0133] In one possible implementation, scene strategies, prompt word templates, and image data can be input into a pre-trained large model, and the output of the pre-trained large model can be used as the pre-analysis result of the image data.

[0134] Feature labels can refer to scene features within an image. For example, feature labels can include labels such as portrait, indoor, road, grassland, night scene, backlight, moving target, and static scene.

[0135] Scene types can include portrait, landscape, document, macro, sports, and low-light types, etc.

[0136] Based on the feature labels of image data, the corresponding scene type can be queried in a preset scene mapping table. For example, in the scene mapping table, the feature labels corresponding to the portrait type can include portrait, static scene, etc., and the feature labels corresponding to the landscape type can include trees, flowers, sky, etc.

[0137] Scene types can correspond to specific scene strategies, which can be used to adapt the image data to the actual scene during analysis and processing. For example, in portrait mode, the defect level corresponding to the white balance dimension needs to be increased by one level, the defect level corresponding to the focus dimension needs to be increased by one level, and the weight of skin tone needs to be increased, meaning that the requirements for skin tone in the image are higher; in landscape mode, the defect judgment condition for overexposure red line can be reduced, while the defect judgment condition for blurry red line can be increased, meaning that the requirements for image blurriness are stricter; in document mode, the requirement for sharpness needs to be increased, and the defect level corresponding to smearing needs to be increased by one level, meaning that the requirement for sharpness is higher; in low-light mode, the defect judgment condition for noise tolerance can be reduced, and the defect judgment condition for underexposure tolerance can be reduced, meaning that the tolerance for dark environments is relaxed.

[0138] In this way, the pre-analysis process of image data can dynamically adjust the focus of analysis according to scene differences, making the preliminary judgment of images in different scenes more targeted, thereby improving the accuracy and consistency of the pre-analysis results, and providing a more stable input basis for subsequent confidence assessment and the generation of final analysis results.

[0139] S204. Calculate the confidence level of the pre-analysis results and determine the analysis results of the image data based on the confidence level assessment results and the pre-analysis results.

[0140] Confidence assessment refers to the assessment of the reliability of pre-analysis results. Its function is to screen the credibility of the initial output of the model and prevent the direct adoption of unstable or inconsistent analysis conclusions.

[0141] The analysis results of image data can be the final analysis conclusions of image defects, which can be directly output, archived, or integrated with a defect management system.

[0142] Understandably, because the pre-defined large model (i.e., the multimodal large model) can exhibit an illusion when analyzing image defects, that is, it may fabricate non-existent defects or exaggerate the severity of defects, it is necessary to suppress the analysis errors caused by the illusion of the pre-defined large model through confidence assessment.

[0143] In one possible implementation, after obtaining the pre-analysis results, the electronic device can perform a confidence assessment on them. The confidence assessment can be judged by combining the image content, analysis strategy, and the degree of consistency between the pre-analysis results to determine the reliability of the pre-analysis results.

[0144] In one possible implementation, the electronic device can classify the pre-analysis results based on the confidence level assessment. When the confidence level is high, the pre-analysis results can be directly used as the analysis results; when the confidence level is in the middle range, a supplementary judgment process can be executed; when the confidence level is low, a manual review process can be triggered, and the pre-analysis results and image data can be sent to the review terminal, after which the final conclusion is written back.

[0145] In one possible implementation, when determining the analysis results, the electronic device can output a structured rating report. This report may include at least image identifiers, rating levels for each dimension, corresponding descriptive terms, a comprehensive conclusion, a confidence level, and a mark indicating whether it was manually reviewed. If a correction is made within the intermediate confidence interval, the report also retains the corrected rating and the source of the correction rule for future tracking.

[0146] In one possible implementation, the structured rating report can also be standardized and corrected. Specifically, a preset standard descriptor mapping table can be obtained, which can be used to indicate the mapping relationship between descriptors and standard descriptors. Non-standard descriptors in the structured rating report can be obtained, and according to the standard descriptor mapping table, the standard descriptors corresponding to the non-standard descriptors can be determined and replaced in the end of the structured rating report, thereby ensuring the standardization and consistency of the analysis results (i.e., the structured rating report) obtained from different image data.

[0147] In this way, by setting independent confidence assessment and hierarchical processing logic for the pre-analysis results, the model output undergoes consistency verification and condition constraints before entering the final use stage, thereby improving the effectiveness of the final analysis results.

[0148] In this embodiment, when it is necessary to analyze image defects, a defect analysis strategy can be constructed first. The defect analysis strategy can refer to a set of rules used to carry out image defect analysis. The analysis strategy can include at least one of rating dimensions, defect levels, descriptive words corresponding to the defect levels, and preset conditions. The analysis strategy is encoded to obtain a structured prompt word template. The encoding process can refer to converting the constructed text and number form analysis strategy into a fixed format, machine-recognizable, and structured standardized text format. The structured prompt word template can refer to a standardized text template with fixed sentence structure and fixed parameter placeholders. In response to receiving the image data to be analyzed sent by the imaging device, the image data and the prompt word template are input into a preset large model to analyze and process the image data to obtain the pre-analysis results of the image data. The pre-analysis results can include a natural language analysis report containing defect level judgments, standard descriptions, and optimization suggestions for each dimension. The confidence level of the pre-analysis results is evaluated. Based on the confidence level evaluation results and the pre-analysis results, the analysis results of the image data are determined. The analysis results can be the final analysis conclusion of the image defects, which can be directly output, archived, or interfaced with the defect management system. In this way, by organizing rating dimensions, defect levels, descriptive terms, and preset conditions into a unified analysis strategy and further encoding them into structured prompt word templates, and combining this with the image understanding capabilities of a multimodal large model and a post-confidence assessment mechanism, image defect analysis can be performed within a unified rule framework. This results in analysis results with clear dimensional divisions, a fixed rating system, and standardized description methods, improving the effectiveness of defect analysis. At the same time, the confidence assessment method suppresses the problem of analysis errors caused by the illusion of a preset large model, further improving the effectiveness of defect analysis. Furthermore, the standardized descriptive terms ensure the standardization and consistency of analysis results obtained from different image data.

[0149] Based on any of the above embodiments, the following, in conjunction with Figure 3 The process of constructing a defect analysis strategy is explained in detail.

[0150] Figure 3 This is a schematic diagram illustrating the process of constructing a defect analysis strategy provided in an embodiment of this application. Please refer to... Figure 3 The method may include:

[0151] S301, Obtain multiple rating dimensions.

[0152] The rating dimensions may include at least one of the following: exposure dimension, white balance dimension, image processing dimension, and focus dimension.

[0153] Exposure dimension can be used to judge abnormalities in the brightness and darkness of an image, such as overexposure and underexposure.

[0154] White balance can be used to assess color cast defects in an image, such as being too yellow, too blue, or too red.

[0155] Image processing dimensions can be used to evaluate image quality defects caused by algorithms such as noise reduction and sharpening, for example, excessive noise, over-sharpening jagged edges, etc.

[0156] Focus dimensions can be used to assess defects in image sharpness, such as overall defocus or local blur.

[0157] S302. Based on multiple rating dimensions, determine at least one sub-dimensional corresponding to each rating dimension.

[0158] Sub-dimensions can refer to the sub-items of inspection that are broken down under the rating dimension. By breaking down broad categories of defects into specific and quantifiable defect types, the judgment criteria can be made more refined.

[0159] For example, the exposure dimension may include sub-dimensions such as evaluating exposure accuracy, dynamic range, contrast and clarity, and performance in special scenes; the white balance dimension may include sub-dimensions such as evaluating white balance accuracy, color reproduction, and skin tone performance; the image processing dimension may include sub-dimensions such as evaluating noise control, texture and detail, sharpening effect, and artifact suppression; and the focus dimension may include sub-dimensions such as evaluating focus accuracy.

[0160] S303. Obtain preset conditions, defect level, and defect judgment conditions corresponding to the defect level.

[0161] Preset conditions can refer to pre-designed logical conditions, red-line conditions used for limitation, and domain-specific conditions.

[0162] Pre-designed logical conditions can refer to the relationship between logical or conditional relationships. For example, a higher defect level can automatically override a lower defect level, and a defect level can be automatically upgraded when a specific condition (such as a red line condition) is met. The red line condition is not affected by the downgrade of other conditions.

[0163] Defect levels can be used to classify the severity of defects, so that different defect behaviors can fall into a unified level system.

[0164] The defect judgment criteria corresponding to the defect level can be used to quantify the judgment criteria for the defect level.

[0165] S304. Determine the descriptive terms corresponding to the defect level based on the defect level.

[0166] Descriptive terms can refer to standardized text used to express the corresponding defect levels, so that the final output uses a consistent language template.

[0167] For example, when the defect level is fatal, the corresponding descriptive terms can be "complete" or "extreme," such as "completely overexposed" or "extremely blurry." When the defect level is major, the corresponding descriptive terms can be "severe" or "abnormal," such as "severe color cast" or "severe noise." When the defect level is obvious, the corresponding descriptive terms can be "obvious" or "manifest," such as "obviously yellowish" or "manifestly hazy." When the defect level is minor, the corresponding descriptive terms can be "slight" or "minor," such as "slight color cast" or "minor exposure."

[0168] S305. Generate a preset defect description based on the descriptive terms corresponding to the defect level and the defect judgment conditions.

[0169] Preset defect descriptions can refer to descriptive phrases generated by semantically combining descriptive words corresponding to defect levels and defect judgment conditions, which are used to constrain the description of the analysis results of subsequent output image data.

[0170] For example, assuming the defect judgment condition corresponding to the defect level is a resolution of less than 30, and the descriptive word corresponding to the defect level is "severe", then the generated preset defect description is "severely insufficient resolution".

[0171] In this way, pre-setting the standardization of defect descriptions and outputting standardized terms can avoid the confusion in defect classification caused by the mixing of terms, while directly adapting to the defect management system and improving the efficiency of statistical analysis.

[0172] S306. Based on the preset defect description, multiple rating dimensions, at least one sub-dimension corresponding to the multiple rating dimensions, preset conditions, defect level, and defect judgment conditions corresponding to the defect level, construct the defect analysis strategy.

[0173] Preset defect descriptions, rating dimensions, sub-dimensions, preset conditions, defect levels, and defect judgment conditions can be written into the analysis strategy template, so that subsequent analyses can directly read and output defect conclusions in a uniform format.

[0174] In this way, the constructed analysis strategy can bind defect levels, judgment conditions, and descriptive terms to form structured content that includes both judgment criteria and result descriptions, which can be used for subsequent image defect recognition. Since the same defect level corresponds to a fixed descriptive term, and the output semantics are constrained by preset defect descriptions, the final generated analysis strategy can maintain consistency in expression and give the image defect analysis results stronger standardization features.

[0175] exist Figure 3In the illustrated embodiment, when it is necessary to construct an analysis strategy for defect analysis, multiple rating dimensions can be obtained. These rating dimensions may include at least one of exposure, white balance, image processing, and focus dimensions. Based on these multiple rating dimensions, at least one sub-dimension corresponding to each rating dimension is determined. A sub-dimension can refer to a subdivided detection item under each rating dimension, breaking down broad categories of defects into specific, quantifiable defect types. Preset conditions, defect levels, and defect judgment conditions corresponding to each defect level are obtained. Based on the defect level, descriptive terms corresponding to each defect level are determined. These descriptive terms can refer to standardized text used to express the defect level, ensuring a consistent language template for the final output. Based on the descriptive terms corresponding to each defect level and the defect judgment conditions, a preset defect description is generated. This preset defect description can refer to a descriptive phrase generated by semantically combining the descriptive terms corresponding to each defect level and the defect judgment conditions. Based on the preset defect description, multiple rating dimensions, at least one sub-dimension corresponding to each rating dimension, preset conditions, defect levels, and defect judgment conditions corresponding to each defect level, an analysis strategy for defect analysis is constructed. In this way, the above methods transform vague manual quality inspection experience into a standardized rule system with clear hierarchy, numerical quantification, and unified description. By combining main dimensions with sub-dimensions, refined defect classification is achieved. Judgment thresholds and standard descriptions are integrated to generate unified defect descriptions, thereby avoiding subjective differences in manual judgment and discrepancies in the understanding of defect standards. At the same time, it provides a regular data source for the subsequent generation of structured prompt word templates, thus improving the effectiveness of defect analysis.

[0176] Based on any of the above embodiments, the following, in conjunction with Figure 4 The process of determining the analysis results of image data is explained in detail.

[0177] Figure 4 This is a schematic diagram illustrating the process of determining the analysis results of image data as provided in an embodiment of this application. Please refer to... Figure 4 The method may include:

[0178] S401. Conduct a confidence assessment on the pre-analysis results and determine the assessment result of the confidence assessment.

[0179] Confidence assessment can refer to scoring from three independent dimensions: physical evidence in the image, scene logic, and stability of repeated detection, and then merging them to calculate a comprehensive confidence score, which quantitatively judges whether the pre-analysis conclusion is objective and accurate, and whether there are any misjudgments.

[0180] The confidence assessment results can be used to quantify the reliability of the pre-analysis results.

[0181] In one possible implementation, the confidence assessment result can be determined as follows: visual consistency is judged on the image data based on the pre-analysis results to obtain a first value; scene consistency is judged on the image data based on the pre-analysis results to obtain a second value; output consistency is judged on the image data based on the pre-analysis results to obtain a third value; and the confidence assessment result is determined based on the first value, the second value, and the third value.

[0182] Among them, visual consistency judgment can refer to comparing the pixels and objective features of the original image data to determine whether the defects pointed out by the pre-analysis results actually exist in the image data. The first value can be used to indicate the accuracy of the image defects determined by the pre-analysis results. It can be understood that the higher the accuracy of the image defects, the higher the first value.

[0183] In one possible implementation, the first value can be determined as follows: extract each defect identified by the multimodal large model in the pre-analysis results, along with its corresponding rating dimension and defect region description; for each defect, call the computer vision layer algorithm of the corresponding rating dimension to perform objective quantitative analysis on the image data to obtain an objective quantitative index value; compare the objective quantitative index value with a preset defect judgment condition threshold to determine whether the objective index supports the judgment conclusion of the large model; divide the total number of defects identified by the multimodal large model by the number of defects whose objective quantitative index matches the judgment conclusion of the large model to obtain the first value.

[0184] For example, suppose the preset large model determines that the image has the following defects: overexposure in the exposure dimension, yellowish tint in the white balance dimension, and blurriness in the focus dimension, a total of three defects. For the overexposure defect in the exposure dimension, the histogram analysis algorithm calculates the percentage of overexposed pixels in the main area, and the result is 65%, exceeding the preset threshold for overexposure defects (e.g., overexposed area occupies more than 50% of the image). Therefore, the objective indicator of this defect is consistent with the model's judgment. For the yellowish tint defect in the white balance dimension, the color saturation and chromaticity distribution are calculated using color space analysis, and the result is that the amount of yellowish tint is lower than the preset threshold. Therefore, the objective indicator of this defect is inconsistent with the model's judgment. For the blurriness defect in the focus dimension, the sharpness score is calculated using the Laplacian variance algorithm, and the result is 25 (lower than the sharpness judgment threshold of 30). Therefore, the objective indicator of this defect is consistent with the model's judgment. The first value is 0.67.

[0185] Scene consistency judgment can refer to determining whether the defect level of the pre-analysis result conforms to the normal judgment logic of the scene by combining the scene to which the image belongs, the preset judgment rules of the scene, and industry prior knowledge. The second value can be used to indicate the degree of logical consistency of the scene determined by the pre-analysis result. It can be understood that the higher the degree of logical consistency of the scene, the higher the second value. Optionally, scene consistency judgment can use knowledge graph reasoning to determine whether the defect level conforms to the scene rules.

[0186] In one possible implementation, the second value can be determined as follows: The scene type of the image data (e.g., portrait scene, landscape scene, low-light scene, etc.) is determined; a scene prior rule set corresponding to that scene type is obtained; the scene prior rule set includes at least one of the following rules: a preset reasonable level range for common defect types in that scene (e.g., skin color cast should be at least a noticeable defect in a portrait scene); a reasonable probability distribution of specific defect levels in that scene (e.g., noise defects are relatively more likely to be noticeable or worse in a low-light scene, which is considered normal); and logical constraints between defect levels in that scene (e.g., in a backlit scene, the defect level in the exposure dimension should not be simultaneously a minor flaw with the defect level in the dynamic range); the defect levels of each dimension output by the multimodal large model are matched and verified against the scene prior rule set one by one; the number of rules satisfying the scene prior rules is divided by the total number of scene prior rules to obtain the second value.

[0187] For example, if an image is identified as a portrait scene, there are four prior rules for this scene: if overexposure exists in a portrait scene, the exposure defect level must be at least a significant defect; if skin tone cast exists in a portrait scene, the white balance defect level must be at least a significant defect; the focus defect in a portrait scene should not be lower than a minor flaw; and in a backlit scene (if it is backlit), exposure defects can be tolerated as minor flaws. If the rating result output by the multimodal large model satisfies three of these rules, the second value is 0.75.

[0188] Output consistency judgment refers to comparing the defect levels of the pre-analysis results generated each time the same image data is repeatedly analyzed to determine if they are consistent. The third value can be used to indicate the degree of consistency among the multiple pre-analysis results of the image data; the more stable the consistency among the multiple pre-analysis results, the higher the third value. Optionally, output consistency judgment can be determined by repeatedly analyzing the same image.

[0189] In one possible implementation, the third value can be determined as follows: Perform K independent defect analyses (K≥2, e.g., K=3) on the same image data, using the same prompt word template and model parameter settings each time, to obtain K sets of pre-analysis results; extract the defect level judgment for each rating dimension from each set of pre-analysis results; for each rating dimension, calculate the pairwise consistency rate of the defect level judgments in the K analysis results: for all C(K,2) paired combinations in the K results, calculate the level difference for each pair. If the levels are completely identical, the contribution of the pair is 1; if the difference is one level (e.g., the difference between obvious defects and severe defects), the contribution of the pair is 0.5; if the difference is two or more levels (e.g., the difference between minor flaws and severe defects), the contribution of the pair is 0; the consistency contribution of this dimension is the sum of the contributions of each pair divided by C(K,2); the sum of the consistency contributions of each dimension divided by the total number of dimensions yields the third value.

[0190] In another possible implementation, to reduce computational costs, the third value can be calculated using a simplified method: perform K=2 independent analyses on the image data and determine whether the final comprehensive defect level (the highest level among all dimensions) of the two analyses is consistent. If consistent, the third value equals 1; the third value equals 0.5; otherwise, the third value equals 0.

[0191] In one possible implementation, the confidence assessment result can be determined based on the first, second, and third values ​​as follows: obtain the first weight corresponding to the visual consistency judgment, the second weight corresponding to the scene consistency judgment, and the third weight corresponding to the output consistency judgment; and determine the confidence assessment result based on the first, second, and third values, the first weight, the second weight, and the third weight.

[0192] The first weight can be used to indicate the degree of influence of visual consistency judgment on the evaluation result, the second weight can be used to indicate the degree of influence of scene consistency judgment on the evaluation result, and the third weight can be used to indicate the degree of influence of output consistency judgment on the evaluation result.

[0193] In one possible implementation, the evaluation result Confidence can be expressed as:

[0194]

[0195] Where a represents the first weight, b represents the second weight, c represents the third weight, VES represents the first value, SCS represents the second value, and OSS represents the third value.

[0196] Among them, the first weight a, the second weight b, and the third weight c satisfy a+b+c=1.

[0197] In one exemplary implementation, visual consistency judgment, being directly based on objective pixel data of the image, possesses the highest objectivity and verifiability, and therefore receives the highest weight. Scene consistency judgment and output consistency judgment both serve as auxiliary verification functions, and are assigned relatively lower and equal weights. For example, the first weight a = 0.4 (corresponding to visual consistency judgment), the second weight b = 0.3 (corresponding to scene consistency judgment), and the third weight c = 0.3 (corresponding to output consistency judgment). It is understood that the above weight values ​​are merely examples and can be adjusted according to the evaluation scenario or project requirements in practical applications. For instance, in portrait scenes, the weight of b can be increased to strengthen the constraint effect of the scene rules.

[0198] S402. When the evaluation result is greater than or equal to the first threshold, the pre-analysis result is determined as the analysis result of the image data.

[0199] The first threshold can be a value preset by the user; for example, the first threshold can be 0.75.

[0200] Understandably, when the evaluation result is greater than or equal to the first threshold, the pre-analysis result can be considered to have high reliability and no obvious risk of misjudgment, and can be used directly.

[0201] When the evaluation result is greater than or equal to the first threshold, it indicates that the consistency of the three evaluation dimensions is high (the average score of each dimension is greater than or equal to the first threshold). The pre-analysis result has sufficient visual evidence from the images, is highly consistent with the prior rules of the scene, and the conclusions of multiple analyses are stable and consistent. This indirectly indicates that the judgment of the multimodal large model does not have obvious illusions or misjudgments, and the pre-analysis result can be directly adopted as the final analysis result.

[0202] S403. When the evaluation result is less than the first threshold and greater than or equal to the second threshold, determine the analysis result of the image data based on the image data, the prompt word template, the first value, the second value, and the third value.

[0203] The second threshold can be a value preset by the user. For example, the second threshold can be 0.5, and the first threshold is greater than the second threshold.

[0204] The values ​​of the first and second thresholds can be adjusted in practical applications. For example, when a more conservative and cautious automatic acceptance strategy is needed, the first threshold can be increased (e.g., 0.80 or 0.85); when a more aggressive and automatic automatic acceptance strategy is needed, the first threshold can be decreased (e.g., 0.70).

[0205] Understandably, when the evaluation result is less than the first threshold and greater than or equal to the second threshold, it can be considered that the pre-analysis result has a certain degree of uncertainty or a certain risk of misjudgment. It cannot be used directly and needs to be re-processed by calling the preset large model to determine the analysis result.

[0206] In one possible implementation, during secondary processing, a prompt phrase "Please make a judgment based solely on visible evidence" can be added. Understandably, the first, second, and third values ​​can be used to pinpoint the credibility gaps.

[0207] Specifically, when the evaluation result is less than the first threshold and greater than or equal to the second threshold, it indicates that the pre-analysis result has uncertainty in at least one dimension. For example, there may be some inconsistency between visual evidence and the judgment of the large model (lower first value), or there may be a conflict between the defect level and the scene rules (lower second value), or there may be fluctuations in the results of multiple analyses (lower third value). In this case, the pre-analysis result cannot be directly accepted, but it still has reference value and needs to be corrected and confirmed through secondary processing (i.e., calling the preset large model again and injecting risk warnings).

[0208] S404. When the evaluation result is less than the second threshold, the image data and pre-analysis results are sent to the reviewer's user device so that the reviewer can perform manual review.

[0209] User equipment can refer to the terminal devices of reviewers, such as computers, quality inspection tablets, and mobile phones, which can be used to manually view original images, verify pre-analysis conclusions, and manually correct defect levels.

[0210] Understandably, when the evaluation result is less than the second threshold, the credibility of the pre-analysis conclusion can be considered poor, and there is a high probability of misjudgment, lack of image evidence, conflicting scene logic, serious fluctuations in multiple results, etc. In other words, the pre-analysis conclusion cannot be used and the defect level needs to be manually corrected by manual review.

[0211] exist Figure 4In the illustrated embodiment, when it is necessary to determine the analysis result of image data, a confidence assessment can be performed on the pre-analysis result to determine the assessment result. The assessment result of the confidence assessment can be used to quantify the reliability of the pre-analysis result. When the assessment result is greater than or equal to the first threshold, the pre-analysis result is determined as the analysis result of the image data. At this time, the pre-analysis result can be considered to have high credibility and no obvious risk of misjudgment, and can be directly adopted. When the assessment result is less than the first threshold but greater than or equal to the second threshold, the analysis result of the image data is determined based on the image data, the prompt word template, the first value, the second value, and the third value. At this time, the pre-analysis result can be considered to have a certain degree of uncertainty or a certain risk of misjudgment, and cannot be directly adopted. It is necessary to call the preset large model again for secondary processing. The first threshold and the second threshold can be values ​​preset by the user, and the first threshold is greater than the second threshold. When the assessment result is less than the second threshold, the image data and the pre-analysis result are sent to the user device of the reviewer for manual review. At this time, the pre-analysis conclusion can be considered to have poor credibility and cannot be adopted. In this way, by using the above method, confidence levels are quantified in three dimensions: vision, scene, and output. Differentiated processing is carried out according to two-level thresholds. High confidence is automatically accepted, medium confidence is marked for review and prompts, and low confidence is pushed for manual review. This can reduce defect misjudgment, reasonably reduce the workload of manual labor, and retain the scores of each item for easy traceability and optimization, thus balancing detection efficiency and judgment accuracy.

[0212] Figure 5 This is a schematic diagram of an image defect analysis device provided in an embodiment of this application. Please refer to... Figure 5 The image defect analysis device 10 includes: a construction module 11, an encoding processing module 12, an analysis processing module 13, and a determination module 14, wherein,

[0213] The construction module 11 is used to construct the analysis strategy for defect analysis. The analysis strategy includes at least one of the following: rating dimension, defect level, descriptive words corresponding to the defect level, and preset conditions.

[0214] The encoding processing module 12 is used to encode the analysis strategy to obtain a structured prompt word template;

[0215] The analysis and processing module 13 is used to, in response to receiving the image data to be analyzed sent by the imaging device, input the image data and prompt word template into a preset large model to analyze and process the image data and obtain the pre-analysis results of the image data;

[0216] The determination module 14 is used to evaluate the confidence level of the pre-analysis results and determine the analysis results of the image data based on the evaluation results of the confidence level assessment and the pre-analysis results.

[0217] The image defect analysis device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0218] In one possible design, building module 11 is specifically used for,

[0219] Multiple rating dimensions are obtained, including at least one of the following: exposure dimension, white balance dimension, image processing dimension, and focus dimension;

[0220] Based on multiple rating dimensions, determine at least one sub-dimension corresponding to each rating dimension;

[0221] Obtain preset conditions, defect levels, and defect judgment conditions corresponding to the defect levels;

[0222] An analysis strategy for defect analysis is constructed based on multiple rating dimensions, at least one sub-dimensional corresponding to the multiple rating dimensions, preset conditions, defect levels, and defect judgment conditions corresponding to the defect levels.

[0223] In one possible design, building module 11 is specifically used for,

[0224] Determine the descriptive terms corresponding to the defect level based on the defect level;

[0225] Generate a preset defect description based on the descriptive terms corresponding to the defect level and the defect judgment conditions;

[0226] Based on the preset defect description, multiple rating dimensions, at least one sub-dimension corresponding to the multiple rating dimensions, preset conditions, defect level, and defect judgment conditions corresponding to the defect level, a defect analysis strategy is constructed.

[0227] In one possible design, the analysis and processing module 13 is specifically used for,

[0228] Based on the image data, determine at least one feature label of the image data;

[0229] Determine the scene type of the image data based on at least one feature label;

[0230] Based on the scene type, determine the scene strategy corresponding to the scene type. The scene strategy is used to adapt the image data to the actual scene during analysis and processing.

[0231] The scene strategy, image data, and prompt word template are input into a preset large model to obtain the pre-analysis results of the image data.

[0232] In one possible design, module 14 is specifically used for,

[0233] Based on the pre-analysis results, visual consistency is judged on the image data to obtain a first value, which is used to indicate the accuracy of the image defects identified by the pre-analysis results.

[0234] Based on the pre-analysis results, the image data is judged for scene consistency to obtain a second value, which is used to indicate the degree of logical consistency of the scene determined by the pre-analysis results.

[0235] Based on the pre-analysis results, the image data is output to determine consistency, and a third value is obtained. The third value is used to indicate the degree of consistency of the image data in multiple pre-analysis results.

[0236] The confidence level assessment result is determined based on the first, second, and third values.

[0237] In one possible design, module 14 is specifically used for,

[0238] When the evaluation result is greater than or equal to the first threshold, the pre-analysis result is determined as the analysis result of the image data;

[0239] When the evaluation result is less than the first threshold and greater than or equal to the second threshold, the analysis result of the image data is determined based on the image data, the prompt word template, the first value, the second value, and the third value.

[0240] If the evaluation result is less than the second threshold, the image data and pre-analysis results are sent to the reviewer's user device so that the reviewer can perform a manual review. If the first threshold is greater than the second threshold, the image data and pre-analysis results are sent to the reviewer's user device.

[0241] The image defect analysis device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0242] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 20 may include: a transceiver 21, a processor 22, and a memory 23.

[0243] Processor 22 executes computer execution instructions stored in memory, causing processor 22 to perform the scheme in the above embodiments. Processor 22 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0244] The memory 23 is connected to the processor 22 via the system bus and completes communication between them. The memory 23 is used to store computer program instructions.

[0245] Transceiver 21 can be used to obtain the task to be run and the configuration information of the task to be run.

[0246] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0247] The electronic device provided in this application embodiment can be the terminal device described in the above embodiments.

[0248] This application also provides a chip for executing instructions, which is used to execute the technical solution of the image defect analysis method in the above embodiments.

[0249] This application also provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the technical solution of the image defect analysis method described in the above embodiments.

[0250] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the image defect analysis method in the above embodiments.

[0251] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0252] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0253] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0254] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0255] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0256] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0257] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0258] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0259] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0260] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0261] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for defect analysis of images, characterized in that, include: An analysis strategy for constructing defect analysis is provided, wherein the analysis strategy includes at least one of rating dimensions, defect levels, descriptive terms corresponding to the defect levels, and preset conditions. The analysis strategy is encoded to obtain a structured prompt word template; In response to receiving image data to be analyzed from an imaging device, the image data and the prompt word template are input into a preset large model to analyze and process the image data and obtain the pre-analysis results of the image data; The confidence level of the pre-analysis results is assessed, and the analysis results of the image data are determined based on the confidence level assessment results and the pre-analysis results.

2. The method according to claim 1, characterized in that, An analysis strategy for constructing defect analysis is provided, wherein the analysis strategy includes at least one of rating dimensions, defect levels, descriptive terms corresponding to the defect levels, and preset conditions. Multiple rating dimensions are obtained, including at least one of exposure dimension, white balance dimension, image processing dimension and focus dimension; Based on the multiple rating dimensions, at least one sub-dimension corresponding to the multiple rating dimensions is determined; Obtain the preset conditions, the defect level, and the defect determination conditions corresponding to the defect level; An analysis strategy for defect analysis is constructed based on the multiple rating dimensions, at least one sub-dimensional corresponding to the multiple rating dimensions, the preset conditions, the defect level, and the defect judgment conditions corresponding to the defect level.

3. The method according to claim 2, characterized in that, Based on the multiple rating dimensions, at least one sub-dimensional corresponding to the multiple rating dimensions, the preset conditions, the defect level, and the defect judgment conditions corresponding to the defect level, an analysis strategy for defect analysis is constructed, including: Based on the defect level, determine the descriptive terms corresponding to the defect level; Generate a preset defect description based on the descriptive terms corresponding to the defect level and the defect determination conditions; An analysis strategy for defect analysis is constructed based on the preset defect description, the multiple rating dimensions, at least one sub-dimension corresponding to the multiple rating dimensions, the preset conditions, the defect level, and the defect judgment conditions corresponding to the defect level.

4. The method according to any one of claims 1-3, characterized in that, The image data and the prompt word template are input into a preset large model to analyze and process the image data, obtaining pre-analysis results of the image data, including: Based on the image data, at least one feature label of the image data is determined; The scene type of the image data is determined based on the at least one feature label; Based on the scene type, a scene strategy corresponding to the scene type is determined, and the scene strategy is used to adapt the image data to the actual scene during analysis and processing; The scene strategy, the image data, and the prompt word template are input into a preset large model to obtain the pre-analysis results of the image data.

5. The method according to any one of claims 1-3, characterized in that, The confidence level of the pre-analysis results is assessed, including: Based on the pre-analysis results, the image data is visually consistent to obtain a first value, which is used to indicate the accuracy of the image defects determined by the pre-analysis results. Based on the pre-analysis results, the image data is subjected to scene consistency judgment to obtain a second value, which is used to indicate the degree of logical consistency of the scene determined by the pre-analysis results; Based on the pre-analysis results, the image data is output to determine consistency, and a third value is obtained. The third value is used to indicate the degree of consistency of the multiple pre-analysis results of the image data. The confidence assessment result is determined based on the first value, the second value, and the third value.

6. The method according to claim 5, characterized in that, Based on the confidence assessment results and the pre-analysis results, the analysis results of the image data are determined, including: When the evaluation result is greater than or equal to the first threshold, the pre-analysis result is determined as the analysis result of the image data; When the evaluation result is less than the first threshold and greater than or equal to the second threshold, the analysis result of the image data is determined based on the image data, the prompt word template, the first value, the second value, and the third value. When the evaluation result is less than the second threshold, the image data and the pre-analysis result are sent to the reviewer's user device so that the reviewer can perform a manual review, where the first threshold is greater than the second threshold.

7. An image defect analysis device, characterized in that, include: The module consists of a construction module, an encoding processing module, an analysis processing module, and a determination module. The construction module is used to construct an analysis strategy for defect analysis, the analysis strategy including at least one of rating dimensions, defect level, descriptive words corresponding to the defect level, and preset conditions; The encoding processing module is used to encode the analysis strategy to obtain a structured prompt word template; The analysis and processing module is used to, in response to receiving image data to be analyzed sent by the imaging device, input the image data and the prompt word template into a preset large model to analyze and process the image data and obtain the pre-analysis result of the image data; The determining module is used to perform a confidence assessment on the pre-analysis results, and determine the analysis results of the image data based on the confidence assessment results and the pre-analysis results.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images