Visual detection method and device based on visual language model, equipment and medium

Abstract

The visual detection method, device, equipment and medium based on a visual language model provided by the embodiments of the present disclosure include: obtaining initial image data of a display panel in each preparation process collected by a detection device, and processing the initial image data to obtain target image data; determining a semantic vector, a semantic label vector and attribute information according to the target image data, wherein the attribute information includes panel position information, feature information and defect information; determining a target process case, process knowledge and process strategy corresponding to the target image data according to the semantic vector and the semantic label vector; and generating a field reasoning result according to the semantic vector, the semantic label vector, the attribute information, the target process case, the process knowledge and the process strategy. The cause of the process defect of the display panel in each preparation process is traced, and a foundation is laid for subsequent execution of a corresponding process defect repair action.

Description

Technical Field

[0001] This invention relates to the field of image processing technology and related technical fields, specifically to a visual detection method, apparatus, device, and medium based on a visual language model. Background Technology

[0002] AOI (Automated Optical Inspection) in the panel display industry is an automated inspection device based on optical imaging and image processing technology. It is mainly used to detect defects in panel display products (such as LCD and OLED) during the production process. AOI captures images of the panel surface through high-resolution cameras and uses advanced algorithms to analyze anomalies in the images, such as scratches, stains, color unevenness, and pixel defects, thereby achieving fast and accurate defect detection.

[0003] In the existing technology, the detection results of defects in panels during the production process based on optical inspection equipment require manual annotation of data, and it is impossible to trace the defects in the production process. The cause of the defects and process adjustment parameters need to be determined by manual debugging multiple times, and it cannot be directly converted into an executable strategy and reusable method.

[0004] Given the problems with existing technologies, there is an urgent need for a visual detection method based on a visual language model. Summary of the Invention

[0005] The embodiments described herein provide a visual inspection method, apparatus, device, and medium based on a visual language model, enabling the tracing of the causes of process defects in various manufacturing processes of a display panel, laying the foundation for subsequent repair actions for the corresponding process defects.

[0006] Firstly, according to the content of this disclosure, a visual detection method based on a visual language model is provided, including: The initial image data of the display panel collected by the detection equipment in each manufacturing process is obtained, and the initial image data is processed to obtain the target image data; Based on the target image data, semantic vectors, semantic tag vectors, and attribute information are determined, wherein the attribute information includes panel position information, feature information, and defect information; Based on the semantic vector and the semantic tag vector, determine the target process case, process knowledge and process strategy corresponding to the target image data; Based on the semantic vector, the semantic tag vector, the attribute information, the target process case, the process knowledge, and the process strategy, a field-based reasoning result is generated.

[0007] In some embodiments of this disclosure, the acquisition of initial image data of the display panel collected by the detection device during each manufacturing process, and the processing of the initial image data to obtain target image data, includes: Acquire initial image data of the display panel collected by the detection equipment during each preparation process; Based on the initial image data, the target region in the initial image data is determined using a target localization algorithm, and the target region is labeled to obtain the target image data.

[0008] In some embodiments of this disclosure, determining the semantic vector, semantic tag vector, and attribute information based on the target image data includes: Based on the semantic description prompts and the target image data, determine the semantic vectors and semantic label vectors corresponding to different target regions in the target image data; Based on the target image data, the semantic vectors and semantic tag vectors corresponding to different target regions in the target image data, the attribute information corresponding to different target regions in the target image data is determined.

[0009] In some embodiments of this disclosure, determining the target process case, process knowledge, and process strategy corresponding to the target image data based on the semantic vector and the semantic tag vector includes: Based on the semantic vector and semantic tag vector, construct the target retrieval conditions; Based on the target retrieval conditions, a similarity retrieval is performed in the multimodal knowledge base, and process cases with similarity values ​​that meet a preset threshold are selected as target process cases. The process knowledge and process strategies corresponding to each target process case are then obtained.

[0010] In some embodiments of this disclosure, the step of performing a similarity search in a multimodal knowledge base based on the target retrieval conditions and selecting process cases whose similarity values ​​meet a preset threshold as target process cases includes: Based on the semantic tag vector, obtain the same initial historical image data as the semantic tag vector from the multimodal knowledge base; Based on the similarity between the initial historical semantic vector corresponding to each of the initial historical image data and the semantic vector, target historical image data whose similarity meets a preset threshold is selected; Obtain the target process case corresponding to the target historical image data.

[0011] In some embodiments of this disclosure, generating a field-based reasoning result based on the semantic vector, the semantic tag vector, the attribute information, the target process case, the process knowledge, and the process strategy includes: Construct a preset reasoning instruction template, wherein the preset reasoning instruction template includes problem statement label, root cause candidate label, evidence citation label, suggested action label, expected impact label and confidence level label; Based on the semantic vector, the semantic tag vector, the attribute information, the target process case, the process knowledge, and the process strategy, determine the text content corresponding to different tag identifiers in the preset reasoning instruction template; The field-based reasoning result is determined based on the text content corresponding to different labels in the preset reasoning instruction template.

[0012] In some embodiments of this disclosure, before acquiring image data of the display panel collected by the detection device in different manufacturing processes and processing the image data to obtain target image data, the method further includes: Obtain production line metadata for different manufacturing processes, wherein the production line metadata includes time data, product batch data, equipment parameter data, and process parameter data.

[0013] Secondly, according to the present disclosure, a visual detection device based on a visual language model is provided, comprising: The data processing module is used to acquire the initial image data of the display panel collected by the detection equipment in each manufacturing process, and to process the initial image data to obtain the target image data. The information determination module is used to determine semantic vectors, semantic tag vectors, and attribute information based on the target image data, wherein the attribute information includes panel position information, feature information, and defect information; The process case determination module is used to determine the target process case, process knowledge, and process strategy corresponding to the target image data based on the semantic vector and the semantic tag vector. The reasoning result determination module is used to generate field-based reasoning results based on the semantic vector, the semantic tag vector, the attribute information, the target process case, the process knowledge, and the process strategy.

[0014] Thirdly, according to this disclosure, a computer device is provided, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any of the first aspects.

[0015] Fourthly, according to the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods described in any of the first aspects.

[0016] The visual inspection method, apparatus, device, and medium based on a visual language model provided in this disclosure first acquires initial image data of a display panel in each manufacturing process collected by the inspection device, and processes the initial image data to obtain target image data. Then, based on the target image data, semantic vectors, semantic tag vectors, and attribute information are determined. Furthermore, based on the semantic vectors and semantic tag vectors, target process cases, process knowledge, and process strategies corresponding to the target image data are determined. Finally, based on the semantic vectors, semantic tag vectors, attribute information, target process cases, process knowledge, and process strategies, a field-based reasoning result is generated. By processing the initial image data in each manufacturing process collected through a visual language model, the causes of process defects in the display panel in each manufacturing process can be traced. Then, based on the text content corresponding to the suggested action tag, corresponding process defect repair actions (such as process parameter adjustment, production line cleaning, etc.) are performed to avoid similar process defects in the future. In addition, the generated field-based inference results also include the text content corresponding to the expected impact label and the confidence level label. Based on the text content corresponding to the expected impact label and the confidence level label, the impact and confidence level of the process defect repair action can be predicted, laying the foundation for the subsequent generation of prediction and early warning information.

[0017] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. It should be understood that the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure, wherein: Figure 1 This is a schematic flowchart of a visual detection method based on a visual language model provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of the structure of a visual inspection device based on a visual language model provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure.

[0019] In the accompanying diagram, markers with the same last two digits correspond to the same elements. It should be noted that the elements in the diagram are schematic and not drawn to scale. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are also within the scope of protection of this disclosure.

[0021] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the specification and in the relevant art, and shall not be interpreted in an idealized or overly formal form unless otherwise explicitly defined herein. As used herein, the statement of “connecting” or “coupling” two or more parts together shall mean that these parts are directly joined together or joined through one or more intermediate components.

[0022] The term "embodiment" as used herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of the phrase "embodiment" in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists, A and B exist simultaneously, or B exists. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0024] Furthermore, in all embodiments of this disclosure, terms such as “first” and “second” are used only to distinguish one component (or part of a component) from another component (or another part of a component).

[0025] In the description of this application, unless otherwise stated, "multiple" means two or more (including two), and similarly, "multiple groups" means two or more (including two groups).

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0027] In view of the problems existing in the prior art, this disclosure provides a visual detection method based on a visual language model. Figure 1 This is a flowchart illustrating a visual detection method based on a visual language model provided in this disclosure. Figure 1 As shown, visual detection methods based on visual language models include: S110. Acquire the initial image data of the display panel collected by the detection equipment in each preparation process, and process the initial image data to obtain the target image data.

[0028] Specifically, the visual inspection method based on a visual language model provided in this disclosure analyzes the initial image data of the display panel collected by an automatic inspection device in each manufacturing process, enabling autonomous diagnosis and decision-making regarding problems existing in the display panel in different manufacturing processes. In the visual inspection method based on a visual language model provided in this disclosure, the initial image data of the display panel collected by the inspection device in each manufacturing process is first acquired, and the initial image data is processed to obtain target image data.

[0029] The initial image data of the display panel collected by the automatic inspection equipment in each manufacturing process is not from a single source, but rather a traceable set of visual evidence, including at least one or more of the following: Full FOV image, which is the original field-of-view image collected by the automatic inspection equipment (it may be the original image before stitching, or a large image after stitching), used to provide background texture, pixel structure, neighborhood consistency, etc.; Review image, which is an enlarged / review image output by the automatic inspection equipment for candidate defect points; Multi-illumination / Multi-channel image, which is an image of the same location under bright field / dark field / polarization / different bands, or RGB / grayscale / polarization angle channels, etc.; Derived Maps, such as: gradient map, DoG (Difference of Gaussian) / LoG (Laplacian of Gaussian) response map, frequency domain amplitude map, 2D→3D heatmap, and shadow peak map, etc.

[0030] In a specific implementation, the initial image data of the display panel collected by the detection device in each manufacturing process is acquired, and the initial image data is processed to obtain target image data, including: acquiring the initial image data of the display panel collected by the detection device in each manufacturing process; determining the target region in the image data based on the initial image data using a target localization algorithm, and annotating the target region to obtain target image data.

[0031] Specifically, after acquiring initial image data of the display panel in each manufacturing process, the automatic detection equipment uses target detection algorithms (such as template matching algorithms, Blob (Binary large object, the largest connected region in an image) analysis algorithms, and deep learning algorithms, such as the YOLO algorithm) to determine the target regions in each initial image data. These target regions are then labeled to obtain the target image data. Labeling the target regions in each initial image data improves the efficiency and accuracy of subsequently determining the semantic vectors and semantic label vectors corresponding to each initial image data.

[0032] The methods for annotating target regions in the initial image data include, but are not limited to, the following: bounding boxes, polygonal outlines, binary / multi-class segmentation masks, and key points or skeletons. Among these, bounding boxes are the most common and lightweight annotation method, polygonal outlines are more suitable for irregular defects, binary / multi-class segmentation masks are suitable for annotating scenes with attributes such as shape, area, and boundary, and key points or skeletons are suitable for annotating scenes with pixel structure and fault boundaries.

[0033] In addition, the obtained target image data also includes the coordinate information of the target region in the initial image data and alignment parameters (such as rotation angle, scaling factor, etc.).

[0034] S120. Based on the target image data, determine the semantic vector, semantic label vector, and attribute information.

[0035] The attribute information includes panel location information, feature information, and defect information.

[0036] After the target image data is obtained by annotating the initial image data in step S110, the annotated target image is input into a visual language model. The model then understands the target image data and infers the corresponding semantic vector, semantic label vector, and attribute information. Processing the target image based on the visual language model yields semantic vectors, semantic label vectors, and attribute information, laying the foundation for subsequent operations based on the parsing results. Examples include marking the prepared display panel as defective, triggering a sorting mechanism, recording in a database, or issuing an alarm.

[0037] In a specific implementation, semantic vectors, semantic tag vectors, and attribute information are determined based on the target image data, including: determining the semantic vectors and semantic tag vectors corresponding to different target regions in the target image data based on the semantic description prompt information and the target image data; and determining the attribute information corresponding to different target regions in the target image data based on the target image data, the semantic vectors and semantic tag vectors corresponding to different target regions in the target image data.

[0038] Specifically, semantic description prompts are pre-set prompts for the target object, such as "describe in detail the visual features of the target area of ​​the target image data and provide defect classification labels".

[0039] The semantic description prompts and target image data are input into the visual language model. Based on the visual language model, semantic vectors and semantic label vectors are generated for the target image data. The semantic vector is used to characterize the visual features corresponding to the target area of ​​the target image data (such as the features of defects). The semantic label vector is used to characterize the classification of the visual features (i.e. the defects) corresponding to the target area of ​​the target image data (such as cold solder joints, color mixing, dirt, scratches, etc.).

[0040] It should be noted that the visual language model includes multiple standard semantic labels. The standard semantic labels are defined based on the manufacturing defects that occurred during the historical manufacturing process of the display panel and under different manufacturing processes. The visual language model determines the semantic labels corresponding to each target region based on the feature analysis of each target region in the target image data.

[0041] After determining the semantic vector and semantic label vector corresponding to the target image data, the visual language model can determine the attribute information corresponding to different target regions in the target image data based on the semantic vector, semantic label vector and target image data.

[0042] Specifically, if a display panel has multiple process defects in a manufacturing process, and these defects affect different areas of the display panel, then the target image data includes multiple target areas, and the semantic vectors and semantic label vectors corresponding to different target areas are different. Therefore, the attribute information corresponding to different target areas is different. The attribute information corresponding to the target areas includes at least panel position information, feature information, and defect information.

[0043] A specific example, such as a semantic vector, is: there is a crack in the left region.

[0044] A specific example, a semantic tag vector, is: break.

[0045] A specific example is that the attribute information corresponding to a target area is: {center coordinates: [x,y], area: 150px², feature information: includes multiple lines, color anomaly: dark}.

[0046] S130. Based on the semantic vector and semantic label vector, determine the target process case, process knowledge and process strategy corresponding to the target image data.

[0047] After determining the semantic vectors and semantic label vectors corresponding to different target regions in the target image data in step S120, retrieval conditions are constructed based on the semantic vectors and semantic label vectors, and similar process case retrieval is performed based on the retrieval conditions to ensure that the subsequently generated field-based reasoning results have empirical and theoretical basis.

[0048] In a specific implementation, the target process case, process knowledge, and process strategy corresponding to the target image data are determined based on the semantic vector and semantic tag vector. This includes: constructing target retrieval conditions based on the semantic vector and semantic tag vector; performing similarity retrieval in a multimodal knowledge base based on the target retrieval conditions; selecting process cases whose similarity values ​​meet a preset threshold as target process cases; and obtaining the process knowledge and process strategy corresponding to each target process case.

[0049] The process involves: based on target retrieval conditions, performing similarity retrieval in a multimodal knowledge base, and selecting process cases whose similarity values ​​meet a preset threshold as target process cases. This includes: obtaining initial historical image data with the same semantic tag vectors in the multimodal knowledge base; selecting target historical image data with similarity values ​​meeting a preset threshold based on the similarity between the initial historical semantic vectors and semantic vectors corresponding to each initial historical image data; and obtaining the target process cases corresponding to the target historical image data.

[0050] Specifically, the retrieval process for similarity searching in a multimodal knowledge base using semantic vectors and semantic label vectors as query vectors is as follows: Initial historical image data with the same semantic label vectors as each target region in the target image data is retrieved from the multimodal knowledge base. Since the obtained initial historical image data includes multiple semantic label vectors, it is categorized according to these vectors. Furthermore, initial historical image data retrieved based on semantic label vectors may have the same semantic label vectors but different process defects. For example, the semantic label vectors corresponding to process defects such as scratches and cracks may be the same, but the process defects themselves are different. In this case, similarity matching is required between the initial historical semantic vectors corresponding to the initial historical image data and the semantic vectors corresponding to the target image data. Image data whose similarity values ​​meet a preset threshold are selected as the target historical image data. After determining the target historical image data, the target process case, the corresponding process index document, and the corresponding process strategy are obtained.

[0051] The process knowledge includes the standard operating procedures, material specifications, qualified standard images, and process problems that occur in different process stages of the process case, as well as the processes that cause different process problems. The process strategy includes the methods used to detect the process problems that occur in different process stages of the process case, the parameter adjustment methods, maintenance actions, etc.

[0052] S140. Generate field-based reasoning results based on target image data, semantic tag vectors, attribute information, target process cases, process knowledge, and process strategies.

[0053] In a specific implementation, based on semantic vectors, semantic tag vectors, attribute information, target process cases, process knowledge, and process strategies, a field-based reasoning result is generated, including: constructing a preset reasoning instruction template, wherein the preset reasoning instruction template includes problem statement tag identifiers, root cause candidate tag identifiers, evidence citation tag identifiers, suggested action tag identifiers, expected impact tag identifiers, and confidence level tag identifiers; determining the text content corresponding to different tag identifiers in the preset reasoning instruction template based on semantic vectors, semantic tag vectors, attribute information, target process cases, process knowledge, and process strategies; and determining the field-based reasoning result based on the text content corresponding to different tag identifiers in the preset reasoning instruction template.

[0054] Specifically, the preset reasoning instruction template exemplifies that: based on semantic vectors, semantic label vectors, attribute information, and the provided target process case, analysis is performed, and reasoning results are generated with fields such as problem statement label identifier, root cause candidate label identifier, evidence citation label identifier, suggested action label identifier, expected impact label identifier, and confidence level label identifier.

[0055] By retrieving target process cases, process knowledge, and process strategies that are associated with semantic vectors and semantic tag vectors from a multimodal knowledge base, the causes of process defects in various manufacturing processes of display panels can be traced. Then, based on the text content corresponding to the suggested action tag, corresponding process defect repair actions (such as process parameter adjustment, production line cleaning, etc.) can be performed to avoid similar process defects in the future.

[0056] In addition, the generated field-based inference results also include the text content corresponding to the expected impact label and the confidence level label. Based on the text content corresponding to the expected impact label and the confidence level label, the impact and confidence level of the process defect repair action can be predicted, laying the foundation for the subsequent generation of prediction and early warning information.

[0057] It should be noted that in the above embodiments, the text content corresponding to the problem statement label is determined based on attribute information, the text content corresponding to the root cause candidate label and the evidence citation label can be determined based on the target process case, process knowledge and process strategy, and the text content corresponding to the suggested action label, expected impact label and confidence label is determined based on the target process case, process knowledge and process strategy.

[0058] The visual inspection method based on a visual language model provided in this disclosure first acquires initial image data of the display panel in each manufacturing process collected by the inspection device, and processes the initial image data to obtain target image data. Then, based on the target image data, semantic vectors, semantic tag vectors, and attribute information are determined. Furthermore, based on the semantic vectors and semantic tag vectors, target process cases, process knowledge, and process strategies corresponding to the target image data are determined. Finally, based on the semantic vectors, semantic tag vectors, attribute information, target process cases, process knowledge, and process strategies, a field-based reasoning result is generated. By processing the initial image data in each manufacturing process collected through a visual language model, the causes of process defects in the display panel in each manufacturing process can be traced. Then, based on the text content corresponding to the suggested action tag, corresponding process defect repair actions (such as process parameter adjustment, production line cleaning, etc.) are performed to avoid similar process defects in the future. In addition, the generated field-based inference results also include the text content corresponding to the expected impact label and the confidence level label. Based on the text content corresponding to the expected impact label and the confidence level label, the impact and confidence level of the process defect repair action can be predicted, laying the foundation for the subsequent generation of prediction and early warning information.

[0059] Based on the above embodiments, the method provided in this disclosure further includes: acquiring production line metadata in different manufacturing processes, wherein the production line metadata includes time data, product batch data, equipment parameter data, and process parameter data.

[0060] In step S110, in addition to acquiring image data of the display panel collected by the detection device in different manufacturing processes, it also includes acquiring production line metadata in different manufacturing processes. Based on the acquired production line metadata in different manufacturing processes, after determining the text content corresponding to the suggested action label in step S140, the equipment parameters and process parameters are adjusted based on the equipment parameter data and process parameter data of the production line metadata.

[0061] Furthermore, in the above embodiments, the field-based reasoning results generated by the display panel in each manufacturing process can be stored as a process case in the multimodal knowledge base.

[0062] Based on the above embodiments, this disclosure also provides a visual detection device based on a visual language model. Figure 2 This is a schematic diagram of the structure of a visual inspection device based on a visual language model provided in an embodiment of this disclosure, as shown below. Figure 2 As shown, the visual detection device based on the visual language model includes: The data processing module 210 is used to acquire the initial image data of the display panel collected by the detection equipment in each preparation process, and to process the initial image data to obtain the target image data. The information determination module 220 is used to determine semantic vectors, semantic label vectors and attribute information based on the target image data, wherein the attribute information includes panel position information, feature information and defect information; The process case determination module 230 is used to determine the target process case, process knowledge and process strategy corresponding to the target image data based on the semantic vector and semantic label vector. The reasoning result determination module 240 is used to generate field-based reasoning results based on semantic vectors, semantic tag vectors, attribute information, target process cases, process knowledge, and process strategies.

[0063] The visual inspection device based on a visual language model provided in this disclosure first acquires initial image data of the display panel in each manufacturing process collected by the inspection equipment, and processes the initial image data to obtain target image data. Then, based on the target image data, semantic vectors, semantic tag vectors, and attribute information are determined. Furthermore, based on the semantic vectors and semantic tag vectors, target process cases, process knowledge, and process strategies corresponding to the target image data are determined. Finally, based on the semantic vectors, semantic tag vectors, attribute information, target process cases, process knowledge, and process strategies, a field-based reasoning result is generated. By processing the initial image data in each manufacturing process collected through a visual language model, the causes of process defects in the display panel in each manufacturing process can be traced. Then, based on the text content corresponding to the suggested action tag, corresponding process defect repair actions (such as process parameter adjustment, production line cleaning, etc.) are performed to avoid similar process defects in the future. In addition, the generated field-based inference results also include the text content corresponding to the expected impact label and the confidence level label. Based on the text content corresponding to the expected impact label and the confidence level label, the impact and confidence level of the process defect repair action can be predicted, laying the foundation for the subsequent generation of prediction and early warning information.

[0064] In some embodiments of this disclosure, the acquisition of initial image data of the display panel collected by the detection device during each manufacturing process, and the processing of the initial image data to obtain target image data, includes: Acquire initial image data of the display panel collected by the detection equipment during each preparation process; Based on the initial image data, the target region in the initial image data is determined using a target localization algorithm, and the target region is labeled to obtain the target image data.

[0065] In some embodiments of this disclosure, determining the semantic vector, semantic tag vector, and attribute information based on the target image data includes: Based on the semantic description prompts and the target image data, determine the semantic vectors and semantic label vectors corresponding to different target regions in the target image data; Based on the target image data, the semantic vectors and semantic tag vectors corresponding to different target regions in the target image data, the attribute information corresponding to different target regions in the target image data is determined.

[0066] In some embodiments of this disclosure, determining the target process case, process knowledge, and process strategy corresponding to the target image data based on the semantic vector and the semantic tag vector includes: Based on the semantic vector and semantic tag vector, construct the target retrieval conditions; Based on the target retrieval conditions, a similarity retrieval is performed in the multimodal knowledge base, and process cases with similarity values ​​that meet a preset threshold are selected as target process cases. The process knowledge and process strategies corresponding to each target process case are then obtained.

[0067] In some embodiments of this disclosure, the step of performing a similarity search in a multimodal knowledge base based on the target retrieval conditions and selecting process cases whose similarity values ​​meet a preset threshold as target process cases includes: Based on the semantic tag vector, obtain the same initial historical image data as the semantic tag vector from the multimodal knowledge base; Based on the similarity between the initial historical semantic vector corresponding to each of the initial historical image data and the semantic vector, target historical image data whose similarity meets a preset threshold is selected; Obtain the target process case corresponding to the target historical image data.

[0068] In some embodiments of this disclosure, generating a field-based reasoning result based on the semantic vector, the semantic tag vector, the attribute information, the target process case, the process knowledge, and the process strategy includes: Construct a preset reasoning instruction template, wherein the preset reasoning instruction template includes problem statement label, root cause candidate label, evidence citation label, suggested action label, expected impact label and confidence level label; Based on the semantic vector, the semantic tag vector, the attribute information, the target process case, the process knowledge, and the process strategy, determine the text content corresponding to different tag identifiers in the preset reasoning instruction template; The field-based reasoning result is determined based on the text content corresponding to different labels in the preset reasoning instruction template.

[0069] In some embodiments of this disclosure, before acquiring image data of the display panel collected by the detection device in different manufacturing processes and processing the image data to obtain target image data, the method further includes: Obtain production line metadata for different manufacturing processes, wherein the production line metadata includes time data, product batch data, equipment parameter data, and process parameter data.

[0070] This application also provides a computer device, please refer to 3 for details. Figure 3 This is a basic structural block diagram of the computer device in this embodiment.

[0071] The computer device includes a memory 510 and a processor 520 that are interconnected via a system bus. It should be noted that only a computer device with components 510-520 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components may be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0072] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0073] The memory 510 includes at least one type of readable storage medium, including non-volatile memory or volatile memory, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. RAM may include static RAM or dynamic RAM. In some embodiments, the memory 510 may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the memory 510 may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, or flash card equipped on the computer device. Of course, the memory 510 may include both internal storage units and external storage devices of the computer device. In this embodiment, the memory 510 is typically used to store the operating system and various application software installed on the computer device, such as the program code of the method described above. In addition, the memory 510 may also be used to temporarily store various types of data that have been output or will be output.

[0074] The processor 520 is typically used to perform the overall operation of a computer device. In this embodiment, the memory 510 is used to store program code or instructions, including computer operation instructions. The processor 520 is used to execute the program code or instructions stored in the memory 510 or to process data, such as program code that runs the methods described above.

[0075] In this article, the bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus system can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0076] Another embodiment of this application also provides a computer-readable medium, which may be a computer-readable signal medium or a computer-readable medium. A processor in a computer reads computer-readable program code stored in the computer-readable medium, enabling the processor to execute the functional actions specified in each step or combination of steps in the above method; and to generate means for implementing the functional actions specified in each block or combination of blocks in the block diagram.

[0077] Computer-readable media include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared memory or semiconductor systems, devices or apparatuses, or any suitable combination thereof, wherein the memory is used to store program code or instructions, the program code including computer operation instructions, and the processor is used to execute the program code or instructions of the above-described methods stored in the memory.

[0078] The definitions of memory and processor can be found in the description of the foregoing computer device embodiments, and will not be repeated here.

[0079] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0080] In the various embodiments of this application, the functional units or modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0081] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] Unless otherwise expressly indicated by the context, the singular form of words used herein and in the appended claims includes the plural form, and vice versa. Thus, when referring to the singular, the plural form of the corresponding term is generally included. Similarly, the terms “comprising” and “including” shall be interpreted as including rather than exclusively. Likewise, the terms “including” and “or” shall be interpreted as including unless such interpretation is expressly prohibited herein. Where the term “example” is used herein, particularly when it follows a set of terms, the “example” is merely exemplary and illustrative and should not be considered exclusive or extensive.

[0083] Further aspects and scope of adaptation become apparent from the description provided herein. It should be understood that various aspects of this application may be implemented individually or in combination with one or more other aspects. It should also be understood that the descriptions and specific embodiments herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0084] Several embodiments of this disclosure have been described in detail above. However, it is obvious that those skilled in the art can make various modifications and variations to the embodiments of this disclosure without departing from the spirit and scope of this disclosure. The scope of protection of this disclosure is defined by the appended claims.

Claims

1. A visual detection method based on a visual language model, characterized in that, include: The initial image data of the display panel collected by the detection equipment in each manufacturing process is obtained, and the initial image data is processed to obtain the target image data; Based on the target image data, semantic vectors, semantic tag vectors, and attribute information are determined, wherein the attribute information includes panel position information, feature information, and defect information; Based on the semantic vector and the semantic tag vector, determine the target process case, process knowledge and process strategy corresponding to the target image data; Based on the semantic vector, the semantic tag vector, the attribute information, the target process case, the process knowledge, and the process strategy, a field-based reasoning result is generated.

2. The method according to claim 1, characterized in that, The acquisition and detection equipment collects initial image data of the display panel during each manufacturing process, and processes the initial image data to obtain target image data, including: Acquire initial image data of the display panel collected by the detection equipment during each preparation process; Based on the initial image data, the target region in the initial image data is determined using a target localization algorithm, and the target region is labeled to obtain the target image data.

3. The method according to claim 1, characterized in that, The step of determining the semantic vector, semantic tag vector, and attribute information based on the target image data includes: Based on the semantic description prompts and the target image data, determine the semantic vectors and semantic label vectors corresponding to different target regions in the target image data; Based on the target image data, the semantic vectors and semantic tag vectors corresponding to different target regions in the target image data, the attribute information corresponding to different target regions in the target image data is determined.

4. The method according to claim 1, characterized in that, The step of determining the target process case, process knowledge, and process strategy corresponding to the target image data based on the semantic vector and the semantic tag vector includes: Based on the semantic vector and semantic tag vector, construct the target retrieval conditions; Based on the target retrieval conditions, a similarity retrieval is performed in the multimodal knowledge base, and process cases with similarity values ​​that meet a preset threshold are selected as target process cases. The process knowledge and process strategies corresponding to each target process case are then obtained.

5. The method according to claim 4, characterized in that, The step of performing a similarity search in a multimodal knowledge base based on the target retrieval conditions, and selecting process cases whose similarity values ​​meet a preset threshold as target process cases, includes: Based on the semantic tag vector, obtain the same initial historical image data as the semantic tag vector from the multimodal knowledge base; Based on the similarity between the initial historical semantic vector corresponding to each of the initial historical image data and the semantic vector, target historical image data whose similarity meets a preset threshold is selected; Obtain the target process case corresponding to the target historical image data.

6. The method according to claim 1, characterized in that, The step of generating a field-based reasoning result based on the semantic vector, the semantic tag vector, the attribute information, the target process case, the process knowledge, and the process strategy includes: Construct a preset reasoning instruction template, wherein the preset reasoning instruction template includes problem statement label, root cause candidate label, evidence citation label, suggested action label, expected impact label and confidence level label; Based on the semantic vector, the semantic tag vector, the attribute information, the target process case, the process knowledge, and the process strategy, determine the text content corresponding to different tag identifiers in the preset reasoning instruction template; The field-based reasoning result is determined based on the text content corresponding to different labels in the preset reasoning instruction template.

7. The method according to claim 1, characterized in that, Before acquiring image data of the display panel collected by the detection device during different manufacturing processes, and processing the image data to obtain the target image data, the method further includes: Obtain production line metadata for different manufacturing processes, wherein the production line metadata includes time data, product batch data, equipment parameter data, and process parameter data.

8. A visual inspection device based on a visual language model, characterized in that, include: The data processing module is used to acquire the initial image data of the display panel collected by the detection equipment in each manufacturing process, and to process the initial image data to obtain the target image data. The information determination module is used to determine semantic vectors, semantic tag vectors, and attribute information based on the target image data, wherein the attribute information includes panel position information, feature information, and defect information; The process case determination module is used to determine the target process case, process knowledge, and process strategy corresponding to the target image data based on the semantic vector and the semantic tag vector. The reasoning result determination module is used to generate field-based reasoning results based on the semantic vector, the semantic tag vector, the attribute information, the target process case, the process knowledge, and the process strategy.

9. A computer device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images