Process control method and apparatus, computing device, storage medium, and program product

Abstract

A process control method and apparatus, a computing device, a storage medium, and a program product. The method (300) comprises: acquiring image data to be inspected of a product, said image data comprising an image to be inspected, and the image to be inspected being an image containing a target feature (310); on the basis of the image data to be inspected, determining target feature data, the target feature data comprising statistical data and / or classification data corresponding to the target feature (320); in response to the target feature data satisfying a preset early-warning rule, generating early-warning data (330); acquiring an inspection result for the image to be inspected, the inspection result comprising the type of the target feature contained in said image (340); and, on the basis of the early-warning data and the inspection result, determining, among a plurality of selectable processes, a process to which the product is to be scheduled (350).

Description

Process control method and device, computing device, storage medium and program product Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to a process control method, a process control apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] During the manufacturing process of products such as screens, factors such as equipment quality, parameter tuning, operational standards, and environmental interference can lead to the production of products that fail to meet certain process requirements or even exhibit defects. To promptly detect these issues and improve product quality, products can be inspected after each process or after a portion of each process to promptly identify defective or defective products. For example, automated optical inspection (AOI) equipment can be used to capture images of products, which are then inspected and classified according to process requirements. In existing inspection systems, the image data output by the AOI equipment is typically submitted to manual inspection and diagnosis. For example, image data can be presented to inspectors through a software interface, who then perform individual inspection and diagnosis to identify defects and determine subsequent actions. However, as product production capacity expands, the efficiency of manual inspection and diagnosis may not meet the requirements of increased capacity, which can negatively impact product production efficiency.

[0003] Summary of the Invention

[0004] In view of this, the present disclosure provides a process control method, a process control apparatus, a computing device, a computer-readable storage medium, and a computer program product, which can alleviate, mitigate, or even eliminate the above-mentioned problems.

[0005] According to one aspect of the present disclosure, a process control method is provided, comprising: obtaining image data to be inspected of a product, the image data to be inspected including an image to be inspected, which is an image containing target features; determining target feature data based on the image data to be inspected, the target feature data including statistical data and / or classification data corresponding to the target features; generating warning data in response to the target feature data satisfying a preset warning rule; obtaining a detection result for the image to be inspected, the detection result including the type of target features contained in the image to be inspected; and determining, based on the warning data and the detection result, a process to which the product will be dispatched from a plurality of optional processes.

[0006] In some embodiments, the process control method further includes: in response to the target feature data not satisfying a preset warning rule, not generating warning data, and scheduling the product to a subsequent production process.

[0007] In some embodiments, based on early warning data and detection results, determining the process to which the product will be scheduled among multiple optional processes includes: constructing a state machine, the state machine including multiple states and state transition conditions between the multiple states, the multiple states including an initial state and at least one terminal state, each terminal state in the at least one terminal state corresponds to an optional process in the multiple optional processes; taking the initial state as the starting point, based on the early warning data and detection results, transferring between the multiple states until reaching one of the at least one terminal state.

[0008] In some embodiments, constructing the state machine includes: determining preset warning data that matches the warning data; and constructing the state machine based on a preset state machine model corresponding to the preset warning data.

[0009] In some embodiments, the state machine is defined by: a state set, including multiple states of the state machine; an initial state; a terminal state set, including at least one terminal state among the multiple states; an input parameter table, including multiple input parameters, at least some of the multiple input parameters are associated with early warning data and detection results; a state transition function, configured to describe a state transition condition for transitioning from one state in the state set to another state, wherein the state transition condition is set based on at least one input parameter among the multiple input parameters.

[0010] In some embodiments, determining target feature data based on the image data to be inspected includes: inputting the image to be inspected into an early warning model, the early warning model being configured to output a target feature prediction based on the image to be inspected, the target feature prediction including the classification and / or position of the target feature contained in the image to be inspected; and determining statistical data and / or classification data related to the target feature based on the target feature prediction output by the early warning model.

[0011] In some embodiments, the image data to be inspected also includes coordinate data of target features in the image to be inspected, and, wherein, based on the image data to be inspected, determining the target feature data includes: determining statistical data and / or classification data related to the target feature based on the coordinate data of the target feature.

[0012] In some embodiments, obtaining the detection results for the image to be inspected includes at least one of the following two items: presenting the image to be inspected to an inspector, and determining the type of target features contained in the image to be inspected based on the input of the inspector; inputting the image to be inspected into a detection model, and determining the type of target features contained in the image to be inspected based on the output of the detection model, wherein the detection model is configured to output type prediction data of the target features in the image to be inspected based on the image to be inspected.

[0013] In some embodiments, obtaining image data of a product to be inspected includes: obtaining image data of the product generated by an optical inspection device; cleaning the obtained image data to remove or correct abnormal data or incomplete data in the image data; and extracting image data to be inspected based on the cleaned image data.

[0014] In some embodiments, extracting the image data to be inspected based on the cleaned image data includes: extracting at least one of the following items based on the cleaned image data: the image to be inspected containing the target feature, the coordinate data of the target feature in the image to be inspected, and the attribute data of the product corresponding to the image to be inspected.

[0015] In some embodiments, the plurality of optional processes include at least one of the following: a manual review process, a product repair process, and a manual verification process.

[0016] In some embodiments, the warning data includes warning data corresponding to at least one of: total number warning, size warning, aggregation warning, and recurrence warning.

[0017] In some embodiments, in response to the target feature data satisfying a preset warning rule, warning data is generated including at least one of the following: in response to the total number of target features reaching a total number threshold, a total number warning is generated; in response to the number of target features with a target size reaching a number threshold corresponding to the target size, a size warning corresponding to the target size is generated; in response to the number of target features appearing within a preset size range reaching a preset threshold, an aggregation warning is generated; in response to the repeated appearance of target features of the same classification in the same image area of ​​multiple images to be inspected, a repeated appearance warning is generated.

[0018] According to another aspect of the present disclosure, a process management device is provided, including: a first acquisition module, configured to: acquire image data to be inspected of a product, the image data to be inspected including an image to be inspected, which is an image containing target features; a first determination module, configured to: determine target feature data based on the image data to be inspected, the target feature data including statistical data and / or classification data corresponding to the target features; a generation module, configured to: generate warning data in response to the target feature data satisfying a preset warning rule; a second acquisition module, configured to: acquire a detection result for the image to be inspected, the detection result including the type of target features contained in the image to be inspected; a second determination module, configured to: determine, from multiple optional processes, the process to which the product will be scheduled based on the warning data and the detection result.

[0019] According to yet another aspect of the present disclosure, a computing device is provided, comprising: a memory configured to store computer-executable instructions; and a processor configured to perform the method described according to the aforementioned aspects when the computer-executable instructions are executed by the processor.

[0020] According to another aspect of the present disclosure, a computer-readable storage medium is provided, storing computer-executable instructions. When the computer-executable instructions are executed, the method described in the above aspects is executed.

[0021] According to yet another aspect of the present disclosure, a computer program product is provided, comprising computer executable instructions, which implement the steps of the method described in the above aspects when executed by a processor.

[0022] These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0024] FIG1 exemplarily shows a schematic diagram of a target feature detection solution in the related art;

[0025] FIG2 schematically illustrates an example scenario in which the technical solutions according to some embodiments of the present disclosure may be applied;

[0026] FIG3 schematically shows an example flow chart of a process control method according to some embodiments of the present disclosure;

[0027] 4A, 4B, and 4C schematically illustrate three examples of aggregation warnings according to some embodiments of the present disclosure;

[0028] FIG5 schematically illustrates an example architecture of an early warning model according to some embodiments of the present disclosure;

[0029] FIG6 schematically illustrates another example architecture of an early warning model according to some embodiments of the present disclosure;

[0030] 7A and 7B are schematic diagrams illustrating convolution transformations of deformable convolutions that can be used in early warning models according to some embodiments of the present disclosure;

[0031] FIG8 schematically illustrates another example architecture of an early warning model according to some embodiments of the present disclosure;

[0032] FIG9 schematically illustrates an example structure of a state machine according to some embodiments of the present disclosure;

[0033] FIG10 schematically illustrates an example state transition process of a state machine according to some embodiments of the present disclosure;

[0034] FIG11 schematically illustrates an example architecture of a detection process control system according to some embodiments of the present disclosure;

[0035] FIG12 schematically illustrates an example detection process for detecting a process control system according to some embodiments of the present disclosure;

[0036] FIG13 schematically shows an example block diagram of a process control device according to some embodiments of the present disclosure;

[0037] FIG14 schematically illustrates an example block diagram of a computing device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0038] In the embodiments of the present disclosure, the description of the terms "one embodiment", "another embodiment", "some embodiments", etc. should be understood as that the specific features, structures or steps described in conjunction with the embodiment are included in at least one embodiment of the present disclosure. In this specification, the expression of the above terms is not necessarily for the same embodiment or example. Moreover, the specific features, structures or steps described can be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine the different embodiments or examples described in this specification and the features in different embodiments or examples, unless they are contradictory. In addition, it should be noted that in the present disclosure, the terms "first" and "second" are used for descriptive purposes only, and should not be understood as indicating or implying relative importance or order, nor should they be understood as implicitly indicating the number of technical features indicated.

[0039] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0040] As previously mentioned, in the related art, for the manufacturing process of products such as screens, after each production process, or after a portion of the production process, an image acquisition device can be used to capture images of the product, and then manually inspected and diagnosed to determine whether the product contains defects, whether it can enter the next production process, or whether it needs repair, etc. As shown in Figure 1, the image acquisition device 110 can be set after the production process link A, and it can capture images of products 122 on the production line 120. The captured images can be transmitted to the terminal device 130, and the terminal device 130 can present the images so that the inspection personnel 140 can view the images, mark defects or other anomalies shown in the images, and determine whether the corresponding products can enter the next production process link B, or whether they need to be repaired or scrapped, etc.

[0041] However, as product production capacity expands, a large amount of image data to be tested will flow into the detection system. By analyzing the relevant technical solutions, the applicant found that there are several problems. On the one hand, due to the limited detection efficiency of manual detection, it is difficult to complete the detection of a large amount of image data in a short period of time. Assuming that safe and stable production is guaranteed by a large increase in the number of inspection personnel, it will inevitably lead to a significant increase in labor costs. At the same time, since the inspection process has high requirements for professional knowledge and long-term concentration, a large increase in inspection personnel may also lead to a decline in inspection quality, which in turn affects product quality. On the other hand, if the inspection link is not completed, the product cannot enter the next link of process production, which will inevitably have an adverse effect on production efficiency and production capacity.

[0042] Based on the above considerations, the applicant has proposed a new process control solution for detection systems, which can solve or at least alleviate the above problems. The solution will be described in detail below with reference to the accompanying drawings.

[0043] FIG2 schematically illustrates an example scenario 200 to which the technical solutions according to some embodiments of the present disclosure may be applied.

[0044] As shown in FIG2 , the scene 200 may include an image acquisition device 210, which may be provided between two production process links, such as between link A and link B shown in FIG2 , and configured to capture images of products 222 on the production line 220. Exemplarily, the image acquisition device 210 may be a separate camera device, or may be integrated into other devices. For example, the image acquisition device 210 may be a camera of an AOI device, and the AOI device may automatically scan the products on the production line through the camera to capture images, and perform preliminary processing and analysis on the images to identify pixels or pixel areas that may have defects or other target features. Exemplarily, in the screen production process, the AOI device may scan the substrate (GLASS) through a camera to capture images of the substrate, and compare the captured image with pre-stored standard product image parameters to obtain the location of the defective points in the image, thereby outputting defect data and an image containing defects.

[0045] The images captured by the image acquisition device 210 can be stored in a database or transmitted to the server 230. Alternatively, the images captured by the image acquisition device 210 can be stored in a database or transmitted to the server 230 after preliminary processing and analysis. The server 230 can be deployed with a computer program or code device for executing the process management solution provided by some embodiments of the present disclosure, so as to control the detection process of the target features based on the image. Optionally, the server 230 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0046] In addition, optionally, the scene 200 may further include a terminal device 240. The terminal device 240 may be a desktop computer, a laptop computer, a smart phone, a tablet computer, a smart watch, etc., but is not limited thereto. A client program may be deployed on the terminal device 240 so that the inspection personnel 250 can set up or view the inspection process, as well as perform operations such as image annotation and diagnosis. The terminal device 240 and the server 230 may be directly or indirectly connected via wired or wireless communication (e.g., via a network 260), and this application does not impose any restrictions on this.

[0047] The above-mentioned image acquisition device 210 or other devices including the image acquisition device 210, the server 230 and the optional terminal device 240, the database and other devices can communicate through the network 260. The network 260 can be a wired network connected by cables, optical fibers, etc., or a wireless network such as 2G, 3G, 4G, 5G, Wi-Fi, Bluetooth, ZigBee, Li-Fi, etc., or it can be an internal connection line of one or several devices.

[0048] FIG3 schematically illustrates an example flow chart of a process control method 300 according to some embodiments of the present disclosure. This process control method 300 can be executed, for example, by the server 230 in the scenario 200 shown in FIG2 , or by the terminal device 240, or can be executed collaboratively by the server 230, the terminal device 240, and optionally other devices. The execution entity of method 300 is not specifically limited herein. As shown in FIG3 , method 300 can include steps 310 to 350, as follows.

[0049] In step 310, the image data to be inspected of the product can be obtained, and the image data to be inspected includes the image to be inspected, which is an image containing the target feature. Optionally, the product can be any type of product, the image to be inspected can be an image of the product acquired by an image acquisition device, and the target feature can be any feature that is desired to be detected. In the present disclosure, the image to be inspected can refer to a single or multiple images to be inspected. For example, in the case where the surface size of the product to be inspected is large, the product can be presented by multiple images to be inspected. Alternatively, in some examples, the image data to be inspected of multiple products can be processed simultaneously. In this case, the image to be inspected can include images to be inspected corresponding to multiple products, wherein each product can correspond to one or more images to be inspected. In the present disclosure, the target feature can be understood as an object in the image to be inspected of interest. Exemplarily, the target feature can be a single pixel point with an abnormality or a collection of multiple adjacent pixels. Further exemplarily, the target feature can refer to a product defect presented in the image to be inspected. For example, in the screen manufacturing process, the product may refer to a glass substrate used in the screen production process. The image to be inspected may be acquired by scanning the glass substrate, and the target feature may refer to a defect in the glass substrate. The defect may include various types of defects, such as dust, residue, scratches, or excessively thin or thick circuit lines. Optionally, in addition to the image to be inspected, the image data to be inspected may also include other data, such as the model and batch of the product corresponding to the image to be inspected, the corresponding production process step, and coordinate data indicating the location of the target feature in the image to be inspected.

[0050] In step 320, target feature data may be determined based on the image data to be inspected. The target feature data may include statistical data and / or classification data corresponding to the target features. Optionally, the target feature data may be determined using a pre-trained model, a pre-programmed algorithm, or other methods. Optionally, the target feature data may include any data corresponding to the target features, such as the number of target features, the size of the target features, the classification of the target features, the number of different types of target features, etc. The data type and determination method of the target feature data may be set according to the actual product inspection requirements, and this disclosure does not specifically limit this.

[0051] In step 330, in response to the target feature data satisfying the preset warning rules, warning data can be generated. Exemplarily, the preset warning rules can be determined based on experience or based on big data. For example, the preset warning rules can be set to issue warnings for specific distribution situations of target features, such as the clustered appearance of target features in certain locations, the frequent appearance of target features of certain categories, the excessive number of target features, etc. These situations may indicate that the current product has certain risks that need attention, or may have a significant impact on subsequent process links or the quality of the final product. Optionally, the generated warning data may include one or more warning data, wherein each warning data may correspond to a warning type, and, optionally, each warning data may be a warning code corresponding to the warning type, or may also be statistical or classification data related to the corresponding warning type. Exemplarily, the generated warning data may be a warning set, which may include a plurality of different types of generated warning data.

[0052] In some embodiments, the early warning data may include one or more types of early warning data, such as early warning data corresponding to at least one of the following: total number warning, size warning, cluster warning, and recurrence warning. These early warning data may reflect a high probability of issues requiring attention in the process steps of the current product or production line. By using one or more of these early warning data, early warnings can be issued regarding issues in the process steps of the product or production line, allowing inspection personnel to prioritize these products and prioritize the annotation and diagnosis of their images to be inspected, so that corresponding issues can be discovered and resolved in a timely manner, thereby improving inspection efficiency and overall production efficiency.

[0053] Specifically, a total count warning may refer to a warning related to the number of target features in the image to be inspected. For example, a total count warning may be generated in response to the total number of target features reaching a total count threshold. For example, for a glass substrate, if the number of defect points contained in the image to be inspected exceeds a total count threshold, a total count warning may be generated. Optionally, the warning data for the total count warning may include at least one of the following: the number of target features, a character or code representing the numerical range of the number of target features, etc.

[0054] A size warning may refer to a warning related to the size of a target feature in an image to be inspected. For example, in response to the number of target features having a target size reaching a number threshold corresponding to the target size, a size warning corresponding to the target size may be generated. Exemplarily, a plurality of target sizes may be set for the size of the target feature, and each target size may correspond to a size range. For example, for defects or defect points in a glass substrate, four target sizes may be set: small size, medium size, large size, and oversized size, wherein the small size may correspond to a size range of less than 5 microns, the medium size may correspond to a size range of greater than or equal to 5 microns and less than 15 microns, the large size may correspond to a size range of greater than or equal to 15 microns and less than 100 microns, and the oversized size may correspond to a size range of greater than or equal to 100 microns and less than 10,000 microns. Exemplarily, the number of target features within each size range can be counted, and when the number within a certain size range reaches a preset number threshold corresponding to the size range, a size warning corresponding to the size range is generated. Optionally, the warning data of the size warning may include at least one of the following: the target size for which the warning occurs or the code of the target size, the number of target features that meet the target size for which the warning occurs or the characters or codes that represent the numerical range of the quantity, etc.

[0055] Aggregation warning may refer to a warning related to the aggregation of many target features in a to-be-inspected image within a small range. For example, in response to the number of target features appearing within a preset size range reaching a preset threshold, an aggregation warning may be generated. Exemplarily, the number of target features within a preset size range may be counted, and an aggregation warning may be generated when the statistical number reaches a preset threshold. The warning data for the aggregation warning may include, for example, at least one of the following: the location of the area where aggregation occurs, the shape of the aggregation, the number or density of target features within the area where aggregation occurs, etc. Alternatively, exemplarily, a pre-written algorithm program or a pre-trained machine learning model may be used to determine whether an aggregation warning exists, and to generate warning data for the aggregation warning. For example, for defects or defect points in a glass substrate, an aggregation warning may be generated when the number of defect points appearing within an area of ​​1mm×1mm reaches a preset threshold. Exemplarily, Figures 4A, 4B, and 4C schematically illustrate examples of three aggregation warnings according to some embodiments of the present disclosure. Figures 4A, 4B, and 4C show schematic diagrams of three glass substrates, where A1 to A3, B2 to B3, C1 to C3, and D2 to D3 represent different panel areas. Depending on specific application requirements, the panels obtained by segmenting the glass substrates can subsequently be used to manufacture corresponding products, such as mobile phones, tablet computers, and televisions. Specifically, Figure 4A schematically illustrates left-side clustering, where defects tend to cluster on the left side of the substrate; Figure 4B schematically illustrates upper-center clustering, where defects tend to cluster in the upper-center portion of the substrate; and Figure 4C schematically illustrates partial clustering, where defects tend to cluster at multiple locations, such as the area where panels A1 and B2 meet, the area where panels C1, C2, and D2 meet, and the area to the lower right of D3. In addition, other warning data may be present, such as determining whether a linear warning or a ring warning exists in the image to be inspected. A linear warning may refer to defect points clustering along a line, while a ring warning may refer to defect points clustering in a ring.

[0056] A recurrence warning may refer to the recurrence of target features or target features of the same classification in the same or similar positions in multiple images to be inspected. For example, the classification and position of the target features in the images to be inspected can be determined with the help of a pre-written algorithm program or a pre-trained machine learning model, and then a recurrence warning can be generated by counting the target features in different images to be inspected. For example, a recurrence warning can be generated in response to the recurrence of target features of the same classification in the same image area of ​​multiple images to be inspected. For example, for defects in glass substrates, if the same defect appears in the same position area (for example, within a range of ±1000μm) in the images to be inspected of multiple consecutive glass substrates (for example, multiple glass substrates in a batch), a recurrence warning can be generated, which may mean that there may be problems with the processing equipment or parameters of one or more process links. Optionally, the warning data of the recurrence warning may include at least one of the following: the classification of the recurring target feature, the number of recurrences, the position of the recurring target feature, an indicator indicating the presence of the recurring target feature, etc.

[0057] It should be understood that the above-mentioned warning types are merely exemplary. In practice, required warning rules can be set and required types of warning data can be generated according to the requirements of different application scenarios.

[0058] At step 340, an inspection result for the image to be inspected can be obtained. The inspection result includes the type of target feature contained in the image to be inspected. Exemplarily, the inspection result for the image to be inspected can be obtained through manual inspection or a machine learning model. Alternatively, the inspection result for the image to be inspected can be obtained only when warning data is generated; alternatively, the inspection results for all images to be inspected can be obtained, with priority given to images with warning data; alternatively, the inspection results for images to be inspected with certain warning data can be obtained. Optionally, the inspection result can include a code associated with the target feature, with different codes representing different target feature types. Exemplarily, for glass substrates, target feature data, such as defect data, can be determined for a single glass substrate or a batch of glass substrates. Warning data can be generated when the target feature data meets preset warning rules. Furthermore, if warning data exists, the images of the glass substrate or batch of glass substrates to be inspected can be submitted to manual inspection for inspection and annotation, or submitted to a pre-trained machine learning model, and inspection results can be obtained based on the output. The inspection result can include a defect code indicating the defect type. Optionally, in addition to the type of target feature determined by detecting and marking the image to be inspected, the detection result may also include type data or other feature data related to the target feature obtained through other detection means or through other channels.

[0059] At step 350, the process to which the product will be dispatched can be determined from among multiple available processes based on the early warning data and the test results. The early warning data can reflect different types of risk warnings related to the target characteristics, while the test results can reflect the specific type of target characteristics identified during the test steps. Based on these two factors, the required subsequent processing flow for the product can be empirically determined, such as whether the product requires repair or re-inspection. Alternatively, the process to which the product will be dispatched can be determined based on the early warning data and test results using a pre-programmed algorithm or other method.

[0060] In some embodiments, multiple optional processes may include at least one of the following: a manual re-inspection process, a product repair process, and a manual verification process. For example, if the determined process is a manual re-inspection process, the production operation of the corresponding product can be suspended and no longer scheduled to the next process link, and the inspection personnel can be notified to re-inspect the corresponding product; if the determined process is a product repair process, the corresponding product can be dispatched to a repair site for product repair; if the determined process is a manual verification process, the corresponding product can be marked, and the inspection personnel can be notified to check the scheduling results to determine whether the corresponding product should be dispatched to the next process link, or whether it needs to be repaired or re-inspected. In addition, according to the needs of the production and manufacturing process of a specific product, there may be other optional processes, such as directly entering the next process link, entering the scrapping process, etc. By setting different optional processes, different subsequent processing can be performed on different products based on early warning data and test results, such as repairing products with target features that have a greater impact on subsequent production process links as soon as possible, and re-inspecting products that need to be re-inspected in a timely manner, so as to speed up the efficiency of the detection process and avoid blocking subsequent production as much as possible.

[0061] Through the above-mentioned process control method 300, on the one hand, based on the product's image data to be inspected, according to the preset warning rules, it is possible to predict whether there is a warning, thereby inferring which products have target features that may affect subsequent production and which can be delayed, so that technicians can give priority to detecting and marking the image data to be inspected or the images to be inspected that have warnings; on the other hand, based on the warning data and the detection results, the subsequent processes can be automatically associated, thereby helping to avoid blocking subsequent production due to excessive detection time. For example, in a defect detection scenario, with the help of warning data, it is possible to allow priority detection of defect data with a high risk level or that may have a greater impact on the product's production and manufacturing process, and automatically determine the subsequent process to which the product is scheduled based on the warning data and the detection results, which can effectively realize the automatic scheduling of defective products, avoid products that have a greater impact on the direct production process or product quality from flowing into the subsequent production process links, and promptly repair or otherwise process such products. In this way, the production efficiency of the product can be effectively improved and good product quality can be guaranteed.

[0062] In some embodiments, the above method 300 may further include: in response to the target feature data not satisfying the preset warning rules, not generating warning data, and scheduling the product to the subsequent production process. For example, if the target feature data does not satisfy any of the preset warning rules, the above warning data may not be generated, and the corresponding product may be directly scheduled to the next production process link. Thus, it can be ensured that products that do not contain target features, or that the target features contained have no effect or little effect on the subsequent production process, can enter the subsequent production process as soon as possible, thereby helping to minimize the blockage of the subsequent production process while ensuring product quality. Taking defect detection in screen production as an example, it can be ensured that products without defects, or containing only low-risk or defects with little (or no) impact on production, can enter the subsequent process link as soon as possible.

[0063] In some embodiments, step 310 may include: acquiring image data of the product generated by an optical inspection device; cleaning the acquired image data to remove or correct abnormal or incomplete data in the image data; and extracting the image data to be inspected based on the cleaned image data. Optionally, the optical inspection device may be an optical imaging device such as a camera, or may be another device including an optical imaging device such as a camera. Data cleaning and extraction can promptly remove or correct abnormal or incomplete data, ensuring that subsequent inspection and analysis based on the image data to be inspected can proceed smoothly, improving inspection efficiency and, in turn, overall production efficiency.

[0064] In some embodiments, based on the image data after cleaning, at least one of the following items can be extracted to form the image data to be inspected: the image to be inspected containing the target feature, the coordinate data of the target feature in the image to be inspected, and the attribute data of the product corresponding to the image to be inspected. The image to be inspected containing the target feature has been introduced in the above embodiment and will not be described in detail here. The coordinate data of the target feature in the image to be inspected can be used to describe the position coordinates of the target feature in the image to be inspected, or to describe the position coordinates of the target feature in the surface of the product to be inspected. For example, the latter can also be determined based on the position coordinates of the target feature in the image to be inspected and the order of the image to be inspected in the multiple images corresponding to the product to be inspected and the preset acquisition order of the multiple images of the product. The attribute data of the product corresponding to the image to be inspected can include one or more of the following items: product model, product size, the process link in which the product is located, product type, etc. These attribute data can be used to locate a specific product in a subsequent detection process, or provide auxiliary information in a subsequent detection process to more accurately generate early warning data, more accurately obtain the detection result for the image to be inspected, etc.

[0065] Taking the screen production process as an example, as described above, after a certain production process, the AOI equipment can automatically scan the glass substrate through a camera to capture an image of the glass substrate. Optionally, the captured image can be stored or transmitted so that the inspection process can be performed later based on this image data. Alternatively, the AOI equipment can process the captured image and compare it with pre-stored standard image parameters of the glass substrate to determine the location of the defect point in the glass substrate, and generate corresponding defect data and defect-containing images, and then store or transmit these two and other optional data so that the inspection process can be performed later based on this image data. Optionally, the AOI equipment can directly store the image data of a glass substrate to a designated storage device or database, or it can also package the image data of a batch of glass substrates and store it in a designated storage device or database, where the number of glass substrates in a batch can be pre-set or can be determined based on production conditions. For example, after the AOI device stores the image data of a batch or a glass substrate in a designated storage device or database, such as a DFS (Distributed File System), a notification can be sent to the device or system executing method 300, so that the device or system can obtain the image data from the designated storage location (such as the storage location indicated by the notification). After obtaining the image data, it can be pre-processed. For example, the image data can be cleaned. For example, the defect data file in the image data can be parsed to check whether there are problems such as missing key fields and abnormal values. If so, the inspection personnel or other management personnel can be notified to perform manual inspection and processing. In addition, for example, data extraction can be performed on the cleaned data, such as extracting various data required for subsequent inspection processes, such as coordinate data of target features in the image, product attribute data, etc., where the product attribute data can include one or more of the following: basic information of the glass substrate, such as the type and model of the corresponding product; basic information of the panel, such as various technical information related to the panel, such as panel size; process link information, such as the name and code of the current process link; and so on. The extracted data can be stored in a database or other storage device and used as the above-mentioned image data to be inspected.

[0066] In some embodiments, in addition to the image to be inspected, the image data to be inspected may also include the coordinate data of the target features in the image to be inspected mentioned above. In such an embodiment, the aforementioned step 320 may include: determining statistical data and / or classification data related to the target features based on the coordinate data of the target features. Exemplarily, the coordinate data of the target features can be obtained by analyzing and processing the image to be inspected, for example, by comparing the image to be inspected with pre-stored standard image parameters of the product. Exemplarily, based on the coordinate data of the target features, one or more of the following data can be obtained by statistics: the total number of target features in the image to be inspected, the number of target features with a target size, the number of target features appearing within a preset size range, etc. In addition, optionally, based on the coordinate data of the target features, other statistical data or classification data corresponding to the target features can also be determined. By generating some or all of the statistical data and / or classification data based on the coordinate data of the target features, it is helpful to improve the efficiency of obtaining these data, thereby improving the efficiency of determining early warning data.

[0067] In some embodiments, the aforementioned step 320 may include: inputting the image to be inspected into an early warning model, the early warning model being configured to output a target feature prediction based on the image to be inspected, the target feature prediction including the classification and / or location of the target features contained in the image to be inspected; and determining statistical data and / or classification data related to the target features based on the target feature prediction output by the early warning model. For example, the classification of target features may refer to classifications related to target features of interest during the early warning phase, such as the clustered location type of target features, the clustered shape type of target features, the broad type of target features, etc., wherein the broad type may be understood as the type of target features of interest during the early warning phase, which may be a specific target feature type (such as a defect code), or a broad type may correspond to multiple specific target feature types, such as major defects, medium defects, and minor defects classified by importance. For example, the location of the target feature may refer to the specific location, clustered location, etc. of the target feature. It should be understood that the target feature prediction obtained with the help of the early warning model can be designed according to specific application requirements, such as based on the type of early warning data desired to be generated. Exemplarily, the early warning model can be pre-trained based on sample data. Exemplarily, a sample data set can be constructed based on image data generated during the historical production and manufacturing process of the product. For example, the images generated during the historical production and manufacturing process of the product can be annotated with the help of manual annotation or other annotation methods, such as the classification of the target features contained in the annotated image, the location of the target features, the type of aggregation of the target features, the aggregation location of the target features, etc. Furthermore, the model can be trained using the annotated sample data, such as using gradient descent or other methods to perform the training process, and the trained model can be used as the above-mentioned early warning model, wherein the trained early warning model can output the classification and / or location data of the target features contained therein based on the input image to be inspected, and optionally output other expected data. Optionally, the early warning model can have any desired model structure, which can be constructed based on an existing model (such as a YOLOv8 model, etc.), or it can be a self-designed model. Exemplarily, based on the target feature prediction data output by the early warning model, one or more of the following data can be obtained: the aggregation of target features in the image to be inspected, target features of the same classification that appear repeatedly at the same or similar positions in multiple images to be inspected, etc. With the help of this pre-trained risk warning model, the classification / location data of target features can be obtained quickly and accurately, which helps to improve the efficiency of determining warning data and increase the richness of warning types.

[0068] Exemplarily, the early warning model can have the model structure shown in Figure 5. As shown in Figure 5, the input of the early warning model 500 can be an image of dimension H×W×C, where H, W, and C represent the height, width, and number of channels of the image, respectively. The early warning model 500 may include a backbone network 510, a feature enhancement network 520, and a detection head network 530, wherein the backbone network 510 can perform feature extraction, the feature enhancement network 520 can perform feature enhancement on the extracted features, and the detection head network 530 can perform target detection on the feature map after feature enhancement and output the detection results. Exemplarily, the backbone network 510 may include several CBS modules, several C2f modules, and an SPPF module. The structure of the CBS module is shown in box 540, which can be composed of a convolutional layer Conv, a normalization layer BN (BatchNormalization), and an activation function layer Silu, where S=2 indicates a stride of 2, and K=3 indicates a convolution kernel size of 3×3. The structure of the C2f module is shown in box 550. It can be composed of two CBS modules at the beginning and end, a split module, n bottleneck modules, and a concatenation module. The structure of the bottleneck module is shown in box 560. It can be composed of two CBS modules. n can be an integer greater than 0 and can be set as required. For example, as shown in Figure 5, n=3 can be set. The SPPF (Spatial Pyramid Pooling-Fast) module is a fast spatial pyramid pooling module. Its structure can refer to the module structure in the relevant technology and will not be described in detail here. Exemplarily, the feature enhancement network 520 may include several C2f modules, concatenation modules, and upsampling modules. The concatenation module can concatenate different features, and the upsampling module can upsample feature maps. For example, the detection head network 530 can include multiple detection head branches, such as the three Detect modules shown in Figure 5, where each detection head branch can perform target detection at different sampling dimensions to detect large targets, medium targets, and small targets respectively. In addition, according to specific needs, a larger number of detection head branches can be designed to optimize detection accuracy, or a smaller number of detection head branches can be designed to reduce model complexity. The early warning model 500 shown in Figure 5 can achieve faster reasoning speed and higher accuracy.

[0069] In some embodiments, deformable convolution can be used to replace part of the conventional convolution in the early warning model 500, so as to adaptively adjust according to the shape and proportion of the detected image. By using irregular convolution kernels, the disadvantage of insufficient sampling of the traditional fixed rectangular convolution kernel structure is solved, and the detection performance of the early warning model is improved. For example, Figure 6 schematically shows an early warning model 600 according to such an embodiment, wherein, as indicated by the dotted box 610, the C2f module in the early warning model 500 can be replaced by the C2f_DCN module, wherein the convolution operation in the C2f module is replaced by the deformable convolution to obtain the C2f_DCN module. Schematically, Figures 7A and 7B show the schematic diagram of the deformable convolution. For conventional two-dimensional standard convolution, for a certain sampling point p0, the output eigenvalue can be determined according to Formula 1:

[0070] Where, R={(-1,-1),(-1,0),…,(0,1),(1,1)}, p n Represents the position element in R, w(p n ) is p n The convolution kernel weight value at position, x is the feature map to be detected, x(p0+p n ) identifies the feature value of the sampling position of the feature map to be detected. In deformable convolution, the output features can be determined according to formula 2:

[0071] Where Δp n is the offset added to the regular convolution sampling point, Δm n In FIG7A , the offset Δp is schematically shown by an arrow. n . As shown in FIG7B , first, as in a conventional convolutional neural network, a conventional convolution kernel can be used to extract features from the input feature map. Then, for the obtained feature map, an additional bias layer can be used to obtain the bias domain of the convolution kernel sampling point. The number of channels can be 2N, which can include the bias in the x and y directions on the 2D plane. The size of the obtained bias domain can be consistent with the input feature map, from which the bias matrix of the convolution kernel sampling can be obtained, thereby obtaining the offset Δp n In addition, the weight coefficient Δm introduced n Irrelevant background information can be suppressed, so that the interference of irrelevant content can be reduced when the model performs feature learning.

[0072] In some embodiments, an attention mechanism CBAM module can be further introduced, which can enable the early warning model to pay more attention to small target information, enhance attention to the target, improve the recognition and positioning accuracy of small targets, and suppress irrelevant features. For example, Figure 8 schematically shows an early warning model 800 according to such an embodiment. As shown in the dotted box 810, an attention mechanism module CBAM (Convolutional Block Attention Module) can be introduced on the basis of the early warning model 600 or the early warning model 500. For example, the CBAM module can include a channel attention module and a spatial attention module. The former can be used to enhance the feature expression of each channel. For example, it can perform a pooling operation on each channel and input the pooling result into a fully connected layer. Then, it passes through an activation function to obtain the weight of each channel, and based on these weights, the feature map of each channel is weighted to emphasize more important channels and suppress irrelevant channels. The latter can be used to emphasize the importance of different positions in the image. For example, it can perform a pooling operation on the feature map along the channel dimension and splice the pooling result along the channel dimension. Then, through convolution processing and activation function, spatial attention weights are obtained. Then, the obtained weights can be applied to the feature map to highlight important image areas. Optionally, the introduced attention mechanism module can also have other structures.

[0073] In some embodiments of the present disclosure, as described above, statistical data and / or classification data corresponding to the target features can be determined based on the coordinate data of the target features in the image to be inspected or based on a pre-trained warning model. Alternatively, in some embodiments, statistical data and / or classification data corresponding to the target features can also be determined based on both the coordinate data of the target features in the image to be inspected and the pre-trained warning model. With the advantages of both, it is possible to more conveniently and accurately generate a variety of warning data based on a variety of warning rules, so as to more effectively warn of various situations or problems that need to be paid attention to in the product production process, thereby improving the overall detection efficiency and detection quality. In addition, optionally, the above-mentioned warning analysis can also be performed with the help of other methods, such as reasoning based on a pre-built knowledge base to obtain warning data, etc.

[0074] In some embodiments, a state machine (or finite state machine (FSM)) can be used to determine the process to which the product will be scheduled based on the early warning data and the test results. In such embodiments, the aforementioned step 350 may include: constructing a state machine, wherein the state machine may include multiple states and state transition conditions between the multiple states, the multiple states including an initial state and at least one terminal state, each terminal state in the at least one terminal state corresponds to an optional process in a plurality of optional processes; starting from the initial state, based on the early warning data and the test results, transferring between the multiple states until reaching one of the at least one terminal state. Exemplarily, each terminal state in the at least one terminal state may correspond to one of the aforementioned optional processes, such as a manual re-inspection process, a product repair process, a manual verification process, etc. And, exemplarily, the state transition conditions between the multiple states may be related to the data in the early warning data and the test results, so as to achieve the scheduling of the product to a suitable subsequent process in the case of different early warning data and test results. Therefore, we can combine the state machine to perform correlation analysis, use the warning data and test results as the trigger variables of the state machine, and automatically associate the subsequent processes, so as to flexibly schedule products to different subsequent processes, avoid blocking subsequent production due to long detection time, and improve production efficiency.

[0075] In some embodiments, the constructed state machine can be defined by the following: a state set, including multiple states of the state machine; an initial state; a terminal state set, including at least one terminal state among the multiple states; an input parameter table, including multiple input parameters, at least some of the multiple input parameters are associated with early warning data and detection results; a state transition function, configured to describe the state transition condition for transitioning from one state in the state set to another state, wherein the state transition condition is set based on at least one input parameter among the multiple input parameters.

[0076] Exemplarily, a state machine may be defined by a five-tuple, for example, a state machine M may be defined as M=<S,A,h,s0,F> , where S can be a non-empty finite set of M, s0 can be the initial state, s0∈S, F can be the terminal state set of M, F∈S, and F can include multiple terminal states. For any given f∈F, f can be called the terminal state of M, A can be the input parameter table of M, h can be the state transition function δ: S×A→S, h(s,a)=s' can indicate that receiving input parameter a in state s will cause the state to transition to s'. For example, Figure 9 shows a state machine structure. As shown in Figure 9, the state machine 900 includes an initial state s0, a terminal state f, and states s1 to s4. As shown in the figure, each node in the state machine 900 can represent a state in its state set, and the connection between the nodes can represent the state transition function. For example, when the input parameter a1 is received in state s0, the transition function h(s0,a1) is called to perform state transition, and after the transition, the state s1 can be entered. The input parameter table of the state machine 900 may include alarm[], code[], and policy, wherein alarm[] may be the aforementioned warning data or a character representing the aforementioned warning data; code[] may be a detection code set, which may include the type of target feature of interest or a character used to represent the type of target feature of interest, such as a defect code of interest in the screen manufacturing process. Subsequently, based on the detection results, it may be determined whether the type of target feature in code[] (such as a defect code) exists, and the corresponding state transition may be implemented; policy may be a policy set, which may include a data threshold policy or an existence policy related to the target feature. For example, the policies in the policy set may define the determination of whether certain warning data exceeds a specified threshold, or whether certain warning data exists, etc. For example, the policy may define the determination of whether the total number of target features is greater than a certain threshold, whether the number of target features of a certain size is greater than a certain threshold, etc. The terminal state f may be a terminal state in the terminal state set F, which may, for example, represent one of the aforementioned optional processes.

[0077] In some embodiments, the structure of the state machine can be pre-defined. After providing the warning data and the test results, the analysis operation can be automatically performed to determine the subsequent process to which the product should be dispatched. Alternatively, in some embodiments, the state machine can be dynamically matched. For example, the state machine can be constructed by the following steps: determining preset warning data that matches the warning data; and constructing the state machine based on a preset state machine model corresponding to the preset warning data. For example, multiple state machines can be pre-defined, each of which can have corresponding preset warning data. The preset warning data can, for example, specify the number of warnings contained in the warning data, the type of warning, the value of some warning data, etc. Subsequently, after the warning data is generated according to the steps described in the above embodiments, the preset warning data that matches the warning data can be determined. For example, the warning data can be considered to match the preset warning data when the warning data is exactly the same as the preset warning data, when the type of warning data contained in the warning data is the same as the type specified by the preset warning data, when the number of warning data contained in the warning data is the same as the number specified by the preset warning data, or when the value of some warning data contained in the warning data falls within the value range of the corresponding warning data specified by the preset warning data. Optionally, the preset warning data can be defined according to specific application requirements, and the conditions for matching the warning data with the preset warning data can also be designed according to specific application requirements, which can be selected from one or more of the above conditions, or can also include conditions different from the above conditions. For example, after determining the preset warning data that matches the warning data, the corresponding predefined state machine can be found, and the predefined state machine can be defined by the aforementioned five-tuple, for example. After finding the corresponding predefined state machine, it is possible to enter the initial state, such as the s0 node shown in Figure 9, and determine the input parameters in the input parameter table based on one or more of the warning data, detection results, metadata, etc. By predefining multiple selectable state machines based on experience or other basis, and dynamically matching the state machine based on the warning data during the actual detection process, a state machine suitable for the current warning situation can be conveniently and adaptively constructed, thereby better adapting to various warning situations and improving the accuracy and reliability of correlation analysis.

[0078] In some embodiments, after matching to a suitable state machine, a detection task can be generated and stored for the image data to be inspected that is currently being inspected. The generated detection task can be provided to a manual detection process or an artificial intelligence (AI) detection process for detection and annotation. Exemplarily, the image to be inspected can be presented to the inspector, and, based on the input of the inspector, the type of target feature contained in the image to be inspected is determined. For example, the inspector can obtain the generated detection task through the interface of the detection system on his terminal device, and detect and annotate the defect type involved in each image to be inspected. Optionally, the annotated defect type can be determined by observing the image to be inspected, or it can be determined in combination with other detection means. Alternatively, exemplarily, the image to be inspected can be input into a detection model, and, based on the output of the detection model, the type of target feature contained in the image to be inspected is determined, wherein the detection model is configured to output prediction data of the type of target feature in the image to be inspected based on the image to be inspected. For example, relevant inspection personnel can obtain the generated inspection tasks through the interface of the inspection system on their terminal devices, and provide the images to be inspected therein to the inspection model. The inspection model can be pre-trained and can be trained to determine the type of target features (such as defects) contained therein based on the input images. Exemplarily, after completing the inspection and annotation of all images to be inspected in a certain inspection task, the inspection results can be saved by clicking "Done" or a similar button. These inspection results can subsequently be used together with the early warning data to control the state transition process in the state machine. Optionally, the above-mentioned inspection process with the help of manual or AI can be implemented after the state machine is constructed, or in parallel with it, or it can be implemented after the early warning data is generated and before the state machine is constructed. The present disclosure does not specifically limit the execution order of the two.

[0079] After obtaining the test results, a correlation analysis can be performed based on the constructed state machine to determine the subsequent process to which the product is scheduled. For ease of understanding, FIG10 schematically illustrates an example state machine 1000 according to some embodiments of the present disclosure. In state machine 1000, the warning data alarm[] includes the number of defects and the number of defect clusters. The number of defects can be understood as the number of target features mentioned in the previous embodiments (such as the number of defects or defect points), and the number of defect clusters can be understood as the presence of cluster warnings. code[GGS] can refer to defect codes that are of particular concern in the current warning situation, are prone to occur, or have a significant impact on subsequent production or product quality. As shown in FIG10, s0 can be the initial state of state machine 1000. In state s0, it can be determined whether there are defect clusters in the warning data alarm[]. If so, the state enters state s1; if not, the state enters state s4. In state s1, it can be determined whether there are GGS in the test results, that is, whether there are target features of the corresponding type. If so, the state enters state s2; if not, the state enters state s3. In state s4, it can be determined whether the number of defects in the warning data alarm[] is greater than 20. If so, the system enters state s3; if not, it enters state s5. In states s2, s3, and s5, the system can transition to the terminal state f. Specifically, in state s2, the Hold operation can be triggered, which means the product is held and no longer dispatched to the next station, and the inspection personnel are notified to re-inspect the product or the image to be inspected; in state s3, the Repair operation can be triggered, which means the product is dispatched to the Repair station for repair; in state s5, the Mark operation can be triggered, which means the defective condition of the product is marked and submitted to the inspection personnel for review to determine which subsequent process the product should be dispatched to. In some embodiments, the state machine records each state transition process so that the relevant process can be traced back when necessary. For example, when a problem occurs, the source of the problem can be determined by tracing back the record. It should be understood that the state machine shown in Figure 10 is merely exemplary. In practice, different input parameter tables, different numbers of states, different state transition processes, different terminal states, etc. can be designed according to the requirements of specific application scenarios.

[0080] In some embodiments, various embodiments of the above method 300 can be executed by a detection process control system 1100 shown in Figure 11. As shown in Figure 11, the detection system 1100 may include a pre-processing module 1110, an early warning module 1120, a state machine analysis and control module 1130, and an artificial / AI detection module 1140. Specifically, the preprocessing module 1110 can execute the data preprocessing process described in the previous embodiments, for example, data cleaning and data extraction of image data output by optical acquisition equipment such as AOI equipment, and convert it into image data to be inspected that can be used by the early warning module 1120; the early warning module 1120 can include an early warning model obtained by training based on sample data, and can generate early warning data based on the image data to be inspected, with the help of the early warning model and preset early warning rules according to the process described in the previous embodiments; the state machine analysis and control module 1130 can use the state machine according to the embodiments described above to perform correlation analysis based on the early warning data and the detection results obtained by manual or AI detection, and automatically determine the subsequent process to which the product should be scheduled. For example, the early warning data received from the early warning module 1120 can be matched with the preset early warning data, and a suitable state machine can be dynamically constructed, and then combined with the detection results, the state transition can be automatically triggered, the subsequent process can be determined, and the recording of the detection process can be completed; the manual / AI detection module 1140 can determine the detection results for the target features in the image to be inspected by presenting the image to be inspected to the inspection personnel or inputting the image to be inspected into the AI ​​model.

[0081] To further facilitate understanding, FIG12 schematically illustrates an example detection process 1200 for detecting a process control system according to some embodiments of the present disclosure. As shown in FIG12 , the entire system operation process can be divided into four parts: a pre-processing part, an early warning part, a state machine analysis and control part, and a manual / AI detection part. These four parts can be implemented by, for example, four modules in system 1100, respectively.

[0082] In the preprocessing section, after receiving a notification from the AOI device, the AOI output data can be pulled. As described in the previous embodiments, the AOI device can capture an image of the product, analyze and process the captured image, and compare it with pre-stored standard product image parameters to obtain coordinate data of target features in the image, such as defect coordinate data. The AOI device can then store the file containing this target feature coordinate data along with the image containing the target feature and send a notification to the inspection process control system. After pulling the data output by the AOI device, steps such as data cleaning and data extraction can be performed. These steps can be performed with reference to the previous embodiments to obtain image data suitable for early warning analysis. The image data can include one or more of the following: an image containing the target object, coordinate data of target features in the image, and metadata describing the model or parameters of the product (such as a glass substrate or panel).

[0083] In the early warning section, model training can be performed in advance based on sample data to generate an early warning model. This early warning model can be used to predict the pre-processed image to be inspected. The prediction results can include one or more of the following: the classification of target features in the image to be inspected, the location of target features, the clustering of target features, etc. Subsequently, based on the output of the early warning model and / or the coordinate data of the target features in the image to be inspected, analysis can be performed according to pre-set early warning rules to generate and output early warning data. In response to the existence of early warning data, the operation of the state machine analysis control section can continue to be executed; otherwise, if no early warning data exists, the corresponding product can be directly scheduled to the subsequent production process flow to avoid blocking the production process.

[0084] In the state machine analysis and control part, dynamic matching can be performed based on the early warning data to match the appropriate state machine in the preset state machine to complete the construction of the state machine. Subsequently, the detection task can be generated based on the image data to be inspected or the image to be inspected, and handed over to manual / AI detection to obtain the detection results for the image to be inspected. Then, the constructed state machine can be used to perform correlation analysis on the early warning data and the detection results, wherein the state transition in the state machine can be triggered according to the state transition function, based on the early warning data and the detection results. After the correlation analysis results of the state machine, the product can be automatically scheduled to the corresponding subsequent process.

[0085] In the manual / AI detection part, the image to be inspected can be presented to the inspector, or the image to be inspected can be input into the detection model to obtain and output the detection results.

[0086] It should be understood that the process 1200 shown in FIG. 12 is merely exemplary. In practice, some of the process steps may be omitted or replaced by other steps, and the process may also include other additional steps.

[0087] Figure 13 schematically illustrates an example block diagram of a process control apparatus 1300 according to some embodiments of the present disclosure. As shown in Figure 13 , the apparatus 1300 may include a first acquisition module 1310 , a first determination module 1320 , a generation module 1330 , a second acquisition module 1340 , and a second determination module 1450 , as follows.

[0088] The first acquisition module 1310 can be configured to: acquire the image data to be inspected of the product, the image data to be inspected includes the image to be inspected, and the image to be inspected is an image containing target features; the first determination module 1320 can be configured to: determine the target feature data based on the image data to be inspected, the target feature data includes statistical data and / or classification data corresponding to the target features; the generation module 1330 can be configured to: generate warning data in response to the target feature data meeting the preset warning rules; the second acquisition module 1340 can be configured to: acquire the detection result for the image to be inspected, the detection result includes the type of target features contained in the image to be inspected; the second determination module 1450 can be configured to: determine the process to which the product will be scheduled among multiple optional processes based on the warning data and the detection result.

[0089] It should be understood that the apparatus 1300 can be implemented in software, hardware, or a combination of software and hardware. Multiple different modules can be implemented in the same software or hardware structure, or one module can be implemented by multiple different software or hardware structures.

[0090] In addition, the apparatus 1300 can be used to implement the method 300 described above, the relevant details of which have been described in detail above and will not be repeated here for the sake of brevity. The apparatus 1300 can have the same features and advantages as those described with respect to the above method.

[0091] Figure 14 schematically shows an example block diagram of a computing device 1400 according to some embodiments of the present disclosure. For example, it may represent the server 230 or the terminal device 240 in Figure 2, or may also represent other types of computing devices for deploying the apparatus 1300 provided by the present disclosure or executing the method 300 provided by the present disclosure.

[0092] As shown, example computing device 1400 includes a processing system 1401, one or more computer-readable media 1402, and one or more I / O interfaces 1403 that are communicatively coupled to one another. Although not shown, computing device 1400 may also include a system bus or other data and command transmission system that couples the various components to one another. The system bus may include any one or a combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of a variety of bus architectures, or may also include, for example, control and data lines.

[0093] Processing system 1401 represents functionality that performs one or more operations using hardware. Thus, processing system 1401 is illustrated as including hardware elements 1404 that can be configured as processors, functional blocks, and the like. This can include implementing application-specific integrated circuits (ASICs) or other logic devices formed using one or more semiconductors in hardware. Hardware elements 1404 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor can be comprised of (a plurality of) semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor-executable instructions can be electronically executable instructions.

[0094] Computer-readable medium 1402 is illustrated as including memory / storage device 1405. Memory / storage device 1405 represents a memory / storage device associated with one or more computer-readable media. Memory / storage device 1405 may include volatile storage media (such as random access memory (RAM)) and / or non-volatile storage media (such as read-only memory (ROM), flash memory, optical disk, magnetic disk, etc.). Memory / storage device 1405 may include fixed media (e.g., RAM, ROM, fixed hard drive, etc.) and removable media (e.g., flash memory, removable hard drive, optical disk, etc.). Exemplarily, memory / storage device 1405 can be used to store the images to be inspected, target feature data, early warning data, detection results, etc. mentioned in the above embodiments. Computer-readable medium 1402 can be configured in various other ways as further described below.

[0095] One or more input / output interfaces 1403 represent functions that allow a user to type commands and information into the computing device 1400 and also allow the information to be presented to the user and / or sent to other components or devices using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice input), a scanner, a touch function (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., a device that can detect motion that does not involve touch as a gesture using visible or invisible wavelengths (such as infrared frequencies)), a network card, a receiver, and the like. Examples of output devices include a display device (e.g., a monitor or projector), a speaker, a printer, a tactile response device, a network card, a transmitter, and the like. Exemplarily, in the embodiments described above, the input device can allow a user (e.g., an inspector) to perform various interactive operations, such as inspecting and annotating an image to be inspected, and the output device can allow a user to view various data and processes in the inspection process.

[0096] The computing device 1400 also includes a process control application 1406. The process control application 1406 may be stored as computer program instructions in the memory / storage device 1405. The process control application 1406, together with the processing system 1401 and the like, may implement all functions of the various modules of the process control device 1300 described with respect to FIG.

[0097] Various techniques may be described herein in the general context of software, hardware, elements, or program modules. Generally, these modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. As used herein, the terms "module," "function," etc. generally refer to software, firmware, hardware, or a combination thereof. A feature of the techniques described herein is that they are platform-independent, meaning that these techniques can be implemented on a variety of computing platforms with a variety of processors.

[0098] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include various media accessible by computing device 1400. By way of example and not limitation, computer-readable media may include "computer-readable storage media" and "computer-readable signal media."

[0099] As opposed to a mere signal transmission, carrier wave, or signal itself, "computer-readable storage media" refers to media and / or devices, and / or tangible storage devices, capable of persistently storing information. Thus, computer-readable storage media refers to non-signal-bearing media. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information (such as computer-executable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage devices, hard disks, cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or other storage devices, tangible media, or articles of manufacture suitable for storing desired information and accessible by a computer.

[0100] "Computer-readable signal media" refers to signal-bearing media configured to transmit instructions to the hardware of computing device 1400, such as via a network. Signal media typically embodies computer-executable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, data signal, or other transport mechanism. Signal media also includes any information delivery media. By way of example, and not limitation, signal media include wired media such as a wired network or direct connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0101] As previously mentioned, hardware elements 1404 and computer-readable medium 1402 represent instructions, modules, programmable device logic and / or fixed device logic implemented in hardware form, which in some embodiments can be used to implement at least some aspects of the technology described herein. Hardware elements can include other implementations in integrated circuits or systems on a chip, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs) and silicon or components of other hardware devices. In this context, hardware elements can be used as processing equipment for executing program tasks defined by the instructions, modules and / or logic embodied by the hardware elements, and hardware devices for storing instructions for execution, such as the computer-readable storage media previously described.

[0102] The aforementioned combinations may also be used to implement the various techniques and modules described herein. Thus, software, hardware, or program modules and other program modules may be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 1404. Computing device 1400 may be configured to implement specific instructions and / or functions corresponding to software and / or hardware modules. Thus, for example, by using a computer-readable storage medium and / or hardware elements 1404 of a processing system, a module may be implemented as a module executable by computing device 1400 as software, at least in part, in hardware. Instructions and / or functions may be executed / operable by, for example, one or more computing devices 1400 and / or processing system 1401 to implement the techniques, modules, and examples described herein.

[0103] The techniques described herein may be supported by these various configurations of computing device 1400 and are not limited to the specific examples of the techniques described herein.

[0104] It should be understood that, for the sake of clarity, embodiments of the present disclosure have been described with reference to different functional units. However, it will be apparent that, without departing from the present disclosure, the functionality of each functional unit can be implemented in a single unit, in multiple units, or as a part of other functional units. For example, the functionality described as being performed by a single unit can be performed by multiple different units. Therefore, reference to a specific functional unit is only considered as a reference to the appropriate unit for providing the described functionality, rather than indicating a strict logical or physical structure or organization. Therefore, the present disclosure can be implemented in a single unit, or can be physically and functionally distributed between different units and circuits.

[0105] The present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon. When the computer-executable instructions are executed, the above-mentioned process control method is implemented.

[0106] The present disclosure provides a computer program product or computer program, which includes computer-executable instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the computing device to perform the process control methods provided in the various embodiments described above.

[0107] Variations on the disclosed embodiments will be understood and effected by those skilled in the art in practicing the claimed subject matter by studying the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A process control method comprising: Acquire image data of a product to be inspected, wherein the image data to be inspected includes an image to be inspected, and the image to be inspected is an image containing target features; Determining target feature data based on the image data to be inspected, wherein the target feature data includes statistical data and / or classification data corresponding to the target feature; In response to the target characteristic data satisfying a preset warning rule, generating warning data; Obtaining a detection result for the image to be inspected, the detection result including a type of target feature contained in the image to be inspected; Based on the early warning data and the detection result, the process to which the product will be dispatched is determined among multiple optional processes.

2. The method according to claim 1, further comprising: In response to the target feature data not satisfying the preset warning rule, the warning data is not generated, and the product is scheduled to a subsequent production process.

3. The method according to claim 1, wherein The process of determining, based on the early warning data and the detection result, the process to which the product will be dispatched from among multiple optional processes includes: Constructing a state machine, the state machine including a plurality of states and state transition conditions between the plurality of states, the plurality of states including an initial state and at least one terminal state, each terminal state in the at least one terminal state corresponding to one of the plurality of optional processes; Taking the initial state as a starting point, the system transfers between the multiple states based on the early warning data and the detection result until reaching one of the at least one terminal state.

4. The method according to claim 3, wherein: The construction state machine includes: Determining preset warning data that matches the warning data; The state machine is constructed based on a preset state machine model corresponding to the preset warning data.

5. The method according to claim 3, wherein: The state machine is defined by: a state set comprising the plurality of states of the state machine; the initial state; a terminal state set, comprising the at least one terminal state among the plurality of states; an input parameter table, comprising a plurality of input parameters, at least a portion of the plurality of input parameters being associated with the early warning data and the detection result; A state transition function is configured to describe a state transition condition for transitioning from one state in the state set to another state, wherein the state transition condition is set based on at least one input parameter of the plurality of input parameters.

6. The method according to claim 1, wherein The determining target feature data based on the image data to be inspected includes: Inputting the image to be inspected into an early warning model, the early warning model being configured to output a target feature prediction based on the image to be inspected, the target feature prediction including the classification and / or position of the target feature contained in the image to be inspected; Based on the target feature prediction output by the early warning model, statistical data and / or classification data corresponding to the target feature are determined.

7. The method according to claim 1, wherein The image data to be inspected also includes coordinate data of target features in the image to be inspected, and, Wherein, determining target feature data based on the image data to be inspected includes: Based on the coordinate data of the target feature, statistical data and / or classification data corresponding to the target feature is determined.

8. The method according to claim 1, wherein The obtaining of the detection result for the image to be detected includes at least one of the following two items: Presenting the image to be inspected to an inspector, and determining the type of target features contained in the image to be inspected based on input from the inspector; The image to be inspected is input into a detection model, and based on the output of the detection model, the type of target features contained in the image to be inspected is determined, wherein the detection model is configured to output type prediction data of the target features in the image to be inspected based on the image to be inspected.

9. The method according to claim 1, wherein: The obtaining of the image data of the product to be inspected comprises: acquiring image data of the product generated by an optical inspection device; Cleaning the acquired image data to remove or correct abnormal data or incomplete data in the image data; The image data to be inspected is extracted based on the cleaned image data.

10. The method according to claim 9, wherein: The extracting the image data to be inspected based on the cleaned image data includes: Based on the cleaned image data, at least one of the following items is extracted: an image to be inspected containing a target feature, coordinate data of the target feature in the image to be inspected, and attribute data of a product corresponding to the image to be inspected.

11. The method according to any one of claims 1 to 10, wherein The multiple optional processes include at least one of the following: a manual re-inspection process, a product repair process, and a manual verification process.

12. The method according to any one of claims 1 to 10, wherein The early warning data includes early warning data corresponding to at least one of the following: total number warning, size warning, aggregation warning, and repeated occurrence warning.

13. The method according to claim 12, wherein: In response to the target characteristic data satisfying a preset warning rule, generating warning data includes at least one of the following: In response to the total number of target features reaching a total number threshold, generating the total number warning; In response to a number of target features having a target size reaching a number threshold corresponding to the target size, generating a size warning corresponding to the target size; generating the aggregation warning in response to the number of target features appearing within a preset size range reaching a preset threshold; In response to target features of the same classification repeatedly appearing in the same image region of a plurality of images to be inspected, the recurrence warning is generated.

14. A process management device comprising: The first acquisition module is configured to: acquire image data to be inspected of the product, wherein the image data to be inspected includes an image to be inspected, and the image to be inspected is an image containing target features; A first determining module is configured to: determine target feature data based on the image data to be inspected, wherein the target feature data includes statistical data and / or classification data corresponding to the target feature; A generating module is configured to: generate warning data in response to the target feature data satisfying a preset warning rule; A second acquisition module is configured to: acquire a detection result for the image to be inspected, wherein the detection result includes a type of target feature contained in the image to be inspected; The second determination module is configured to: determine, based on the early warning data and the detection result, the process to which the product will be dispatched from a plurality of optional processes.

15. A computing device comprising: a memory configured to store computer-executable instructions; A processor configured to perform the method according to any one of claims 1 to 13 when the computer executable instructions are executed by the processor. 16 . A computer-readable storage medium storing computer-executable instructions, wherein when the computer-executable instructions are executed, the method according to claim 1 is performed.

17. A computer program product comprising computer executable instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

Images