Visual detection method, system and training device

Abstract

The invention provides a visual detection method and system and a training device. The method comprises the following steps: acquiring a training set; training the initial model based on the training set to obtain a visual detection model; controlling a first shooting device to collect a detection image of the to-be-detected object; inputting the detection image into a visual detection model to obtain a flaw detection result of the to-be-detected object; wherein the step of obtaining the training set comprises the substeps of controlling the non-contact optical sensor to scan the surface topography information of the sample; determining size information and position information of flaws on the sample based on the surface topography information; based on the position information, controlling a second shooting device to collect a training image of the sample; and marking the training images based on the size information to obtain a training set. According to the method, efficient, stable and reusable visual detection can be realized.

Description

Technical Field

[0001] This invention relates to the field of visual inspection, and in particular to a visual inspection method, system and training device. Background Technology

[0002] Visual inspection technology, as an industrial inspection method based on computer vision, has played a crucial role in product quality control across various manufacturing industries. This technology acquires image data of product surfaces through image acquisition devices and uses algorithms to analyze and determine the presence of defects, offering advantages such as automation, non-contact operation, and high repeatability. In typical applications, the system relies on pre-collected training samples to build an inspection model, where the accuracy and consistency of defect annotation have a critical impact on model performance. Currently, defect annotation typically relies on manual identification and labeling, and its efficiency and quality are closely related to the experience and focus of the personnel.

[0003] Therefore, there is an urgent need for a visual inspection method, system, and training device to achieve efficient, stable, and reusable visual inspection. Summary of the Invention

[0004] One or more embodiments of the present invention provide a visual inspection method. The method includes: acquiring a training set; training an initial model based on the training set to obtain a visual inspection model; controlling a first imaging device to acquire inspection images of an object to be inspected; inputting the inspection images into the visual inspection model to obtain defect detection results for the object to be inspected; wherein, acquiring the training set includes: controlling a non-contact optical sensor to scan the surface morphology information of a sample; determining the size and location information of defects on the sample based on the surface morphology information; controlling a second imaging device to acquire training images of the sample based on the location information; and annotating the training images based on the size information to obtain the training set.

[0005] One aspect of the present invention provides a visual inspection system. The system includes a first acquisition module, a training module, a second acquisition module, and a detection module. The first acquisition module is configured to acquire a training set. The training module is configured to train an initial model based on the training set to obtain a visual inspection model. The second acquisition module is configured to control a first imaging device to acquire inspection images of an object to be inspected. The detection module is configured to input the inspection images into the visual inspection model to obtain defect detection results for the object to be inspected. The first acquisition module is further configured to: control a non-contact optical sensor to scan the surface morphology information of a sample, wherein the surface morphology information is three-dimensional data; determine the size and position information of the sample based on the surface morphology information; control the second imaging device to capture training images of the sample based on the position information; and annotate the training images based on the size information to obtain the training set.

[0006] One or more embodiments of the present invention provide a visual detection training device for training the aforementioned visual detection model, comprising: a non-contact optical sensor for scanning surface morphology information of a sample; a second imaging device, the second imaging device including a second camera assembly and a second light assembly, for acquiring training images of the sample; a multi-degree-of-freedom mobile platform for mounting the second camera assembly; and a controller for controlling at least one of the non-contact optical sensor, the second imaging device, and the multi-degree-of-freedom mobile platform, and receiving and processing data from the non-contact optical sensor, the second imaging device, and / or the multi-degree-of-freedom mobile platform.

[0007] The embodiments of the present invention have at least the following beneficial effects: the training set obtained by the visual detection model is not an ordinary training set, but a training set with "absolute truth values". During model training, the annotation of each defect is not estimated manually, but is a precise value calculated by a sophisticated second imaging device after capturing images, effectively solving the pain points of inaccuracy and inconsistency in traditional manual annotation. Attached Figure Description

[0008] The present invention will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, the same reference numerals denote the same structures, wherein: Figure 1 This is a schematic diagram illustrating an application scenario of a visual inspection system according to some embodiments of the present invention; Figure 2 This is an exemplary flowchart of a visual inspection method according to some embodiments of the present invention; Figure 3 This is an exemplary flowchart of obtaining a training set according to some embodiments of the present invention; Figure 4 This is an exemplary block diagram of a visual inspection system according to some embodiments of the present invention; Figure 5 This is an exemplary block diagram of a visual inspection training apparatus according to some embodiments of the present invention. Detailed Implementation

[0009] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of the present invention. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0010] It should be understood that the terms “system,” “device,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0011] As indicated in this invention and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0012] This invention uses flowcharts to illustrate the operations performed by the system according to embodiments of the invention. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0013] Some embodiments of the present invention provide a visual inspection method, which can be executed by a visual inspection system, including model training and defect detection. Model training includes acquiring a training set, training an initial model based on the training set, and obtaining a visual inspection model. Defect detection includes controlling a first imaging device to acquire a detection image of the object to be inspected, and inputting the detection image into the visual inspection model to obtain the defect detection result of the object to be inspected.

[0014] In some embodiments, model training and defect detection can be performed by different devices. For example, model training can be performed by a vision inspection training device, and defect detection can be performed by a detection device with a pre-trained vision inspection model built in, used on-site.

[0015] The following will provide a detailed explanation of model training, defect detection, and related equipment.

[0016] Figure 1 This is a schematic diagram illustrating an application scenario of a visual inspection system according to some embodiments of the present invention. For example... Figure 1 The application scenario 100 of the visual inspection system shown may include a processor 110, an imaging device 120, a network 130, and a storage device 140.

[0017] Processor 110 can process data and / or information obtained from a visual inspection training device or a component thereof, an inspection device or a component thereof, or a component of a visual inspection system. Based on this data, information, and / or processing results, the processor can execute program instructions to perform one or more functions described in this invention.

[0018] In some embodiments, processor 110 may include one or more sub-processing devices (e.g., a single-core processing device or a multi-core multi-chip processing device). By way of example only, processor 110 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), an application-specific instruction processor (ASIP), a graphics processing unit (GPU), a physical processor (PPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), a microprocessor, or any combination thereof.

[0019] In some embodiments, the processor 110 may include a controller for a visual inspection training device, a processing unit in an inspection device, etc.

[0020] In some embodiments, the processor 110 may control the components of the visual inspection training device to complete model training, and / or control the components of the inspection device to complete defect detection.

[0021] The shooting device 120 is used to acquire images of the subject.

[0022] In some embodiments, the subject of the photograph may include a sample and / or an object to be tested.

[0023] In some embodiments, the shooting device may include a light assembly 120-1 and a camera assembly 120-2.

[0024] In some embodiments, the imaging device 120 may include a first imaging device for capturing detection images of the object to be detected, and a second imaging device for capturing training images of the sample.

[0025] In some embodiments, the imaging device 120 may communicate directly or via the network 130 with the processor 110 to upload the acquired images to the processor or receive instructions from the processor.

[0026] For more information on the shooting equipment, please refer to [link / reference]. Figures 2-3 The relevant description in the document.

[0027] Network 130 can connect the various components of the system and / or connect the system to external resources. Network 130 enables communication between the components and with other parts outside the system, facilitating the exchange of data and / or information.

[0028] In some embodiments, network 150 can be any one or more of wired or wireless networks. For example, network 150 may include cable networks, fiber optic networks, telecommunications networks, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, ZigBee networks, near field communication (NFC), device internal buses, device internal wiring, cable connections, etc., or any combination thereof. Network connections between the various parts can be achieved using one or more of the above methods.

[0029] Storage device 140 can be used to store data and / or instructions generated during the operation of the vision inspection system. Storage device 140 may include one or more storage components, each of which may be a separate device or part of another device. In some embodiments, storage device 140 may include random access memory (RAM), read-only memory (ROM), mass storage, removable memory, volatile read-write memory, etc., or any combination thereof. Exemplarily, mass storage may include disks, optical disks, solid-state drives, etc. In some embodiments, storage device 140 may be implemented on a cloud platform. By way of example only, the cloud platform may include private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, internal cloud, multi-tiered cloud, etc., or any combination thereof.

[0030] In some embodiments, the storage device can communicate with the processor 110 and the imaging device 120 via the network 130 to obtain data generated by the processor 110 and / or the imaging device 120, or to send data required by the processor 110.

[0031] In some embodiments, processor 110 can process data and / or information obtained from a visual inspection training device or a component thereof, an inspection device or a component thereof, or a component of a visual inspection system. The processor can execute program instructions based on this data, information, and / or processing results to perform one or more functions described in this invention. For example, processor 110 (such as a controller) can control a second imaging device in imaging device 120 (e.g., a second camera assembly and a second lighting assembly mounted on a multi-degree-of-freedom mobile platform) via network 130 to capture images of samples to acquire training images. Simultaneously, the processor can control a non-contact optical sensor to scan the surface morphology information of the same sample. The acquired raw image data and three-dimensional morphology data are transmitted to processor 110 via network 130 for processing. The processor can automatically determine the size and location information of defects based on the three-dimensional morphology data and accurately annotate the training images accordingly to generate a high-quality training set. The generated training set can be stored in storage device 140 via network 130. The processor can call the training set in storage device 140 to execute a model training algorithm, train, and obtain a visual inspection model. The finally trained model file can be stored in storage device 140 via network 130 for subsequent detection.

[0032] In some embodiments, the processor can load a pre-trained visual inspection model from storage device 140 via network 130. When an object to be inspected enters the workstation, the processor controls a first imaging device (e.g., a first camera assembly and a first lighting assembly fixed on the production line) in imaging device 120 via network 130 to acquire an inspection image of the object. The acquired inspection image data is sent to processor 110 via network 130. The processor can input the inspection image into the loaded visual inspection model for inference analysis to generate a defect detection result. The defect detection result can be uploaded to storage device 140 via network 130 for recording, and can also be used to trigger subsequent sorting, alarm, and other operations.

[0033] Figure 2 This is an exemplary flowchart of a visual inspection method according to some embodiments of the present invention. Figure 2 As shown, process 200 includes the following steps. In some embodiments, process 200 may be executed by a processor.

[0034] Step 210: Obtain the training set.

[0035] The training set refers to the set of sample data used for model training.

[0036] In some embodiments, the training set may include a large number of labeled training images, and the labels may include defect information in the training images.

[0037] In some embodiments, the processor can capture training images corresponding to the samples (collecting samples) and annotate the training images to obtain a training set. For more detailed information, please refer to [link to relevant documentation]. Figure 3 The relevant description in the document.

[0038] Step 220: Train the initial model based on the training set to obtain the visual detection model.

[0039] An initial model refers to a model with predetermined parameters that has not been learned or has undergone some pre-training before formal training. For example, a visual detection model with predetermined parameters that has not been learned.

[0040] In some embodiments, the processor can input a large number of labeled training images from the training set into an initial model, construct a loss function based on the output of the initial model and the defect information in the labels, and iteratively update the parameters in the initial model using gradient descent or other feasible training methods based on the loss function. When the training termination condition is met, the training ends, and a trained visual detection model is obtained. The training termination condition may include at least one of the following: the loss function converges, or the number of iterations reaches a threshold.

[0041] In some embodiments, the visual detection model is a model obtained by training an initial training model based on a training set.

[0042] In some embodiments, the visual detection model can be a machine learning model, such as the YOLO (You Only LookOnce) model, the Faster R-CNN (Faster Region-based Convolutional Neural Network) model, the SegNet (Segmentation Network) model, or other models capable of detecting defects through image analysis.

[0043] Visual inspection models can be used to detect defects in images. The input of a visual inspection model can include the image of the object to be inspected, and the output can include the defect detection results of the object.

[0044] The object to be tested refers to the subject of the test, whose specific characteristics (such as the presence or absence of defects) need to be determined through testing. The object to be tested includes various industrial products; this invention does not limit the specific product type.

[0045] Product type refers to the different classifications of products based on the surface characteristics of the object being tested. For example, based on the material and / or roughness of the surface, products can be classified as smooth and reflective, rough and matte, transparent / semi-transparent, or other complex surfaces. Another example is classification based on the surface color of the object. Product type can also be distinguished according to other feasible criteria, which can be determined according to actual needs.

[0046] An image to be detected refers to the image data of the object to be detected, presented in the form of an image.

[0047] Defect detection results refer to the conclusions regarding the defect status of the object under inspection. In some embodiments, defect detection results may include at least one of the following: presence of a defect, defect location, defect size, defect type, and result confidence level.

[0048] A defect refers to a flaw or non-compliance with quality standards on the surface of an object under inspection. Defects may occur during manufacturing or use and affect the product's function, safety, or appearance. For example, in mobile phone screen inspection, defects may be scratches, cracks, or bright / dark spots; in metal casing inspection, defects may be dents, bumps, or stains.

[0049] The defect location refers to the coordinate information of the defect in space. The defect location can be the absolute position based on the sample's own coordinate system, or it can be relative to the entire object to be inspected or its relative position in the image.

[0050] Defect size refers to a quantitative description of the area occupied by a defect in three-dimensional space. Defect size includes information about the physical size of the defect.

[0051] Defect types refer to category labels that classify defects based on their shape, cause, or physical characteristics. Examples include scratches, dents, bumps, stains, and cracks.

[0052] Result confidence refers to the degree of certainty a visual inspection model has about its output defect detection results, and can be represented by a value between 0 and 1. The higher the value, the higher the accuracy of the model's output.

[0053] Step 230: Control the first imaging device to acquire the detection image of the object to be detected.

[0054] An imaging device refers to a hardware unit used to acquire image data of an object being photographed. In some embodiments, the object being photographed may include at least one of an object to be detected and a sample. A first imaging device refers to an imaging device used when photographing an object to be detected.

[0055] In some embodiments, the shooting device may include a lighting assembly and a camera assembly.

[0056] Lighting components are components that provide illumination for shooting. They can create specific light paths to highlight key features (such as blemishes) on the surface of the subject.

[0057] A camera assembly refers to a component used to capture images. In some embodiments, a camera assembly may include at least one camera, such as a natural light camera or a hyperspectral camera. The natural light camera is used to capture RGB three-channel images visible to the human eye, while the hyperspectral camera is used to acquire spectral information for each pixel to analyze material composition and invisible features.

[0058] For further instructions on the object to be detected and the image to be detected, please refer to the relevant content in step 220.

[0059] In some embodiments, the first imaging device may include a first lighting assembly and a first camera assembly.

[0060] In some embodiments, the processor may acquire a lighting scheme library and acquire a detection image of the object to be detected based on at least one first lighting scheme in the lighting scheme library. The lighting scheme library refers to a mapping rule that stores the correspondence between product types and lighting schemes.

[0061] A lighting scheme library refers to a mapping rule that stores the correspondence between product types and lighting schemes.

[0062] For further explanation on acquiring detection images based on the illumination scheme library, please see the description below.

[0063] Step 240: Input the detected image into the visual detection model to obtain the defect detection result of the object to be detected.

[0064] In some embodiments, the processor can input the detection image into a visual detection model and obtain the output of the visual detection model. The detection image can be a first image and a second image directly captured by a first imaging device, or it can be a first feature image obtained by fusing features from the first image and the second image.

[0065] In some embodiments, the processor can also input the detection vector obtained by vector concatenation into the visual detection model and obtain the output of the visual detection model.

[0066] In some embodiments, the processor can obtain the defect detection result of the object to be inspected based on the output of the visual inspection model. For more details on the defect detection result, please refer to the relevant content of step 220.

[0067] In some embodiments of the present invention, a pre-trained visual inspection model is used to detect defects in the object to be inspected, which reduces the degree of human intervention, improves inspection efficiency, reduces the probability of missed or false detections, and thus better ensures the accuracy of defect detection.

[0068] Additionally, it should be noted that steps 210-220 are model training steps, and steps 230-240 are defect detection steps. The model training and defect detection steps can be performed using different devices. For example, the model training step may use a visual inspection training device equipped with a non-contact optical sensor to scan 3D surface topography information, while the defect detection can use actual on-site inspection equipment that does not require a non-contact optical sensor (see below). Figure 5 To scan three-dimensional surface topography information, only two-dimensional images need to be captured.

[0069] Figure 3 This is an exemplary flowchart illustrating the acquisition of a training set according to some embodiments of the present invention. Figure 3 As shown, the process of acquiring the training set can be performed by the visual detection training device 500, the controller 540, or the processor.

[0070] Step 211: Control the non-contact optical sensor to scan the surface morphology information of the sample.

[0071] A sample (collected sample) is a sample object used to collect model training data.

[0072] In some embodiments, the sample may include historical test items that have undergone defect detection and whose detection results are accurate.

[0073] Surface morphology information refers to comprehensive data that characterizes the features of a sample surface in three-dimensional space and is related to its geometric structure.

[0074] In some embodiments, surface topography information includes three-dimensional point cloud data of the sample. For example, the surface topography information of the sample may include the three-dimensional coordinates (X, Y, Z) of multiple points on the sample surface.

[0075] Surface morphology information can reflect the physical characteristics of the sample surface, such as height, depth, tilt, texture, and micro-contour. This information can be used to more accurately assess defects such as the depth of dents, protrusions, warping, deformation, burrs, and scratches.

[0076] In some embodiments, the processor can control a non-contact optical sensor to scan the surface morphology information of a sample. For example, multiple samples can be sequentially fixed on a rotating stage within the scanning range of the non-contact optical sensor, or multiple samples can be sequentially transported to the scanning range of the non-contact optical sensor via a conveyor belt; the non-contact optical sensor scans the samples within the scanning range to obtain a set of three-dimensional coordinates of multiple points on the sample surface.

[0077] In some embodiments, the processor can preprocess the set of three-dimensional coordinates of the sample to obtain surface morphology information of the sample.

[0078] Preprocessing can include one or more of the following: denoising, registration and stitching, and meshing. Denoising refers to filtering out noise caused by environmental interference or surface reflections. Registration and stitching involves merging multiple 3D coordinate data acquired from multiple angles into a complete sample point cloud model. Meshing involves converting the 3D coordinate set into a triangular mesh model for easier computation and visualization.

[0079] For detailed information on non-contact optical sensors, please refer to [link / reference]. Figure 5 The relevant description in the document.

[0080] Step 212: Based on the surface morphology information, determine the size and location information of the defects on the sample.

[0081] The size information of a defect refers to the quantitative representation of the set of parameters of the spatial extent occupied by the defect.

[0082] In some embodiments, the dimensional information of the defect includes at least one of the following: length, width, height, depth, surface area, and volume.

[0083] The location information of a defect refers to its spatial coordinates. These spatial coordinates can be coordinates in the same coordinate system as the surface morphology information of the sample, or coordinates in the sample coordinate system. The sample coordinate system refers to a coordinate system established with the center point of the sample as the origin.

[0084] In some embodiments, the location information of the defect may include the three-dimensional coordinates (x, y, z) of the defect center point.

[0085] In some embodiments, the processor can align and register the surface topography information of the scanned sample with the corresponding reference model. Based on the alignment and registration result, it calculates the distance deviation from each point in the surface topography information to the corresponding point in the reference model, and identifies points with a distance deviation greater than a threshold as potential defects. The reference model refers to an ideal, flawless, or design-compliant digital 3D surface model (e.g., a CAD model), which can be preset. The distance deviation can be preset based on prior experience and / or actual needs.

[0086] In this embodiment, the quantitative standardization of the feature (such as defect, flaw) classification process is achieved through automatic classification based on measurement data thresholds. Standardization can reduce subjective errors, improve classification consistency, and solve the problem of lack of quantitative support in traditional automated detection (such as the iMerit model).

[0087] In some embodiments, the processor can separate continuous potential defect points into independent defect clouds by using region growing or clustering algorithms (such as DBSCAN, Euclidean clustering, etc.), and determine the defect point cloud data corresponding to the defect cloud; and perform quantitative measurement on the defect cloud based on the defect point cloud data.

[0088] In some embodiments, quantitative measurement of defect clouds may include calculating or estimating at least one of the size, surface area, volume, and location of the defect cloud. For example, the processor may calculate the length, width, and depth (or height) of each defect cloud in three-dimensional space based on the coordinates of each point in the defect point cloud data, such as by obtaining its approximate size and orientation through principal component analysis (PCA) of the point cloud; estimate the surface area of ​​the defect based on the length, width, and depth (or height) data; estimate the actual or redundant volume of the defect cloud by comparing it with a baseline model; and calculate the center coordinates (x, y, z) of the defect in three-dimensional space based on the coordinates of each point in the defect point cloud data.

[0089] In some embodiments, the processor can output a structured list of defects based on the result of quantitative measurement of the defect cloud. The list of defects includes at least one defect entry corresponding to at least one defect on the sample surface. Each defect entry includes information describing the defect characteristics such as defect ID, 3D coordinates of the center point (X, Y, Z), size (length, width, height), type (pit, bump, scratch, etc.), and severity.

[0090] Step 213: Based on the location information, control the second imaging device to acquire training images of the samples.

[0091] Training images refer to the images of samples used as training samples for a model.

[0092] The second imaging device refers to the imaging equipment used when photographing the sample. For detailed information on imaging equipment, please refer to [link / reference needed]. Figure 2 The relevant description in step 230.

[0093] In some embodiments, the number of training images collected for a single sample can be multiple.

[0094] In some embodiments, the second shooting device may include a second camera assembly and a second lighting assembly.

[0095] In some embodiments, the processor can determine multiple candidate shooting strategies based on multiple second lighting schemes corresponding to the second light component and multiple shooting angles corresponding to the second camera component, and control the second shooting device to acquire multiple training images based on the multiple candidate shooting strategies.

[0096] The second lighting scheme refers to the lighting scheme used when photographing samples.

[0097] In some embodiments, the second lighting scheme includes the light type and / or illumination angle of at least one light in the second lighting assembly (e.g., ring light, coaxial light, strip light, etc.), and may also include other lighting-related parameters, such as the switching on and off of the lamp, brightness, on-time, triggering sequence of multiple different lights, etc.

[0098] The shooting angle refers to the relative direction between the camera's optical axis and the normal to the sample surface.

[0099] A candidate shooting strategy refers to a determined combination of shooting parameters to be selected. In some embodiments, a candidate shooting strategy includes the shooting angle and corresponding second lighting scheme for controlling the second camera component to acquire images.

[0100] In some embodiments, the processor can combine at least one lighting scheme and multiple shooting angles to obtain multiple candidate shooting strategies.

[0101] In some embodiments, the processor may sequentially control the second imaging device to capture images based on each of a plurality of candidate imaging strategies in order to obtain a plurality of training images of the sample.

[0102] In some embodiments, the processor can pre-obtain the coordinates of each component in the visual inspection training device (e.g., a non-contact optical sensor, a second imaging device, a multi-degree-of-freedom mobile platform, etc.) in the same world coordinate system, and determine the transformation matrix between the second camera component and the non-contact optical sensor in the second imaging device through hand-eye calibration. When the coordinates of a point in the coordinate system of the non-contact optical sensor are known, the coordinates of that point in the camera coordinate system can be calculated using the transformation matrix.

[0103] In some embodiments, the processor can plan the movement path of the multi-degree-of-freedom mobile platform based on the location information corresponding to the defect, and efficiently control the multi-degree-of-freedom mobile platform equipped with the second camera component to move, so that it carries the second camera component to traverse each defect point; for each defect, the processor can control the multi-degree-of-freedom mobile platform to adjust the shooting angle of the second camera component based on the shooting angle included in the candidate shooting strategy; when executing each candidate shooting strategy, the processor controls the second lighting component to switch to the second lighting scheme corresponding to the candidate shooting strategy being executed; the processor triggers the second camera component to take a picture, obtains the training image of the defect under each candidate shooting strategy, and associates the training image of the defect with the defect ID, for example, by associating the corresponding defect ID in the image file name or metadata, thereby obtaining multiple training images of the sample.

[0104] In some embodiments, the processor may also determine the defect type based on size information and location information, select a target shooting strategy from multiple candidate shooting strategies based on at least one of defect type, location information and shooting angle, and acquire multiple training images based on the target shooting strategy.

[0105] In some embodiments, for each defect, the processor can construct a defect feature vector based on the location and size information corresponding to the defect, and automatically determine the defect type based on the defect feature vector using a pre-trained machine learning classifier (such as SVM, decision tree, etc.) or a pre-set rule matching algorithm.

[0106] In some embodiments, the processor can determine the optimal shooting angle for shooting the defect based on the defect type. For example, for surface depressions, side lighting and shallow angle shooting are needed to highlight the shadows, while for protrusions, direct lighting or light at different angles are needed to highlight them. Based on the optimal shooting angle for shooting the defect, at least one candidate shooting strategy including the optimal shooting angle is invoked from the candidate shooting strategies to determine at least one target shooting strategy.

[0107] In some embodiments, the processor can plan the movement path of the multi-degree-of-freedom mobile platform to efficiently traverse the shooting angles included in each target shooting strategy; when executing each target shooting strategy, control the second lighting component to switch to the second lighting scheme corresponding to the target shooting strategy being executed; trigger the second camera component to take a picture, obtain the training image of the defect under one or more of the aforementioned target shooting strategies, and associate the training image of the defect with the defect ID, for example, by associating the corresponding defect ID in the image file name or metadata.

[0108] In some embodiments, the processor can sequentially determine at least one target shooting strategy for each defect in the sample, and complete the shooting according to each target shooting strategy to obtain the training image of the sample.

[0109] In some embodiments of the present invention, multiple candidate shooting strategies are used to capture samples, thereby obtaining multiple different training images and improving the richness of the training set. The candidate shooting strategies are further selected based on the defect type to obtain the target shooting strategy. Shooting samples based on the target shooting strategy can ensure the richness of the training set while avoiding data redundancy, improving model training efficiency, and reducing the risk of overfitting.

[0110] In some embodiments, the second imaging device includes a second natural light camera and a second hyperspectral camera.

[0111] When acquiring training images of the samples, the processor can control the second natural light camera to capture the third image of the samples and control the second hyperspectral camera to capture the fourth image of the samples; the features of the third image and the features of the fourth image are fused to obtain a fused second feature image; the second feature image is used as a training image.

[0112] In some embodiments, the exposure time difference between the second natural light camera and the second hyperspectral camera meets a preset requirement. For a detailed explanation of the preset requirement, please refer to the relevant descriptions of the first natural light camera and the first hyperspectral camera below.

[0113] In some embodiments, the processor may process the third image and the fourth image to obtain features of the third image and features of the fourth image, fuse the features of the third image and features of the fourth image to obtain a fused second feature image, and determine the second feature image as a training image.

[0114] The features of the third image can be represented by the third feature map. The third feature map includes spatial information about the sample's surface texture, edge features, shape, and defect locations.

[0115] The features of the fourth image can be represented by the fourth feature map. The fourth feature map includes spectral information about the sample's material, chemical composition, molecular structure, etc.

[0116] In some embodiments, the third feature map and the fourth feature map are aligned in spatial dimensions.

[0117] The process of obtaining the fused second feature image is similar to that of obtaining the fused first feature image; see the relevant description of the first feature image below.

[0118] Step 214: Label the training images based on size information to obtain the training set.

[0119] In some embodiments, the processor can label the training images based on size information in various ways to obtain labeled training images, thereby obtaining a training set. For example, the processor can automatically label the training images based on size information.

[0120] In some embodiments, the processor can project defect point cloud data from the 3D point cloud data onto the training image based on the shooting parameters of the second camera component to form defect scatter clusters; fit the contours of the defect scatter clusters with polygons to obtain defect contours; and annotate defect information on the training image based on the defect contours.

[0121] Shooting parameters refer to parameters that describe the characteristics of the camera when capturing training images. In some embodiments, shooting parameters may include camera intrinsic parameters and camera extrinsic parameters.

[0122] Camera intrinsic parameters are parameters that describe the characteristics of a camera itself, such as focal length, principal point, and distortion coefficient. In some embodiments, camera intrinsic parameters can be obtained by the processor directly calling camera parameters.

[0123] Camera extrinsic parameters refer to parameters that describe the position and orientation of the camera in three-dimensional space. In some embodiments, the camera extrinsic parameters are related to the motion parameters and pose of the multi-degree-of-freedom mobile platform, and can be obtained by a processor through sensors deployed on the multi-degree-of-freedom mobile platform.

[0124] In some embodiments, the processor can obtain camera intrinsic and extrinsic parameters corresponding to the training image, and based on the camera intrinsic and extrinsic parameters, project the defective point cloud data in the 3D point cloud data onto the training image to obtain multiple two-dimensional coordinates of the defective point cloud data in the training image, and determine the defective scattered point clusters in the training image based on the multiple two-dimensional coordinates of the defective point cloud data in the training image.

[0125] A defect profile is a closed line that reflects the boundary of a cluster of defect points and can be used to describe the shape and extent of a defect.

[0126] In some embodiments, the processor can fit the outline of the defect scatter cluster with a polygon to obtain the defect outline.

[0127] In some embodiments, the processor can also generate a bounding box corresponding to the defect cloud and obtain the three-dimensional spatial coordinates of the eight vertices of the bounding box; project the three-dimensional spatial coordinates of the eight vertices of the bounding box onto the training image to generate a two-dimensional minimum bounding rectangle corresponding to the defect cloud, and use the two-dimensional minimum bounding rectangle as the defect contour. Here, the bounding box refers to the smallest cuboid that tightly encloses the spatial shape of the defect and whose sides are parallel to the axes of the three-dimensional coordinate system.

[0128] In some embodiments, the processor can automatically acquire the contour information of the defect contour.

[0129] Contour information is data that reflects the characteristics of a defect's contour. In some embodiments, contour information can be represented in multiple ways. For example, when the defect contour is a regular rectangle, it can be represented by the rectangle's length and width, as well as any one of its vertices (such as the vertex with the smallest coordinate value), for example, Bounding Box(x_min, y_min, width, height); when the defect contour is an irregular polygon, it can be represented by the coordinates of at least one point on the polygon contour, for example, Polygon(x1, y1, x2, y2, ..., xn, yn).

[0130] In some embodiments, the processor can search the defect list based on the defect ID corresponding to the defect cloud corresponding to the defect contour, retrieve relevant information about the defect corresponding to the defect cloud (such as size information, coordinate information, defect type, etc.), and determine the contour information based on the size information and / or coordinate information in the relevant information. For a detailed explanation of the defect list, please refer to [link to relevant documentation]. Figure 2 The relevant explanations are in the text.

[0131] In some embodiments, the processor can generate at least one defect information based on contour information, defect type, and defect ID, and annotate the defect information on the training image. For example, the contour information, defect type, and defect ID corresponding to the defect cloud are written into a standard annotation file format (such as COCO JSON, Pascal VOC XML, YOLO txt, etc.) to determine a defect information for annotation, and the defect information is annotated to the corresponding training image. When all defects in a training image are annotated, the annotation of the training image is completed.

[0132] In some embodiments, the defect information may also include the size information of the defect, such as length, width, height, or depth.

[0133] Once the annotation of at least one training image is completed, the training set can be determined based on the training images with the annotation of defective information.

[0134] In this embodiment, the image annotation process is automated. The software automatically adds labels based on scanned data, eliminating the bottleneck of manual annotation and improving efficiency by 50%, making it suitable for mass production. Simultaneously, the training set expansion process is automated by generating images from point clouds to automatically expand the dataset and improve model robustness. For example, variant training images (such as those with rotation, scaling, and noise addition) can be generated from scanned 3D point clouds to expand the training set.

[0135] In some embodiments of the present invention, during the model training stage, a non-contact optical sensor is used to sense the three-dimensional surface topography information of the sample, and then a two-dimensional image is captured. The two-dimensional image is labeled based on the three-dimensional surface topography features to obtain a training set. This can obtain a training set with "absolute truth values". The labels in the training set are not estimated manually, but determined by precision instrument measurement. This solves the pain point of inaccurate manual labeling and inconsistent results due to strong subjectivity. It provides a good data foundation for training a high-precision visual inspection model. Using this training set is beneficial to obtaining a more accurate visual inspection model, thereby obtaining more accurate defect detection results.

[0136] In some embodiments, the lighting scheme library may include at least one first lighting scheme.

[0137] The first lighting scheme refers to the lighting scheme used when photographing the object to be inspected. In some embodiments, a lighting scheme may include the light type and / or illumination angle of at least one light in the first lighting assembly, and may also include other lighting-related parameters, such as the switching on and off of the light, brightness, on-time, triggering sequence of multiple different lights, etc.

[0138] In some embodiments, the processor may pre-configure one or more first lighting schemes, each of which may have a unique scheme ID and description.

[0139] For example, the processor can pre-set multiple first lighting schemes, and when applying different first lighting schemes, control the light type, illumination angle, on / off state, brightness, on-time, triggering sequence, etc. of at least one light in the first lighting component. For example, the lighting scheme library may include five first lighting schemes, such as: Scheme 1: high-angle ring light, brightness 100%; Scheme 2: low-angle strip light (side light), brightness 80%; Scheme 3: coaxial light; Scheme 4: backlight; Scheme 5: multi-light source combination (such as ring light + side light).

[0140] In some embodiments, the processor can obtain the product ID of the object to be inspected for different product types. For example, the product type can be obtained through barcode / QR code scanners, RFID readers, visual recognition, product information provided by a host computer MES system, etc., and the corresponding product ID can be determined.

[0141] The product ID is a unique identifier that reflects the type of product being tested.

[0142] In some embodiments, each product type has its corresponding lighting scheme library, and each lighting scheme library includes multiple first lighting schemes corresponding to that product type. That is, a product type has a corresponding product ID, and the product ID has a corresponding lighting scheme library. Through the correspondence between product type and product ID, the corresponding lighting scheme library can be determined, and the processor can call the first lighting scheme in the corresponding lighting scheme library.

[0143] In some embodiments, the first lighting scheme corresponding to a product type can exist in multiple ways. For example, in the initial stage, the first lighting scheme in the lighting scheme library corresponding to a certain product type can be a null value or a preset scheme. As defect detection is continuously executed, the processor can update the first lighting scheme corresponding to that product type based on the defect detection results to obtain better lighting effects. For example, during the defect detection process, if the lighting scheme that obtains a better defect detection result (e.g., the best effect in a batch of detections) is not in the lighting scheme library, the best lighting scheme can be added to the lighting scheme library, or one or more lighting schemes in the lighting scheme library can be replaced.

[0144] In some embodiments, the processor can control the light source controller to sequentially apply multiple first lighting schemes from the lighting scheme library to capture images of the object to be detected, thereby obtaining a detection image of the object. For example, when the object to be detected is within the shooting range of the first imaging device, the processor can obtain the product ID corresponding to the product type of the object to be detected, and query the lighting scheme library based on the product ID. If the product ID is found, one or more first lighting schemes from the lighting scheme library are invoked; if the product ID is not found, the default lighting scheme is used to illuminate the object, and a detection image of the object to be detected is obtained.

[0145] In some embodiments, the defect detection result of the object to be tested includes either normal output or abnormal output.

[0146] Normal output means that the defect detection results can be output clearly and accurately.

[0147] Abnormal output refers to at least one of the following situations: missed detection, false detection, confidence level within a fuzzy range, or increased false alarm rate. Among these, missed detection and false detection can be detected through manual review, and the processor can determine the occurrence of missed detection and / or false detection by acquiring manually input data; confidence level within a fuzzy range can be detected based on the processor's own statistical results, such as when, for a certain product, the system output confidence level is continuously within a "fuzzy range" (e.g., confidence level between 0.4 and 0.6), or when the false alarm rate suddenly increases.

[0148] In some embodiments, in response to an abnormal defect detection result of the object under test, a test alert can be issued to perform a lighting scheme test on the object under test. For example, in response to receiving a manually input signal to trigger the test, or detecting that the confidence level is in an ambiguous range or the false alarm rate suddenly increases, the processor can trigger a lighting scheme test and issue a test alert.

[0149] In some embodiments, the processor can acquire test data, determine the optimal lighting scheme based on the test data, and update the lighting scheme library corresponding to the product type of the object under test based on the optimal lighting scheme.

[0150] The test data stores data containing at least one preset lighting scheme, which is used to photograph a specific product type. That is, different test data can be set for different product types. In some embodiments, the test data can be preset.

[0151] In some embodiments, the processor can instruct assembly line workers or robots to move items with abnormal defect detection results to a debugging station and determine their product type. Based on the product type, the processor retrieves corresponding test data, automatically calling each preset lighting scheme in the test data sequentially, completing image capture, and analyzing the captured images. For example, for each lighting scheme, the processor can control the first lighting component to switch to that scheme, taking multiple photos of the item (e.g., two or more) using the same camera parameters, averaging the features acquired from the multiple photos, and inputting the average value into a visual inspection model for analysis to determine the test results. The test results record the preset lighting scheme ID, the test analysis results, and their confidence levels. The test analysis results include the results of defect analysis based on the images acquired during the testing phase.

[0152] In some embodiments, the processor can determine the optimal lighting scheme based on the test results corresponding to each preset lighting scheme. For example, for an object to be tested that is known to have defects but is missed in detection, the goal is to find the scheme that can highlight the defects to the greatest extent, and the preset lighting scheme with the highest confidence in the defects can be selected as the optimal lighting scheme. As another example, for an object to be tested that is known to be without defects but is falsely detected as having defects, the goal is to find the scheme that is least likely to generate erroneous signals, and the preset lighting scheme that makes the background most uniform and has the lowest confidence in the test analysis results for defects (or the highest confidence in the absence of defects) can be selected as the optimal lighting scheme.

[0153] In some embodiments, the processor can automatically update the correspondence between the product ID of the object to be detected with the determined best lighting scheme and replace the original first lighting scheme corresponding to the product ID in the lighting scheme library, and record the optimization log (such as time, product ID, old scheme, new scheme, optimization reason, etc.).

[0154] In some embodiments, the processor can periodically update the lighting scheme library. For example, after a preset time period or after testing a preset number of objects to be tested, a lighting scheme test is automatically triggered. The processor can call the test data corresponding to different product types, execute optimization processes, and verify or update the first lighting scheme in the lighting database to adapt to situations such as light source aging and minor changes in product materials.

[0155] In some embodiments of the present invention, for specific types of products that are falsely detected or missed, the processor can automatically try different preset lighting schemes, determine the best lighting scheme and update the lighting scheme library. Through adaptive optimization, the defect detection accuracy of specific products can be significantly improved, the false detection and missed detection rates can be reduced, and the lighting strategy can be automated and intelligently managed, reducing the reliance on human experience.

[0156] In some embodiments, the first imaging device may include a first natural light camera and a first hyperspectral camera.

[0157] When acquiring detection images of the object to be detected, the processor can control the first natural light camera to capture a first image of the object to be detected, control the first hyperspectral camera to capture a second image of the object to be detected, fuse the features of the first image and the features of the second image to obtain a fused first feature image, and use the first feature image as the detection image.

[0158] In some embodiments, the exposure time difference between the first natural light camera and the first hyperspectral camera meets a preset requirement. For example, the preset requirement may be that the exposure time difference is zero, or that the exposure time difference is less than a preset difference value, such as 1ms, 3ms, 5ms, etc.

[0159] In some embodiments, the processor can process the first image and the second image using a pre-trained image processing model, and fuse the features of the first image and the features of the second image to obtain a fused first feature image.

[0160] In some embodiments, the image processing model may include at least one feature determination layer, a weight determination layer, and a fusion layer.

[0161] The feature determination layer is used to determine the features of an image. The processor can input a first image into the feature determination layer, and the feature determination layer outputs the features of the first image; and / or input a second image into the feature determination layer, and the feature determination layer outputs the features of the second image. In some embodiments, the feature determination layer can be a Convolutional Neural Network (CNN) model.

[0162] In some embodiments, two different feature determination layers can be set to process the first image and the second image respectively, or a single feature determination layer can be set to process the first image and the second image.

[0163] The features of the first image can be represented by the first feature map, which includes spatial information about the surface texture, edge features, shape, and defect location of the object to be detected.

[0164] The features of the second image can be represented by the second feature map, which includes spectral information about the material, chemical composition, and molecular structure of the object to be detected.

[0165] In some embodiments, the first feature map and the second feature map are aligned in spatial dimensions, that is, the first feature map and the second feature map are the same size, and points at the same position correspond to the same position of the object to be detected.

[0166] In some embodiments, the processor may determine the weights of the features of the first image and the features of the second image during the feature fusion process, and perform feature fusion on the two based on the weights of the features of the first image and the features of the second image.

[0167] The determination of weights is related to the detection target. The detection target refers to the defects that need to be focused on during detection. For example, if the detection target is a defect with strong spatial features, such as surface scratches, then the weights corresponding to the features in the first image can be increased. Conversely, if the detection target is a defect related to changes in material or composition, such as oil stains, then the weights corresponding to the features in the second image can be increased.

[0168] In some embodiments, the processor can determine the weights corresponding to the features of the first image and the features of the second image through a weight determination layer. In some embodiments, the weight determination layer can be an attention mechanism or a custom fusion layer.

[0169] The input to the weight determination layer can include a manually input detection target or a first feature map and a second feature map output by the feature determination layer. The output can include the features of the first image and the weights corresponding to the features of the second image. When the input to the weight determination layer is a manually input detection target, the weight determination layer can call up pre-stored weight data according to the detection target and output it; when the input to the weight determination layer is a first feature map and a second feature map, the weight determination layer can output weights according to the features of both.

[0170] In some embodiments, the processor can concatenate the first feature map and the second feature map along the channel dimension, and input the concatenated features into the first sub-network of the feature fusion layer, outputting a weight map. The pixel value of each pixel in the weight map is between 0 and 1; a larger value indicates a greater weight corresponding to the feature of the first image at that pixel location. The first sub-network may include convolutional layers, activation functions, and a sigmoid function.

[0171] In some embodiments, the processor can input the weight map, the first feature map, and the second feature map output by the first sub-network of the feature fusion layer into the second sub-network of the feature fusion layer, perform weighted fusion of the features of multiple pixels in the first feature map and the second feature map based on the weights in the weight map, obtain the first feature image, and use the first feature image as the detection image.

[0172] By using weighted maps, feature fusion can be performed separately for the features of different pixels, thereby flexibly focusing on information sources (first natural light camera or first hyperspectral camera) that are more useful for detecting different defect features. This helps to more clearly reflect the defects of the object to be detected in the detection image.

[0173] In some embodiments, the processor may also extract feature vectors from the first image and the second image respectively. For example, a 512-dimensional CNN feature vector is extracted from the first image, and a 100-dimensional (or 512-dimensional, the same dimension as the feature vector of the first image) spectral feature vector is extracted from the second image. The CNN feature vector and the spectral feature vector are then concatenated to obtain the vector to be detected, and the vector to be detected is used as the input to the visual detection model.

[0174] In some embodiments of the present invention, by fusing different image data from different information sources and fusing the features of the image data according to weights, the defects of the object to be detected in the image can be made more distinct, which is conducive to the visual detection model outputting more accurate results.

[0175] It should be noted that the above descriptions of the various processes are merely illustrative and do not limit the scope of the invention. Those skilled in the art can make various modifications and changes to the processes under the guidance of this invention. However, these modifications and changes are still within the scope of this invention. For example, steps 210 and 220 can be combined, steps 211 and 212 can be combined, or step 210 can be split. Another example is adding a storage step between the steps.

[0176] Figure 4 This is an exemplary block diagram of a visual inspection system according to some embodiments of the present invention. Figure 4As shown, the visual detection system 400 may include a first acquisition module 410, a training module 420, a second acquisition module 430, and a detection module 440.

[0177] The first acquisition module 410 is configured to acquire the training set.

[0178] In some embodiments, the first acquisition module 410 is further configured to control a non-contact optical sensor to scan the surface morphology information of the sample, determine the size and location information of defects on the sample based on the surface morphology information, control a second imaging device to capture training images of the sample based on the location information, and annotate the training images based on the size information to obtain a training set.

[0179] Training module 420 is configured to train an initial model based on the training set to obtain a visual detection model.

[0180] The second acquisition module 430 is configured to control the first imaging device to acquire detection images of the object to be detected.

[0181] The detection module 440 is configured to input the detection image into the visual detection model to obtain the defect detection result of the object to be detected.

[0182] The first acquisition module 410 executes steps 210, 211-214, the training module 420 executes step 220, the second acquisition module 430 executes step 230, and the detection module executes step 240.

[0183] In some embodiments, the visual inspection system 400 and its modules may be configured on the processor 110. Further detailed description of the functions performed by the visual inspection system 400 and its modules can be found in [reference needed]. Figures 2-3 The relevant description in the document.

[0184] Figure 5 This is an exemplary block diagram of a visual detection training apparatus according to some embodiments of the present invention. Figure 5 As shown, the visual inspection training device 500 may include a non-contact optical sensor 510, a second imaging device 520, a multi-degree-of-freedom motion platform 530, and a controller 540.

[0185] The non-contact optical sensor 510 is used to scan the surface morphology information of a sample.

[0186] The non-contact optical sensor 510 is a sensor that does not require physical contact with the object being measured and uses the optical properties of light, such as reflection, refraction, absorption, scattering, or interference, to obtain the position, distance, speed, shape, color, surface state, or other physical quantities of the target object.

[0187] In some embodiments, the non-contact optical sensor may include, but is not limited to, one or more combinations of white light interferometers, laser rangefinders, laser interferometers, and spectral confocal sensors. The white light interferometer measures based on white light interference; it is non-contact, with a probe diameter of 0.05-5 mm. Measurable materials include glass, metals, ceramics, and plastics. The minimum aperture diameter is approximately 0.05-0.2 mm, e.g., 0.1 mm. The sampling frequency can reach 40 kHz, the resolution can reach 0.05 nm, and the repeatability error is approximately 3 nm-7 nm, e.g., 5 nm. It can perform nanometer-level measurements for the quantitative identification of defects (such as depth, width, and roughness) and intelligent annotation, achieving an innovative fusion of measurement and visual inspection, and supporting spiral full-range scanning.

[0188] Meanwhile, the integration of the white light interferometer sensor with the vision system enables the detection of various materials and products, such as screen glass and metal casings. Non-contact measurement avoids damage to the product, improves accuracy to 0.03-0.05nm, reduces false positives, and is suitable for detecting display cracks and casing scratches, effectively improving detection accuracy.

[0189] In some embodiments, the non-contact optical sensor 510 can send the scanned surface topography information to the controller 540 for determining the size and location information of defects.

[0190] For detailed information on the surface morphology of the scanned samples, please refer to [link / reference]. Figure 3 Step 211 and its related description.

[0191] The second imaging device 520 is used to acquire training images of the samples.

[0192] In some embodiments, the second shooting device 520 may include a second camera assembly and a second lighting assembly.

[0193] In some embodiments, the second camera assembly may further include a zoom lens, an adapter, a displacement sensor, and a zoom motor. The zoom lens is an adjustable focal length lens, such as a 12x zoom lens; the adapter is a connecting component for establishing a physical, stable connection between the camera, lens, and other components; the displacement sensor is a sensing component for calibrating the positional relationship between the lens and the non-contact optical sensor, facilitating automatic adjustment of the position between the lens and the non-contact optical sensor when changing lenses; the zoom motor is a miniature drive device that drives the zoom lens group inside the zoom lens to move, thereby changing the focal length of the zoom lens.

[0194] In some embodiments, the second camera assembly may receive camera control instructions from the controller 540 and change at least one of the focal length and shooting angle according to the camera control instructions.

[0195] In some embodiments, the field of view of the second shooting device 520 is from 1.4mm×1.8mm to 10.92mm×14.56mm, supports automatic zoom, and can stitch together images for defects larger than 15mm×15mm, that is, take multiple images, one image contains the defect part, and stitch together multiple images to obtain an image containing the complete defect.

[0196] In some embodiments, the second lighting component may include, but is not limited to, one or more combinations of coaxial light, strip light, and ring light composed of multiple light source points. The second lighting component may receive lighting control commands from the controller 540 and change at least one of the illumination angle and wavelength according to the lighting control commands.

[0197] In some embodiments, the second camera component and the second light component can cooperate with each other to capture training images of the samples from different shooting angles using different light combinations and light angles.

[0198] For more detailed information on the second camera assembly, the second lighting assembly, and the acquisition of training images, please refer to [link to relevant documentation]. Figure 3 Step 213 and related descriptions.

[0199] The multi-degree-of-freedom mobile platform 530 is used to carry a second camera assembly, enabling the second camera assembly to move to a target position and / or adjust to a target shooting angle.

[0200] In some embodiments, the multi-degree-of-freedom mobile platform 530 includes a "gantry" three-axis module (X, Z, θy axis modules) (not shown) and a "table" two-axis module (Y, θz axis modules) (not shown).

[0201] In some embodiments, the three-axis module straddles the base like a "gantry" (gate frame). The crossbeam (X-axis) and column (Z-axis) of the "gantry" can be made of marble to ensure structural and thermal stability, providing a solid foundation for high-precision motion. In some embodiments, the X-axis control sensor of the three-axis module moves horizontally along the crossbeam of the gantry (travel 460-500mm); the Z-axis control sensor of the three-axis module moves vertically up and down on the column (travel 300-400mm); the θy-axis of the three-axis module is a high-precision rotary table integrated at the end of the Z-axis, which can drive the sensor to rotate 360° around the horizontal axis (Y-axis).

[0202] In some embodiments, the X and Z axes are driven by high-performance linear motors, guided by cross roller guides, and provided with real-time position feedback by high-resolution (0.05-0.2μm, such as 0.1nm) Renishaw grating rulers to form a fully closed-loop control.

[0203] In some embodiments, the two-axis module (Y, θz axis module) is located below the "gate frame" and serves as a platform for carrying and positioning the workpiece to be tested.

[0204] In some embodiments, the Y-axis of the two-axis module controls the entire worktable to move horizontally (360mm) in a direction perpendicular to the X-axis of the three-axis module. The Y-axis of the two-axis module, together with the X and Z axes of the three-axis module, constitute a complete Cartesian coordinate system (XYZ three-dimensional space).

[0205] In some embodiments, the θz axis of the two-axis module is an air-bearing turntable that carries the workpiece and drives it to rotate 360° around the vertical axis (Z-axis).

[0206] The combination of the three-axis module and the two-axis module allows the workpiece to be adjusted to the optimal measurement position by rotation without the need for reclamping during measurement.

[0207] In some embodiments, the multi-degree-of-freedom mobile platform 530 can receive movement control commands from the controller 540 to move the second camera assembly to the target location. Further details can be found in [link to relevant documentation]. Figure 3 The relevant description in step 213.

[0208] The controller 540 is used to control at least one of the non-contact optical sensor 510, the second imaging device 520, and the multi-degree-of-freedom mobile platform 530, and to receive and process data from the non-contact optical sensor 510, the second imaging device 520, the multi-degree-of-freedom mobile platform 530, and / or the user.

[0209] For example, the controller 540 can receive surface topography data from the non-contact optical sensor 510 and process the surface topography data to determine at least one of the size information, location information, and defect type of the defect on the sample.

[0210] For example, the controller 540 can receive multiple training images from the second imaging device 520 and annotate the multiple training images based on the size information of the defects to obtain a training set.

[0211] For example, the controller 540 can receive sample data input by the user (e.g., product parameters such as screen size or shell curvature) and automatically generate a scanning path for controlling the non-contact optical sensor 510 to scan, and / or generate a shooting path for controlling the second shooting device 520 to shoot, through a built-in modeling core (such as the OpenCascade 3D CAD core). It can also correct the position of the second camera component in the non-contact optical sensor 510 and / or the second shooting device 520 based on the position data returned by the sensors in the non-contact optical sensor 510 and / or the second shooting device 520.

[0212] In some embodiments, the controller 540 can also generate variant images by adding noise or changing angles based on the point cloud data corresponding to the training images, and annotate the variant images to achieve automatic expansion of the training set.

[0213] In some embodiments, the controller 540 may also perform other functions. For example, it may perform shape error assessment using a Gaussian / Chebyshev algorithm, provide a three-dimensional color deviation map, measure the roughness of the sample surface, update the training set, defects, and related features, and publish to a low-cost platform (such as a conventional vision device) via over-the-air (OTA) updates.

[0214] The visual detection training device 500 can be used to acquire a training set and train an initial model based on the training set to obtain a trained visual detection model.

[0215] In some embodiments, samples for obtaining the training set can be manually placed in the scanning area of ​​the non-contact optical sensor 510. After receiving a start command (such as manually pressing a start button, or in response to sensing that a sample has been placed), the visual inspection training device 500 performs a spiral full scan using the non-contact optical sensor 510 to obtain surface morphology data of the sample. The surface morphology data is analyzed by the controller 540 to obtain the size and location information of defects. The second imaging device 520 is controlled to take pictures of the sample based on the size and / or location information of the defects. During the shooting process, the second camera component in the second imaging device 520 can be moved by the multi-degree-of-freedom moving platform 530 to obtain multiple training data at different positions and shooting angles. The training images are labeled based on the size and location information of the defects to obtain labeled training images to form a training set. The controller 540 trains the initial model based on the training set to obtain a trained visual inspection model.

[0216] For detailed instructions on obtaining the training set and training the model, please refer to [link / reference]. Figure 2 Steps 210-220 Figure 3 And its related descriptions.

[0217] The 500 visual inspection training device is suitable for training models related to the detection of appearance defects, such as cracks in displays, pixel anomalies, color distortion, foreign object embedding, as well as scratches, dents, and bending deformations on various product exterior parts. Experimental results show that the detection repeatability error is <5nm, the efficiency is 50% higher than manual annotation, the false negative rate is <1%, and the accuracy reaches 99%.

[0218] In some embodiments of the present invention, the visual inspection training device integrates the entire process of training set acquisition, training sample labeling and model training during model training, reducing human intervention, effectively avoiding errors caused by human mistakes, obtaining a more accurate visual inspection model, which is conducive to ensuring the accuracy of defect detection and improving the level of product quality control.

[0219] The basic concepts have been described above. It is clear that the detailed disclosure above is merely illustrative and does not constitute a limitation of the present invention. Although not explicitly stated herein, various modifications, improvements, and corrections may be made to the present invention by those skilled in the art. Such modifications, improvements, and corrections are suggested in this invention and therefore remain within the spirit and scope of the exemplary embodiments of the present invention.

[0220] Meanwhile, specific terms are used to describe embodiments of the invention. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of the invention. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this invention do not necessarily refer to the same embodiment. Furthermore, certain features, structures, or characteristics in one or more embodiments of the invention can be appropriately combined.

[0221] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this invention are not intended to limit the order of the processes and methods of this invention. Although the foregoing disclosure has discussed some embodiments of the invention that are currently considered useful through various examples, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of this invention. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely through software solutions, such as installing the described system on existing servers or mobile devices.

[0222] Similarly, it should be noted that, in order to simplify the description of this invention and thus aid in the understanding of one or more embodiments, the foregoing description of this invention sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of the invention requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0223] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of the invention are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0224] For each patent, patent application, patent application publication, and other material, such as articles, books, specifications, publications, and documents, referenced in this invention, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this invention, as well as documents that limit the broadest scope of the claims of this invention (currently or subsequently appended to this invention). It should be noted that if there are any inconsistencies or conflicts between the descriptions, definitions, and / or terminology used in the appended materials of this invention and the content of this invention, the descriptions, definitions, and / or terminology used in this invention shall prevail.

[0225] Finally, it should be understood that the embodiments described in this invention are merely illustrative of the principles of the invention. Other variations may also fall within the scope of this invention. Therefore, alternative configurations of the invention are considered as examples rather than limitations, and are regarded as consistent with the teachings of the invention. Accordingly, the embodiments of the invention are not limited to those explicitly described and illustrated herein.

Claims

1. A visual inspection method, characterized in that, include: Obtain the training set; An initial model is trained based on the training set to obtain a visual detection model; Control the first imaging device to acquire detection images of the object to be detected; The detected image is input into the visual detection model to obtain the defect detection result of the object to be detected; The acquisition of the training set includes: Control the non-contact optical sensor to scan the surface morphology information of the sample; Based on the surface morphology information, the size and location information of the defects on the sample are determined; Based on the location information, control the second imaging device to acquire training images of the sample; The training images are labeled based on the size information to obtain the training set.

2. The method as described in claim 1, characterized in that, The number of training images acquired for at least one of the samples is multiple, and the second imaging device includes a second camera component and a second lighting component; The step of controlling the second imaging device to capture training images of the sample based on the location information includes: Multiple candidate shooting strategies are determined based on multiple second lighting schemes corresponding to the second lighting component and multiple shooting angles corresponding to the second camera component. The second lighting scheme includes the light type and / or illumination angle of at least one light in the second lighting component. The candidate shooting strategies include the shooting angle for controlling the second camera component to perform image acquisition and the corresponding second lighting scheme. Based on the aforementioned multiple shooting strategies, the second shooting device is controlled to acquire multiple training images of the sample.

3. The method as described in claim 2, characterized in that, The determination of multiple shooting strategies based on multiple second lighting schemes corresponding to the second lighting component and multiple shooting angles corresponding to the second camera component includes: Based on the size information and the location information, the defect type is determined; A target shooting strategy is selected from the plurality of candidate shooting strategies based on at least one of the defect type, the location information, and the shooting angle.

4. The method as described in claim 1, characterized in that, The surface topography information includes the three-dimensional point cloud data of the sample; and / or, The dimensional information includes at least one of the following: length, width, height, surface area, and volume of the defect.

5. The method as described in claim 1, characterized in that, The second imaging device includes a second camera assembly; The annotation of the training image based on the size information includes: Based on the shooting parameters of the second camera component, the defective point cloud data in the three-dimensional point cloud data is projected onto the training image to form defective scattered point clusters; The defect outline is obtained by fitting the outline of the scatter cluster of defects with a polygon. Based on the defect contours, defect information is labeled on the training images.

6. The method as described in claim 1, characterized in that, The first imaging device includes a first natural light camera and a first hyperspectral camera; The control of the first imaging device to acquire detection images of the object to be detected includes: The first natural light camera is controlled to capture a first image of the object to be detected, and the first hyperspectral camera is controlled to capture a second image of the object to be detected. The features of the first image and the features of the second image are fused to obtain a fused first feature image; The first feature image is used as the detection image.

7. The method as described in claim 1, characterized in that, The second imaging device includes a second natural light camera and a second hyperspectral camera; The process of controlling the second imaging device to capture training images of the samples includes: Control the second natural light camera to capture a third image of the sample, and control the second hyperspectral camera to capture a fourth image of the sample; The features of the third image and the features of the fourth image are fused to obtain a fused second feature image; The second feature image is used as the training image.

8. The method as described in claim 1, characterized in that, The first shooting device includes a first lighting assembly and a first camera assembly; The control of the first imaging device to acquire detection images of the object to be detected includes: Obtain a lighting scheme library corresponding to the product type of the object to be tested. The lighting scheme library includes at least one first lighting scheme, and the first lighting scheme includes the light type and / or illumination angle of at least one light in the first lighting component. The detection image of the object to be detected is acquired based on the at least one first illumination scheme; The method further includes: In response to an abnormal defect detection result of the object under test, a test reminder is issued to conduct a lighting scheme test on the object under test; Acquire test data and determine the optimal lighting scheme based on the test data; The lighting scheme library corresponding to the product type of the object to be tested is updated based on the optimal lighting scheme.

9. A visual inspection system, characterized in that, It includes a first acquisition module, a training module, a second acquisition module, and a detection module; The first acquisition module is configured to acquire the training set; The training module is configured to train an initial model based on the training set to obtain a visual detection model; The second acquisition module is configured to control the first imaging device to acquire detection images of the object to be detected; The detection module is configured to input the detection image into the visual detection model to obtain the defect detection result of the object to be detected; wherein... The first acquisition module is further configured as follows: Control a non-contact optical sensor to scan the surface topography information of a sample, wherein the surface topography information is three-dimensional data; Based on the surface morphology information, the size and location information of the defects on the sample are determined; Based on the location information, control the second imaging device to capture training images of the sample; The training images are labeled based on the size information to obtain the training set.

10. A visual detection training device, characterized in that, Training the visual detection model as described in claim 1 includes: Non-contact optical sensors are used to scan the surface morphology information of samples; The second imaging device, which includes a second camera assembly and a second lighting assembly, is used to acquire training images of the samples. A multi-degree-of-freedom mobile platform for mounting the second camera assembly; and The controller is used to control at least one of the non-contact optical sensor, the second imaging device, and the multi-degree-of-freedom mobile platform, and to receive and process data from the non-contact optical sensor, the second imaging device, and / or the multi-degree-of-freedom mobile platform.

Images