Automatic full detection system and detection method based on machine vision

Abstract

The invention relates to the technical field of machine vision, and discloses an automatic full-inspection system and method based on machine vision, and the system comprises an image collection unit which is used for collecting an image of a to-be-detected product; the programmable light source unit is used for changing illumination conditions; the data storage unit is used for storing models and data; the calculation processing unit is used for generating the position of a potential abnormal point based on the physical rendering prior model and the reference image; reconstructing local three-dimensional information and a differentiable scene model; extracting multi-dimensional information of the potential abnormal points; and carrying out reverse solving to obtain physical attribute parameters of the potential abnormal points, and forming physical attribute vectors. According to the method, the physical rendering prior model is constructed in the offline stage, and the optimal sparse illumination condition combination is determined for online acquisition based on the causal relationship graph between the defect physical attributes and the optimal observation illumination conditions, so that efficient sparse image acquisition is realized, and the detection efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, specifically to an automated full inspection system and inspection method based on machine vision. Background Technology

[0002] In the industrial product manufacturing sector, surface quality inspection is a crucial step in ensuring product functionality, reliability, and appearance quality. Currently, surface defect inspection of industrial products mainly relies on two types of technical methods. The first type is manual visual inspection. This method directly depends on the subjective judgment of the inspectors, and its results are easily affected by uncertain factors such as the inspectors' fatigue level, professional skill level, and inspection environment, leading to problems such as inconsistent inspection standards and a high rate of misjudgment.

[0003] The second category is automated inspection equipment using fixed cameras and light sources. While this type of equipment replaces manual labor to some extent, its technical solution has several limitations. First, in the data acquisition phase, to cover all potential defect types, the system typically needs to acquire a large number of images under different lighting conditions. This results in high data redundancy and long image acquisition cycles, directly reducing the overall operating efficiency of the inspection system.

[0004] Secondly, in the defect identification and analysis stage, existing automated equipment mainly relies on two-dimensional image processing algorithms or deep learning classification models. These methods identify defects by analyzing two-dimensional features such as pixels and textures in images. Their detection results only reflect the two-dimensional image features of the defect and cannot directly reflect the physical properties of the defect in three-dimensional space, such as depth, width, or volume. Therefore, when the physical morphology of the defect is complex or its optical features are not significant, these methods suffer from insufficient identification accuracy and struggle to establish unified, quantitative evaluation standards based on physical quantities.

[0005] Finally, in the defect assessment and determination stage, existing technologies typically compare extracted two-dimensional features with preset fixed thresholds. This method results in a subjective assessment and cannot analyze the evolution of defect attributes over time or production batches, thus lacking the ability to predict changes in production process conditions or future product failure risks. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an automated full inspection system and method based on machine vision, which solves the problems of low data acquisition efficiency, inability to accurately quantify the physical properties of defects, and subjective and unpredictable evaluation standards in existing technologies.

[0007] To achieve the above objectives, the first aspect of the present invention provides an automated full inspection system based on machine vision, comprising:

[0008] Image acquisition unit, used to acquire images of the product under test;

[0009] Programmable light source unit, used to change lighting conditions according to preset instructions;

[0010] Data storage unit, used to store models and data;

[0011] The computational processing unit is configured to perform the following operations:

[0012] Operation a: Based on the physical rendering prior model of the ideal product stored in the data storage unit, and the reference image acquired by the image acquisition unit under multiple lighting conditions provided by the programmable light source unit, generate the location of potential anomalies;

[0013] Operation b: For the potential anomaly points and their neighborhoods, based on the reference image, reconstruct a differentiable scene model containing local 3D information and light field information.

[0014] Operation c: In the differentiable scene model, perform virtual detection to extract multidimensional information of the potential anomaly points;

[0015] Operation d: Combining the multidimensional information with the physical rendering prior model, the physical attribute parameters of the potential anomaly point are obtained through reverse engineering, and these physical attribute parameters form a physical attribute vector.

[0016] Operation e: Based on the physical attribute vector, perform a quantitative evaluation of the potential anomalies.

[0017] Preferably, before executing operation a, the computational processing unit is further configured to perform the following operations: by introducing virtual defects into the physical rendering prior model and simulating imaging results under different lighting conditions, learn and construct a causal relationship map revealing the relationship between the physical properties of defects and the optimal observation lighting conditions; and, based on the causal relationship map, determine a set of optimal sparse lighting condition combinations for acquiring the reference image during online detection, and instruct the programmable light source unit to use this combination. Wherein, for a given virtual defect physical property φ... k Its optimal observation lighting conditions This can be determined by maximizing the information gain function H:

[0018]

[0019] In the formula, L j Let H(φ) be an illumination condition in the illumination space Λ. k |L j ) is used to measure L under illumination conditions j Below, based on the defect attribute φk This results in changes to the image information.

[0020] In one specific embodiment, the differentiable scene model is constructed based on a neural radiation field. This model maps three-dimensional spatial point coordinates x and a two-dimensional observation direction d to a volume density σ and a viewpoint-dependent color c. This mapping relationship F... NeRF (.) can be represented as:

[0021] (c,σ)=F NeRF (x,d;Θ);

[0022] In the formula, Θ represents all trainable weight parameters of the multilayer perceptron network;

[0023] By integrating the output of this field along the camera ray r(t), the pixel color C(r) of the image from any virtual viewpoint can be rendered:

[0024]

[0025] In the formula, t ncar and t far These are predefined near and far integration boundaries; T(t) is the integral from t along the ray. ncar The cumulative transmittance up to t, σ(r(t)) is the volume density at point r(t) in three-dimensional space, and T(t) represents the probability that light can reach point r(t) without being blocked, which is defined as:

[0026]

[0027] In the formula, exp(.) is an exponential function, σ(r(s)) is the volume density at point r(s) in three-dimensional space, and t is a scalar depth parameter along the camera ray, serving as the independent variable of the function and the upper limit of the integral.

[0028] Preferably, the virtual detection includes at least one of the following operations: virtual depth focusing, which calculates rendered images with different focal plane depths in the differentiable scene model to determine the focal plane depth that can maximize the feature signal of the potential anomaly point; and virtual illumination detection, which calculates rendered images under new virtual illumination conditions in the differentiable scene model to determine the illumination conditions that can maximize the feature signal of the potential anomaly point.

[0029] In one specific embodiment, the physical property vector includes at least one of the following parameters: the geometric shape parameter of the defect, the material optical property variation parameter, and the surface stress distribution parameter.

[0030] Preferably, the operation of quantifying the potential outlier includes: inputting the physical attribute vector into a pre-trained quantification evaluation model to output a continuous value characterizing the acceptability of the potential outlier.

[0031] Furthermore, the data storage unit is also used to construct a time-series database, and the computing processing unit is also used to: store the physical attribute vector and the continuous value of each detected potential anomaly point into the time-series database; and, when performing quantitative evaluation, retrieve and refer to the historical data stored in the time-series database to analyze the evolution trend of defect attributes over time or batches.

[0032] In a more specific embodiment, the computational processing unit is further configured to: apply one or more virtual environmental stresses and aging conditions to the differentiable scenario model; predict the future trend of the physical attribute vector based on the virtual conditions; and use the predicted trend as a risk factor to assist in the quantitative assessment.

[0033] Preferably, the physical rendering prior model is a differentiable renderer built on a bidirectional reflection distribution function.

[0034] A second aspect of this invention provides an automated full-inspection method based on machine vision, comprising the following steps:

[0035] Step a: Based on the physical rendering prior model of the ideal product and the benchmark images of the product under test collected under multiple lighting conditions, generate the locations of one or more potential anomalies.

[0036] Step b: For the potential anomaly points and their neighborhoods, reconstruct a differentiable scene model containing local 3D information and light field information based on the reference image;

[0037] Step c: In the differentiable scene model, perform virtual detection to extract multidimensional information of the potential anomaly points;

[0038] Step d: Combining the multidimensional information with the physical rendering prior model, the physical attribute parameters of one or more potential anomalies are obtained through reverse engineering. These one or more physical attribute parameters form a physical attribute vector.

[0039] Step e: Quantitatively evaluate the potential anomalies based on the physical attribute vector.

[0040] This invention provides an automated full-inspection system and method based on machine vision. It has the following beneficial effects:

[0041] 1. This invention constructs a physical rendering prior model in the offline stage and determines the optimal combination of sparse lighting conditions for online acquisition based on the causal relationship between defect physical properties and optimal observation lighting conditions, thereby achieving efficient sparse image acquisition and improving detection efficiency.

[0042] 2. This invention reconstructs a differentiable scene model containing three-dimensional information and light field information for potential anomalies, performs virtual detection in this model, and then combines a physical rendering prior model to solve in reverse to decouple the physical attribute parameters of the defects. This solves the problems of insufficient defect recognition accuracy and inconsistent quantification standards caused by relying on two-dimensional image classification in the prior art, thereby improving the recognition accuracy.

[0043] 3. This invention obtains objective, continuous values ​​by inputting physical attribute vectors into a quantitative evaluation model, and combines them with a time-series database for evolution trend analysis and risk prediction. This enables a physical standard-based, traceable, and forward-looking evaluation, solving the problems of strong subjectivity, inconsistent standards, and inability to predict quality risks caused by relying on human experience or simple threshold judgments in existing technologies, and further improving the accuracy of the evaluation. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the system architecture of the present invention;

[0045] Figure 2 This is a schematic diagram of the method flow of the present invention.

[0046] Among them, 10 is an image acquisition unit; 20 is a programmable light source unit; 30 is a data storage unit; and 40 is a computing and processing unit. Detailed Implementation

[0047] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] To better understand the present invention, the above content will be described in detail below with reference to specific embodiments.

[0049] Please see the appendix Figure 1 , Figure 1 This is a schematic diagram of the system architecture of an automated full inspection system based on machine vision provided in an embodiment of the present invention. The system includes:

[0050] Image acquisition unit 10, programmable light source unit 20, data storage unit 30, and computing and processing unit 40;

[0051] Image acquisition unit 10 is used to acquire images of the product under test. In one specific embodiment, the image acquisition unit 10 may be one or more industrial cameras. Each industrial camera includes an image sensor, such as a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) sensor, and a matching lens assembly. The image acquisition unit 10 is arranged in a predetermined position to ensure that its field of view can cover all or part of the detection area of ​​the product under test. After receiving a trigger signal from the computing processing unit 40, the image acquisition unit 10 performs an exposure operation and converts the captured optical image into digital image data.

[0052] The programmable light source unit 20 is used to change the illumination conditions according to preset instructions from the computing processing unit 40. In a specific embodiment, the programmable light source unit 20 may be an array light source composed of multiple independently controllable light-emitting diodes (LEDs), or a digital light processor (DLP) paired with a main light source. The computing processing unit 40 can send instructions to the programmable light source unit 20 to control the direction, intensity, color, and spatial distribution pattern of the illumination light, thereby forming the required illumination conditions on the surface of the product under test.

[0053] The data storage unit 30 is used to store the models and data required for system operation, and is a non-transitory computer-readable storage medium, such as a solid-state drive (SSD), a hard disk drive (HDD), or random access memory (RAM). The content stored includes, but is not limited to: a physical rendering prior model of the ideal product, a causal relationship map constructed during the offline learning phase, the original benchmark image acquired by the image acquisition unit 10, intermediate data generated during processing (such as difference maps and saliency maps), and the finally decoupled physical attribute vectors and quantitative evaluation results.

[0054] The computing processing unit 40 is used to perform all data processing and control tasks in the detection method of the present invention; in a specific embodiment, the computing processing unit 40 may be an industrial control computer (IPC), a workstation, or a server equipped with a central processing unit (CPU) and a graphics processing unit (GPU); the GPU can help accelerate parallel computing-intensive tasks, such as physically based rendering and training and rendering of differentiable scene models.

[0055] During system operation, the computing processing unit 40 establishes a communication connection with the image acquisition unit 10, the programmable light source unit 20, and the data storage unit 30 via a data bus, input / output interface, or network interface. The computing processing unit 40 first reads a preset illumination command from the data storage unit 30 and sends it to the programmable light source unit 20 to generate the first illumination condition. Subsequently, the computing processing unit 40 sends a synchronization trigger signal to the image acquisition unit 10 to acquire a product image under this illumination condition. The acquired image data is transmitted back to the computing processing unit 40 for processing and can be stored in the data storage unit 30. This process is repeated until image acquisition under all preset illumination conditions is completed.

[0056] In a specific embodiment:

[0057] The construction of the physical rendering prior model stored in data storage unit 30 first involves acquiring the three-dimensional geometric data of the ideal product. This three-dimensional geometric data can be exported from the product's computer-aided design (CAD) model and converted into a three-dimensional mesh model composed of vertices and faces (e.g., a triangular mesh) to describe the morphology of the ideal product's surface.

[0058] Subsequently, the optical properties of the material on the ideal product surface are modeled and described using the bidirectional reflectance distribution function (BRDF). The BRDF defines the reflection characteristics of light when interacting with an opaque surface. For a given surface point, it is defined as a function whose input is the incident light direction ω. i and the direction of departure (observation) ω o The output is the ratio of emitted radiance to incident irradiance.

[0059] In this embodiment, the BRDF model based on microsurface theory is adopted, and its mathematical expression is:

[0060]

[0061] In the formula: n is the macroscopic normal vector of the surface;

[0062] ω i It is the normalized incident light direction vector;

[0063] ω o It is the normalized outgoing direction vector;

[0064] ω h It is a half-range vector, defined as:

[0065] D(ω h ) is the normal distribution function, which describes the probability density of the direction of the normal to the half-path vector of a micro-surface. Its specific form is determined by parameters such as roughness.

[0066] F(ω i ,ω h The α is Fresnel's equation, used to describe the reflectance of light at different incident angles. Its specific form is determined by parameters such as the refractive index of the material.

[0067] G(ω i ,ω o ,ω h () is a geometric occlusion function used to describe the self-occlusion and self-shadowing effects between micro-surfaces;

[0068] The parameters in the BRDF model described above (e.g., base color, metallicity, roughness, refractive index) are obtained by measuring and calibrating an ideal product sample or directly from the product design specifications. These parameters, combined with three-dimensional geometric data, constitute a complete physical description of the ideal product.

[0069] Finally, this physical description is integrated into a differentiable renderer, forming a complete physical rendering prior model. This renderer calculates the path from a surface point x along the direction ω based on the rendering equation. o Emitted radiance L o (x,ω o The calculation formula is as follows:

[0070] L o (x,ω o )=∫ Ω f r (x,ω i ,ω o )L i (x,ω i )(ω i ·n)dω i ;

[0071] In the formula: L i (x,ω i ) is from the direction ω i The radiance incident on point x; Ω is the integral domain covering the entire hemispherical region of point x.

[0072] Therefore, the renderer's implementation (e.g., using a computational graph framework that supports automatic differentiation) is differentiable, meaning it can calculate the gradient of the rendered image pixel values ​​with respect to the model input parameters (e.g., BRDF parameters, light source position, light source intensity). This gradient information forms the basis for subsequent inverse solving of physical property parameters.

[0073] Furthermore, before performing the online full inspection task, the computational processing unit 40 executes an offline pre-learning process to construct a causal relationship map revealing the relationship between the physical properties of defects and the optimal observation lighting conditions. This process utilizes the previously constructed physical rendering prior model.

[0074] The process begins by programmatically introducing predefined virtual defects into the physically rendered prior model. Virtual defects are specific modifications to the local physical property parameters of an ideal product model. For example, geometric virtual defects (such as dents) are implemented by applying Gaussian displacement to the vertex coordinates of a specific region of the 3D mesh model; material virtual defects (such as stains) are implemented by modifying the optical properties of a specific region's material, such as changing the base color or roughness parameters in its bidirectional reflectance distribution function (BRDF). In this way, a set of virtual defects {φ} containing various typical defect types is generated. k}, where each φ k It represents a specific deviation in physical properties.

[0075] Subsequently, the calculation processing unit 40 performs calculations on each virtual defect φ k In a discrete set of illumination conditions Λ={L j The process is iterated through in the set. For each lighting condition L in the set... j Perform the following rendering operations:

[0076] Using a physically based rendering prior model, render an image of the ideal product under this lighting condition. ideal (L j ).

[0077] virtual defect φ k It is applied to the physical rendering prior model and renders an image of a defective product under that lighting. defect (φ k ,L j ).

[0078] Next, to quantitatively evaluate the effectiveness of each illumination condition in revealing a specific defect, the computational processing unit 40 calculates the value of each illumination condition L. j The resulting defect φ k Information gain H(φ) k ,L j In this embodiment, the information gain is measured by calculating the Koolbek-Loebler divergence between the feature distributions of the two rendered images, expressed by the formula:

[0079] H(φ k |L j ) = D KL (P defect ||P ideal );

[0080] In the formula: D KL (P defect ||P ideal P represents the probability distribution of defect image features.defect Relative to the ideal image feature probability distribution P ideal KL divergence; P defect It is composed of image I defect (φ k ,L j The probability distribution of the generated normalized pixel intensity histogram; P ideal It is composed of image I ideal (L j The probability distribution of the generated normalized pixel intensity histogram.

[0081] For each type of virtual defect φ k The computational processing unit 40 determines the uniquely corresponding optimal observation illumination conditions by solving a maximization problem. The calculation formula is:

[0082]

[0083] In the formula, L j Let H(φ) be an illumination condition in the illumination space Λ. k |L j ) is used to measure L under illumination conditions j Below, based on defect attribute v k This results in changes to the image information.

[0084] The optimal observation lighting conditions It refers to the lighting condition among all the alternative lighting conditions that makes the difference between the defective image and the ideal image most significant.

[0085] Finally, all virtual defect types and their corresponding optimal observation lighting conditions are represented as key-value pairs. The data is organized and stored to form a causal relationship graph. This graph is physically stored in data storage unit 30 for later use in determining the optimal combination of sparse illumination conditions required for online detection. The data structure of this graph can be a hash table or a search tree to enable fast lookups.

[0086] After the computational processing unit 40 completes the construction of the causal relationship map, it performs an optimization process based on the map to determine an optimal combination of sparse illumination conditions. This combination will be used in the subsequent online detection stage to cover all known typical defect types with minimal image acquisition.

[0087] This determination process is modeled as a set coverage problem. First, a universal set U to be covered is defined, which consists of all virtual defect types {φ} generated in the preceding steps. k The structure is represented by the formula:

[0088] U = {φ1, φ2, ..., φN};

[0089] Where N is the total number of virtual defect types.

[0090] Next, based on the causal relationship map stored in data storage unit 30 Construct a subset S for each unique optimal observation illumination condition appearing in the causal relationship graph. Each of them generates a corresponding subset S j The subset S j Includes all of As the type of defect with optimal observation illumination conditions, its mathematical formula is expressed as:

[0091]

[0092] Therefore, set S is the set of all these subsets, that is:

[0093] S = {S1,S2,…,S} M};

[0094] M represents the total number of uniquely optimal observation lighting conditions.

[0095] The goal of this problem is to select a minimum number of subsets from a set S to form a subset combination. This ensures that the union of these selected subsets completely covers the entire set U. That is, it satisfies the formula:

[0096]

[0097] Among them, each subset S selected into combination C j The corresponding lighting conditions That is, it is selected into the final optimal combination of sparse lighting conditions.

[0098] To solve this set covering problem, computation processing unit 40 performs the following steps:

[0099] Initialize an empty set of covered defects U. covered And an empty final lighting combination L final .

[0100] When U covered If the set is not equal to the universal set U, repeat the following operations:

[0101] a. In all the subsets S that have not yet been selected j ∈S, select the defect that covers the most uncovered defects (i.e., |S) j \U covered The largest subset S best ;

[0102] b. Set S best All defects included are added to the covered defect set U. covered middle;

[0103] c. Will be with S best Corresponding lighting conditions Add to the final lighting combination L final middle;

[0104] The algorithm terminates, and the output L is... final This refers to a set of optimal sparse lighting conditions used to acquire reference images during online detection.

[0105] The computing processing unit 40 will calculate this lighting combination L final The data is stored in the data storage unit 30 or directly used to instruct the programmable light source unit 20 so as to perform efficient image acquisition during the online detection stage.

[0106] In one embodiment, after entering the online detection stage, the computational processing unit 40 first performs sparse sampling and potential anomaly generation operations.

[0107] The computational processing unit 40 retrieves the optimal combination of sparse lighting conditions determined during the offline pre-learning phase from the data storage unit 30:

[0108] L final ={L1,L2,…,L K}, where K is the total number of sparse illuminations. For each illumination condition L in this combination... i The computing and processing unit 40 sends control commands to the programmable light source unit 20 to generate the corresponding illumination mode. Simultaneously, it sends a synchronization trigger signal to the image acquisition unit 10 to acquire the image of the product under test under illumination L. i Image I below test (L i This process is repeated K times to eventually obtain a sparse reference image set containing K images.

[0109] After obtaining the sparse reference image set, the computational processing unit 40 performs a difference calculation process to identify regions that are inconsistent with the ideal product. For each illumination condition L i The computational processing unit 40 performs the following two operations:

[0110] Using the physically based rendering prior model stored in data storage unit 30, the ideal product is rendered under the same lighting conditions L. i Reference image I below ideal (L j ).

[0111] Calculate the reference image I of the product under test test (L j (and ideal product reference image I) ideal (L j The pixel-level differences between the two are used to generate a difference map D. i This calculation can be achieved by subtracting pixels one by one and taking the absolute value, expressed by the following formula:

[0112] D i (p)=|I test (L i ,p)-I ideal (L i ,p)|;

[0113] In the formula, p represents the pixel coordinates in the image.

[0114] Subsequently, the computational processing unit 40 generates difference maps under all lighting conditions. The data is then fused to generate a global saliency map M. sal This fusion operation aims to aggregate anomalous signals under all lighting conditions, ensuring that defects that only appear under specific lighting conditions are not overlooked. A specific fusion method is to take the maximum value of all difference maps at each pixel, calculated using the following formula:

[0115]

[0116] In this way, the global saliency graph M sal The high-brightness area on (p) corresponds to the area on the test product that differs significantly from the ideal product.

[0117] Finally, the computational processing unit 40 computes the global saliency map M. sal Processing is performed to generate the locations of one or more potential outliers. This processing includes:

[0118] For M sal Binarization is performed using a preset threshold to initially segment out salient regions.

[0119] Connectivity analysis is performed on the binarized image to identify each independent salient region.

[0120] Calculate the centroid coordinates of each connected component and use these coordinates as the location of potential outliers in that region.

[0121] The final output of this step contains a list of one or more Points of Interest (POI) coordinates. These coordinates precisely indicate the locations on the product under test that require further detailed analysis, providing targets for subsequent local scene reconstruction.

[0122] Subsequently, for each generated potential anomaly point (POI), the computational processing unit 40 performs a reconstruction process of a locally differentiable scene model. This model is used to characterize the 3D geometric and light field information of the POI and its neighborhood.

[0123] For a given POI, the computation processing unit 40 first processes the sparse reference image set stored in the data storage unit 30. From the image, column image patches centered on the POI are extracted. Simultaneously, the camera intrinsic and extrinsic parameters and illumination condition parameter L corresponding to these image patches are also extracted. i It was also extracted and used as input data for model training.

[0124] In this embodiment, the locally differentiable scene model is constructed based on a neural radiation field, denoted as F. NeRF The network takes the Cartesian coordinates x = (x, y, z) of a three-dimensional point in space and the observation direction d = (θ, φ) in two-dimensional unit spherical coordinates as input, and outputs the volume density σ of the point in space and the viewpoint-dependent color c = (r, g, b). This mapping relationship can be expressed as:

[0125] (c,σ)=F NeRF (x,d;Θ);

[0126] In the formula, Θ represents all trainable weight parameters of the multilayer perceptron network.

[0127] To render images from this implicit representation under arbitrary new viewpoints or lighting conditions, the computational processing unit 40 employs a volumetric rendering principle. For a pixel in the image, its corresponding camera ray r(t) = o + td is first determined, where o is the position of the camera optical center and td is the depth along the ray direction. Therefore, the color value C(r) of this pixel is calculated by integrating the color and density along the ray path, using the following formula:

[0128]

[0129] In the formula, t ncar and t far These are predefined near and far integration boundaries; T(t) is the integral from t along the ray. ncar The cumulative transmittance up to t, σ(r(t)) is the volume density at point r(t) in three-dimensional space, and T(t) represents the probability that light can reach point r(t) without being blocked, which is defined as:

[0130]

[0131] In the formula, exp(.) is an exponential function, σ(r(s)) is the volume density at point r(s) in three-dimensional space, and t is a scalar depth parameter along the camera ray, serving as the independent variable of the function and the upper limit of the integral.

[0132] The computational processing unit 40 trains the network parameters Θ through an optimization process. The goal of this process is to minimize the difference between the image rendered by the model and the actual acquired reference image patch. For each ray r in the training set... j Its loss function L is defined as the mean square error between the rendered color and the real pixel color, expressed by the following formula:

[0133]

[0134] in, It refers to the pixel color rendered by the model, while C gt (r j Θ is the true pixel color sampled from the corresponding baseline image patch. Gradient descent-type optimization algorithms (such as the Adam optimizer) are used to update the network weight parameters Θ until the loss function converges to a minimum.

[0135] After training, the optimized network F is obtained. NeRF This constitutes a locally differentiable scene model for the neighborhood of the POI. This model is stored in data storage unit 30 and provides the foundation for subsequent virtual detection and inverse physical property solving.

[0136] After reconstructing a locally differentiable scene model for a potential outlier (POI), the computational processing unit 40 utilizes the trained neural radiation field network F NeRF A virtual detection process is performed to explicitly extract the three-dimensional geometry and light field information within the neighborhood of the POI from the implicit model.

[0137] First, the computational processing unit 40 performs a virtual depth focusing operation to obtain high-precision 3D surface geometry within the neighborhood of the POI. For each pixel in the rendered image, the computational processing unit 40 calculates a desired depth value along that ray. This calculation is achieved by weighted integrating the points along the ray path according to their contribution to the final pixel color. Its mathematical expression is:

[0138]

[0139] The weighting function w(t) is defined as follows:

[0140] w(t) = T(t)·σ(r(t));

[0141] In this expression, t is the depth parameter along the ray direction; r(t) is a 3D point at depth t on the ray; σ(r(t)) is generated by the network F. NeRF The output is the volume density at that point; T(t) is the light ray from the near boundary t near The cumulative transmittance propagating to depth t is defined in the same way as in the previous steps. By performing this calculation on all rays within the local scene, a fine depth map of the POI neighborhood can be generated.

[0142] Subsequently, the computational processing unit 40 performs a virtual illumination detection operation to extract the local light field response characteristics of the surface. This operation is performed on the three-dimensional surface defined by virtual depth focusing. The computational processing unit 40 first selects one or more analysis points x from this three-dimensional surface. surf For each analysis point x surf The computational processing unit 40 generates a discrete set of virtual observation directions {d} covering the hemisphere above it. j}

[0143] For each virtual observation direction d in this set j The calculation and processing unit 40 calculates the coordinates x of the analysis point. surf and the virtual observation direction d j As input, query the trained network F NeRF (x surf ,d j This allows us to obtain a corresponding color output value c, Θ). j This process requires no additional image acquisition; it can be completed simply by querying the model within the computational processing unit 40.

[0144] By repeating the above query, a set of data pairs (d) is finally obtained. j ,c j This data set describes the surface point x under fixed original illumination conditions. surf How the color of the emitted light changes with the direction of observation constitutes a discrete sampling of the local reflection characteristics of that point.

[0145] The final output of this virtual probing step is a high-precision depth map of the POI neighborhood, along with sampled data of local reflectance characteristics at one or more surface points. This multidimensional information is then passed to the next processing step for precise decoupling of physical properties.

[0146] Subsequently, after obtaining the depth map and local reflection characteristic data of the neighborhood of potential anomalies (POIs), the computational processing unit 40 performs a physical attribute decoupling and quantization process. Through reverse rendering optimization, it reverse-solves the observed optical phenomena into a set of parameters with clear physical meaning.

[0147] The process begins by defining a physical property vector Ψ to be optimized. This vector contains all relevant parameters describing the physical state of the POI region. For example, this vector can be defined as:

[0148] Ψ=(Δx,Δy,Δz,c base ,m,α,…);

[0149] In the formula, Δx, Δy, and Δz represent geometric deviations relative to the ideal product surface (e.g., depth of indentation or height of protrusion); c base α represents the base color of the surface; m represents the metallicity; α represents the micro-surface roughness. These parameters directly correspond to the geometric and material parameters in the physical rendering prior model.

[0150] The computational processing unit 40 constructs this solution process as an optimization problem, the goal of which is to find a set of optimal physical property parameters Ψ. * This ensures that the result rendered by the physically based rendering prior model under these parameters best matches the data obtained through virtual probing in the previous step. This optimization process utilizes the differentiable property of the physically based rendering prior model.

[0151] For solving the geometric properties, a geometric loss function L is defined. geo It is used to measure the difference between the local surface modified by parameters (Δx, Δy, Δz) and the depth map obtained by virtual probing.

[0152] To solve for material properties, a material loss function L is defined. mat This function measures the effect of lighting conditions L given the initial illumination. i Under the conditions of geometry parameters, the material parameter (c) base The physical rendering prior model controlled by ,m,α) renders the images in each virtual observation direction d. j The color L on render (Ψ,d j ), and the color c obtained through virtual illumination detection. j The difference between them. The loss function can be expressed as:

[0153]

[0154] Therefore, the total loss function L total The weighted sum of the above losses is calculated as follows:

[0155] L total =λ geo L geo +λ mat L mat ;

[0156] Where, λ geoand λ mat These are preset weighting coefficients.

[0157] The computational processing unit 40 uses gradient descent to minimize the total loss function L. total In each iteration, the gradient of the loss function with respect to the physical property vector Ψ is calculated using automatic differentiation. The value of Ψ is updated in the opposite direction of the gradient, and its calculation formula is as follows:

[0158]

[0159] Among them, Ψ t Here, η is the parameter value for the t-th iteration, η is the learning rate, and Ψ is the parameter value for the t-th iteration. t+1 It is the parameter value for the (t+1)th iteration.

[0160] This iterative process continues until the loss function converges or the preset maximum number of iterations is reached. The final converged parameter vector Ψ * This represents the precise quantification of the physical properties of the POI region. For example, if the final result is Δz = -0.05 mm, this is the physical quantification of the depth of a pit. This physical property vector Ψ * It is stored in data storage unit 30 and used as input for the final acceptability assessment.

[0161] When the computational processing unit 40 completes the decoupling and quantization of the physical properties of a potential outlier (POI) and obtains its precise physical property vector Ψ * Next, an acceptability assessment process will be performed. This process compares the quantified physical deviations with pre-set quality standards to determine whether the product under test meets the factory specifications.

[0162] The process first requires a predefined acceptable tolerance template stored in data storage unit 30. This template is a multidimensional boundary set, denoted as T, corresponding to the physical property vector Ψ structure. For each component Ψ in the physical property vector Ψ... j (For example, indentation depth, surface roughness, etc.), a corresponding acceptable range [H] is defined in this template. j U j The calculation formula is as follows:

[0163]

[0164] In the formula: ψ j It is the j-th component in the physical attribute vector;

[0165] H j This is the acceptable lower limit for that component;

[0166] U jThis is the acceptable upper limit for this component;

[0167] P is the dimension of the physical property vector ψ;

[0168] These tolerance limits H j and U j It is pre-set according to product design specifications or quality control standards.

[0169] The computational processing unit 40 processes the optimal physical property vector Ψ obtained in the previous step. * Each component Perform a Boolean comparison operation with the corresponding interval in the acceptable tolerance template, expressed as follows:

[0170]

[0171] The result of this operation is B. j It is a Boolean value, when Located within its corresponding acceptable range [H] j U j [Inner time, B] j If true, then false; otherwise, it is false.

[0172] After performing the above judgment on all physical attribute components, the calculation processing unit 40 calculates based on all Boolean results. A final evaluation of the POI is then performed. In this embodiment, a POI is deemed acceptable if and only if all its physical property components meet the tolerance requirements. That is, the final evaluation result A is... POI The calculation formula is obtained by performing a logical AND operation on all Boolean values:

[0173]

[0174] If A POI If the value is true, it indicates that the physical deviation at the POI is within an acceptable range.

[0175] The processing unit 40 repeats this evaluation process for all detected POIs. If the evaluation results of all POIs on the product under test are acceptable, the product is judged to be a qualified product. If the evaluation result of any POI is unacceptable, the product is judged to be a non-qualified product. The final judgment result is output and can be used to trigger subsequent automated sorting or alarm devices.

[0176] In another specific embodiment, in order to predict potential quality risks in the production process, the present invention further includes time-series management of historical testing data and risk assessment based on virtual conditions.

[0177] First, the computational processing unit 40 executes the process of constructing and applying the time-series database. After each online test of a single product under test, the final quantized physical attribute vector Ψ is obtained. * Along with its corresponding unique product identifier and precise timestamp of test completion, this data, along with its corresponding product identifier and the timestamp of test completion, is stored as a data record in the time-series database located in data storage unit 30. The structure of each data record is (t... stamp ,PID,Ψ * ), where t stamp The timestamp is used for data entry, and the PID is used for product identification. Through continuous monitoring, this time-series database accumulates, forming a historical record of production quality that is time-based and measured by precise physical property deviations.

[0178] Based on this time-series database, the computational processing unit 40 executes a risk prediction process based on virtual conditions. This process is triggered periodically by the computational processing unit 40 or according to instructions. The computational processing unit 40 first extracts time-series data of the variation of specific physical attribute components over time from the time-series database.

[0179] For a selected physical property component Ψ j Its time series is (Ψ) j (t1),Ψ j (t2),…,Ψ j (tnow)). The computational processing unit 40 applies a pre-defined time series prediction model (e.g., Autoregressive Integrated Moving Average (ARIMA) or Long Short-Term Memory (LSTM) network) to analyze this sequence in order to predict the physical property at a future time point t. future The value is denoted as Ψ. j (t future Its formula is:

[0180] ψ j (t future )=f predict (ψ j (t1),ψ j (t2),…,ψ j (t now ));

[0181] Among them, f predict This represents the time series forecasting algorithm used. This forecasting process is performed independently on all physical property components of interest, thereby combining them into a complete future physical property vector ψ. j (t future ).

[0182] Here ψ(t) future The vector represents a future time point t. futureThe virtual state of the produced product is shown above. The computational processing unit 40 then performs a risk assessment on this virtual state. This assessment process directly reuses the aforementioned acceptability assessment logic, that is, the predicted future physical attribute vector ψ(t) future The components ψ j (t future It is compared with the acceptable tolerance template stored in the data storage unit 30.

[0183] For each predicted physical property component ψ j (t future The judgment operation is performed, and the formula is expressed as:

[0184] B j,future =(H j ≤ψ j (t future )≤U j );

[0185] Among them, H j and U j These are the upper and lower limits defined for the attribute component in the tolerance template. If the predicted value of any component exceeds its tolerance range, i.e., there is at least one B... j,future If the result is false, then the system determines that at time point t... future There is a high probability of exceeding the tolerance level.

[0186] When this risk is detected, the computational processing unit 40 generates a predictive maintenance alert. This alert clearly indicates the specific physical parameter expected to exceed tolerances, its predicted value, and the expected time point t. future This alarm can be used to prompt operators to perform equipment checks or adjust process parameters to prevent the actual generation of non-conforming products.

[0187] Please see the appendix Figure 2 , Figure 2 This is a flowchart illustrating an automated full-inspection method based on machine vision provided in an embodiment of the present invention. The method includes:

[0188] Step a: Based on the physical rendering prior model of the ideal product and the benchmark images of the product under test collected under multiple lighting conditions, generate the locations of one or more potential anomalies.

[0189] In this embodiment, the computational processing unit 40 loads a pre-defined physical rendering prior model containing the product's geometry and material optical properties from the data storage unit 30. Simultaneously, the image acquisition unit 10 acquires multiple frames of reference images of the product under test under various preset lighting conditions. The computational processing unit 40 uses the prior model to render multiple frames of reference images corresponding to the reference image acquisition conditions. By calculating the pixel-level differences between the reference images and the reference images, one or more regions where the differences exceed a preset threshold are identified, and the location coordinates of these regions are output as potential anomalies.

[0190] Step b: For each potential anomaly point and its neighborhood, reconstruct a differentiable scene model based on the benchmark image, incorporating local 3D information and light field information.

[0191] In summary, for each potential anomaly point identified in step a, the computational processing unit 40 defines a neighborhood centered on that point. Based on the image data corresponding to this neighborhood in multiple reference images, a locally differentiable scene model is reconstructed through an optimization process. This model is a continuous function, whose inputs are three-dimensional coordinates and the observation direction, and whose outputs are the volume density and color value of the point, thereby characterizing the three-dimensional geometric and directional reflection characteristics of the neighborhood.

[0192] Step c: In the differentiable scene model, perform virtual probing to extract multidimensional information of potential anomalies;

[0193] In this embodiment, the computational processing unit 40 performs virtual probing on the locally differentiable scene model generated in step b. This process involves intensive sampling within the space defined by the model to calculate and extract multidimensional information. This information includes the three-dimensional coordinates of a series of spatial points within the model region, surface normal vectors, and color and brightness values ​​under different virtual observation directions.

[0194] Step d: Combining multidimensional information with the physical rendering prior model, reverse-engineer one or more physical attribute parameters of potential anomalies to form a physical attribute vector;

[0195] In this embodiment, the computational processing unit 40 performs a reverse solution process. This process sets an objective function to quantify the difference between the multidimensional information extracted in step c and the rendering result of the parameterizable physical model. The physical model here is based on the prior model of step a, but its key physical properties (such as pit depth, scratch width, and refractive index) are adjustable parameters. These physical property parameters are adjusted through an iterative optimization algorithm to minimize the value of the objective function. When the algorithm meets the preset convergence condition, the current set of physical property parameter values ​​is determined as the solution result and organized into a physical property vector.

[0196] Step e: Quantitatively evaluate potential anomalies based on physical attribute vectors;

[0197] In this embodiment, the computational processing unit 40 performs a quantitative evaluation of the physical attribute vector generated in step d. Each component of this vector is compared with the preset tolerance range of the corresponding physical attribute stored in the data storage unit 30. If all components are within their respective tolerance ranges, the potential anomaly is determined to meet the quality specifications. Conversely, if any component exceeds its tolerance range, it is determined to be non-compliant. Finally, based on the determination results of all potential anomalies, an evaluation conclusion for the entire product under test is output.

Claims

1. An automated full inspection system based on machine vision, characterized in that, include: Image acquisition unit, used to acquire images of the product under test; Programmable light source unit, used to change lighting conditions according to preset instructions; Data storage unit, used to store models and data; Computational processing unit, used for: Based on the physical rendering prior model of the ideal product stored in the data storage unit, and the reference image acquired by the image acquisition unit under multiple lighting conditions provided by the programmable light source unit, the location of potential anomalies is generated. Based on the reference image, a differentiable scene model with local 3D information and light field information is reconstructed for the potential anomalies and their neighborhoods. In the differentiable scenario model, virtual detection is performed to extract multidimensional information of the potential anomalies; By combining the multidimensional information with the physical rendering prior model, the physical attribute parameters of the potential anomaly points are obtained by reverse engineering, forming a physical attribute vector. Based on the physical attribute vector, the potential anomalies are quantitatively evaluated.

2. The automated full inspection system based on machine vision according to claim 1, characterized in that, The computing processing unit is also used for: Before generating the location of the potential anomaly point, the following is also included: By introducing virtual defects into the physical rendering prior model and simulating imaging results under different lighting conditions, a causal relationship map revealing the relationship between the physical properties of defects and the optimal observation lighting conditions is learned and constructed. Based on the causal relationship map, an optimal combination of sparse lighting conditions is determined for acquiring the reference image during online detection, and the programmable light source unit is instructed to use this combination.

3. The automated full inspection system based on machine vision according to claim 1, characterized in that, The differentiable scene model is constructed based on neural radiation fields.

4. The automated full inspection system based on machine vision according to claim 1, characterized in that, The virtual detection includes: Virtual depth focusing involves calculating rendered images with different focal plane depths in the differentiable scene model to determine the focal plane depth that maximizes the feature signals of the potential anomalies. Virtual illumination detection involves calculating a rendered image under new virtual illumination conditions within the differentiable scene model to determine the illumination conditions that maximize the feature signals of the potential anomalies.

5. The automated full inspection system based on machine vision according to claim 1, characterized in that, The physical property vector includes: the geometric morphology parameters of the defect, the material optical property variation parameters, and the surface stress distribution parameters.

6. The automated full inspection system based on machine vision according to claim 1, characterized in that, In the quantitative evaluation of the potential outliers, the physical attribute vector is input into a pre-trained quantitative evaluation model to output a continuous value representing the acceptability of the potential outliers.

7. The automated full inspection system based on machine vision according to claim 6, characterized in that, The data storage unit is also used to construct a time-series database, and the computing processing unit is also used to: The physical attribute vector and the continuous value of each detected potential anomaly are stored in the time series database; Furthermore, when performing quantitative evaluation, the computing processing unit also retrieves and refers to historical data stored in the time-series database to analyze the evolution trend of defect attributes over time or batches.

8. The automated full inspection system based on machine vision according to claim 7, characterized in that, The computing processing unit is also used for: In the differentiable scenario model, one or more virtual environmental stresses and aging conditions are applied; Based on the virtual conditions, predict the future trend of the physical attribute vector; The predicted trend is used as a risk factor to assist in quantitative assessment.

9. The automated full inspection system based on machine vision according to claim 1, characterized in that, The physical rendering prior model includes a differentiable renderer built on a bidirectional reflection distribution function.

10. A machine vision-based automated full inspection method, using a machine vision-based automated full inspection system as described in any one of claims 1-9, characterized in that, Includes the following steps: Step a: Based on the physical rendering prior model of the ideal product and the benchmark images of the product under test collected under multiple lighting conditions, generate the locations of one or more potential anomalies. Step b: For each potential anomaly point and its neighborhood, reconstruct a differentiable scene model based on the reference image, using local 3D information and light field information. Step c: In the differentiable scene model, perform virtual detection to extract multidimensional information of the potential anomaly points; Step d: Combining the multidimensional information with the physical rendering prior model, reverse-engineer one or more physical attribute parameters of the potential anomaly point to form a physical attribute vector; Step e: Quantitatively evaluate the potential anomalies based on the physical attribute vector.

Images