Industrial application-oriented three-dimensional visual detection method, system and equipment

Abstract

The invention discloses a three-dimensional visual detection method, system and equipment for industrial application, and relates to the related field of three-dimensional computer vision, and the method comprises the steps: obtaining a three-dimensional point cloud data set of a target product, carrying out the multi-dimensional feature analysis based on the three-dimensional point cloud data set, and constructing a multi-modal feature vector set; performing three-dimensional visual defect detection according to the multi-modal feature vector group to generate a defect detection quality report; and making a control decision instruction according to the defect detection quality report, and automatically controlling the target product through the control decision instruction. The technical problem that existing industrial application-oriented three-dimensional visual detection is insufficient in detection accuracy and comprehensiveness is solved, and the technical effect of improving the detection accuracy and comprehensiveness is achieved.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional computer vision, and in particular to a three-dimensional visual detection method, system and equipment for industrial applications. Background Art

[0002] In the field of industrial production, accurate detection of product quality is crucial to ensuring production efficiency, improving product qualification rate and maintaining corporate reputation. As an efficient and accurate detection method, three-dimensional visual inspection has a significant impact on industrial applications. At present, the main method to solve the problem of three-dimensional visual inspection of industrial products is to first obtain the three-dimensional point cloud data of the product, then perform simple feature extraction and analysis based on this data, and then perform defect detection and generate a report according to preset rules, and finally formulate control decision instructions based on the report. Because the existing methods are only based on simple feature extraction and analysis, and lack in-depth mining and comprehensive consideration of three-dimensional point cloud data, when faced with complex and changeable product defects, the accuracy and comprehensiveness of the detection are insufficient, and it is difficult to accurately identify various potential defects, which makes the generated quality report less accurate and the formulation of control decision instructions lack scientific basis, affecting the effect of automated control.

[0003] Among the current related technologies, three-dimensional visual inspection for industrial applications has technical problems such as insufficient accuracy and comprehensiveness of detection. Summary of the Invention

[0004] The present application provides a three-dimensional visual inspection method, system and equipment for industrial applications. After obtaining a three-dimensional point cloud dataset of a target product, multi-dimensional feature analysis is performed based on the three-dimensional point cloud dataset to construct a multimodal feature vector group. Three-dimensional visual defect detection is then performed based on the multimodal feature vector group to generate a defect detection quality report. Finally, control decision instructions are formulated according to the defect detection quality report, and the target product is automatically controlled through the control decision instructions. These technical means solve the technical problems of insufficient detection accuracy and comprehensiveness in existing three-dimensional visual inspection for industrial applications, and achieve the technical effect of improving detection accuracy and comprehensiveness.

[0005] The present application provides a three-dimensional visual inspection method for industrial applications, comprising: obtaining a three-dimensional point cloud dataset of a target product, performing multidimensional feature analysis based on the three-dimensional point cloud dataset, and constructing a multimodal feature vector group; performing three-dimensional visual defect detection based on the multimodal feature vector group, and generating a defect detection quality report; formulating control decision instructions according to the defect detection quality report, and automatically controlling the target product through the control decision instructions.

[0006] In a possible implementation, a three-dimensional point cloud dataset of a target product is obtained, a multi-dimensional feature analysis is performed based on the three-dimensional point cloud dataset, a multi-modal feature vector group is constructed, and the following processing is performed: the target product is collaboratively scanned by multiple sensors to obtain a scanning dataset, wherein the scanning dataset includes initial three-dimensional point cloud data and multi-view RGB image data; noise suppression is performed on the initial three-dimensional point cloud data, and point cloud completion is performed on the initial three-dimensional point cloud data based on the suppression result to determine the three-dimensional point cloud data of the target product; three-dimensional geometric features are extracted based on the three-dimensional point cloud data, and surface texture features are extracted based on the multi-view RGB image data; the three-dimensional geometric features are fused with the surface texture features to construct the multi-modal feature vector group.

[0007] In a possible implementation, the three-dimensional geometric features are fused with the surface texture features to construct the multimodal feature vector group, and the following processing is performed: the three-dimensional geometric features and the surface texture features are normalized to generate normalized data; the fusion weights of the three-dimensional geometric features and the surface texture features are dynamically allocated using an attention mechanism to generate a weighted fusion feature vector; joint dimensionality reduction is performed based on the weighted fusion feature vector, the normalized data is matched according to the dimensionality reduction result, and the multimodal feature vector group is constructed.

[0008] In a possible implementation, three-dimensional visual defect detection is performed based on the multimodal feature vector group to generate a defect detection quality report, and the following processing is performed: standardization processing is performed based on the multimodal feature vector group to generate a standard feature matrix; three-dimensional visual defect analysis is performed on the target product according to the standard feature matrix to generate a defect detection result; defect density analysis is performed based on the defect detection result to obtain defect density data and generate a first quality score; defect size analysis is performed based on the defect detection result to obtain size distribution data and generate a second quality score; defect position analysis is performed based on the defect detection result to obtain position offset data and generate a third quality score; defect risk level is defined according to the defect density data, the size distribution data and the position offset data, and quality assessment is performed based on the defect risk level in combination with the first quality score, the second quality score and the third quality score to generate comprehensive quality score data; and the comprehensive quality score data is added to the defect detection quality report.

[0009] In a possible implementation, a three-dimensional visual defect analysis is performed on the target product according to the standard feature matrix to generate a defect detection result, and the following processing is performed: sliding window segmentation is performed on the standard feature matrix to extract local feature sub-blocks, multi-scale feature fusion is performed based on the local feature sub-blocks, and defect candidate areas are generated; boundary screening is performed based on the defect candidate areas to determine the bounding box coordinates, and the bounding box coordinates are reversely projected onto the three-dimensional point cloud dataset for three-dimensional visual analysis to obtain three-dimensional defect information; and the three-dimensional defect information is added to the defect detection result.

[0010] In a possible implementation, control decision instructions are formulated according to the defect detection quality report, the target product is automatically controlled by the control decision instructions, and the following processing is performed: the defect detection quality report is parsed to obtain defect type data, and a control decision tree is constructed according to the defect type data and the defect risk level; the real-time production line status information of the target product is retrieved to traverse the control decision tree, and the initial control instruction set is extracted, and the initial control instruction set is conflict-resolved according to the product constraints to determine the executable control instruction queue; the drive trajectory of the device to be controlled is analyzed according to the executable control instruction queue to generate the control decision instruction.

[0011] In a possible implementation, a control decision tree is constructed based on the defect type data and the defect risk level, and the following processing is performed: based on the mapping of the defect type data and the control action, a defect type-control action mapping rule library is constructed; based on the response of the defect risk level and the control action, a response intensity parameter is constructed; multiple batches of simulated control are performed on the target product to generate execution effects, and a decision tree structure is constructed based on the execution effects; based on the decision tree structure, the defect type-control action mapping rule library is integrated with the response intensity parameter to construct the control decision tree.

[0012] In a possible implementation, the driving trajectory of the device to be controlled is analyzed according to the executable control instruction queue to generate the control decision instruction, and the following processing is performed: parsing the executable control instruction queue to extract the three-dimensional point cloud data of the target device; performing motion analysis based on the three-dimensional point cloud data of the target device to construct a driving trajectory simulation scheme; performing collision detection on the driving trajectory simulation scheme to screen a candidate trajectory set, performing multi-objective optimization based on the candidate trajectory set to generate a target execution trajectory; performing control encoding based on the target execution trajectory to generate the control decision instruction.

[0013] The present application also provides a three-dimensional visual inspection system for industrial applications, including: a multimodal feature vector group construction module, used to obtain a three-dimensional point cloud dataset of a target product, perform multidimensional feature analysis based on the three-dimensional point cloud dataset, and construct a multimodal feature vector group; a defect detection module, used to perform three-dimensional visual defect detection based on the multimodal feature vector group, and generate a defect detection quality report; an automation control module, used to formulate control decision instructions according to the defect detection quality report, and perform automated control of the target product through the control decision instructions.

[0014] The present application also provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing a three-dimensional visual inspection method for industrial applications when executing the executable instructions stored in the memory.

[0015] This application proposes a 3D visual inspection method, system, and device for industrial applications. First, a 3D point cloud dataset of a target product is acquired. Based on this dataset, multidimensional feature analysis is performed to construct a multimodal feature vector group. Three-dimensional visual defect detection is then performed based on this multimodal feature vector group, generating a defect detection quality report. Finally, control decision instructions are formulated based on the defect detection quality report, and the target product is automatically controlled using these control decision instructions. This achieves the technical effect of improving the accuracy and comprehensiveness of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention are briefly introduced below. Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in precise order. Instead, various steps may be processed in reverse order or simultaneously as needed. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0017] Figure 1 A flowchart of a three-dimensional visual inspection method for industrial applications provided in an embodiment of the present application.

[0018] Figure 2 A schematic structural diagram of a three-dimensional visual inspection system for industrial applications provided in an embodiment of the present application.

[0019] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0020] Explanation of the reference numerals: multimodal feature vector group construction module 10 , defect detection module 20 , automation control module 30 , input device 301 , processor 302 , memory 303 , output device 304 . DETAILED DESCRIPTION

[0021] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0022] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0023] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict, and the terms “first\second” involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. The terms “including” and “having” and any variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or modules that are not clearly listed or that are inherent to these processes, methods, products or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used herein are for the purpose of describing the embodiments of this application only.

[0024] The present invention provides a three-dimensional visual inspection method for industrial applications. Figure 1 As shown, the method includes: Step S100: obtaining a three-dimensional point cloud dataset of a target product, performing multi-dimensional feature analysis based on the three-dimensional point cloud dataset, and constructing a multi-modal feature vector group.

[0025] Specifically, 3D point cloud data of the target product is acquired using devices such as laser scanners, structured light scanners, or binocular stereo cameras. These sensors can capture 3D information about the object's surface with high precision and resolution. Each point in the 3D point cloud data contains 3D coordinates (X, Y, Z) that describe the object's surface shape.

[0026] The data collection process involves placing the target product on the scanning platform, ensuring it is within the sensor's scanning range. The sensor is activated and scanned in all directions according to the preset scanning path and parameters. The raw data output by the sensor is a point cloud, with each point containing 3D coordinates. Deep learning algorithms (such as denoising models based on convolutional neural networks) are used to remove noise from the point cloud data. For example, the PointNet++ architecture can be used to analyze point-by-point point cloud data to identify and remove noise points. The point cloud data is then processed using data augmentation techniques (such as rotation, scaling, and translation) to increase data diversity and robustness. For example, the point cloud data can be randomly rotated by a certain angle to simulate different scanning perspectives. After noise removal and data augmentation, a preprocessed point cloud dataset is generated for subsequent analysis.

[0027] Geometric features are extracted from the point cloud data in the preprocessed point cloud dataset using geometric analysis algorithms (such as curvature calculation, normal estimation, and shape factors). For example, by calculating the local curvature of each point, sharp edges and smooth areas on the object surface can be identified. Surface texture features are extracted from the point cloud data using texture analysis algorithms (such as gray-level co-occurrence matrix and wavelet transform). For example, the roughness and directionality of the surface texture can be quantified by calculating the eigenvalues of the gray-level co-occurrence matrix. The extracted geometric and texture features are combined into a multimodal feature vector for subsequent defect detection. For example, a feature vector may contain multiple feature dimensions, such as the point's 3D coordinates, curvature, normal direction, and texture roughness.

[0028] In one possible implementation, a 3D point cloud dataset of a target product is obtained, multi-dimensional feature analysis is performed based on the 3D point cloud dataset, and a multimodal feature vector group is constructed. Step S100 further includes step S110, in which a multi-sensor collaborative scan is performed on the target product to obtain a scan dataset, which includes initial 3D point cloud data and multi-view RGB image data. Specifically, the multi-sensor configuration includes a laser displacement sensor array (for high-precision 3D point cloud data acquisition), a high-resolution area array camera (for acquiring multi-view RGB image data), and a structured light projection device (for projecting a dynamic coded grating onto the surface of the target product to assist the laser displacement sensor in acquiring more accurate 3D information). The sensors and projection device are arranged in a surround configuration, covering the target product's inspection area to ensure all-round scanning. The structured light projection device projects a dynamic coded grating onto the surface of the target product, synchronously triggering the laser displacement sensor and area array camera to perform scanning. The laser displacement sensor acquires initial 3D point cloud data, and the area array camera acquires RGB image data from the corresponding viewpoints. Based on the motion trajectory of the target product and the coordinates of the inspection station, the time synchronization signal is used to align the data collected by multiple sensors to generate spatiotemporally consistent original 3D point cloud data and multi-view RGB image datasets.

[0029] For example, 12 laser displacement sensors and eight high-resolution area array cameras are deployed in a surround layout, covering the inspection area of the target product. A structured light projection device projects a dynamically coded raster, synchronously triggering the sensors and cameras to scan. Assuming the target product is an automotive part moving at a constant speed on a conveyor belt, time synchronization signals ensure that the sensors and cameras collect data at the same time, generating a temporally and spatially consistent dataset.

[0030] Step S120 performs noise suppression on the initial 3D point cloud data and, based on the suppression results, completes the initial 3D point cloud data to determine the 3D point cloud data of the target product. Specifically, noise suppression is performed on the initial 3D point cloud data using a statistical filtering algorithm (such as voxel filtering or Gaussian filtering). For example, a voxel filtering algorithm can be used to divide the point cloud data into a grid of small voxels, and the points within each voxel are averaged to remove noise points. Alternatively, a deep learning model (such as PointNet++ or U-Net architecture) can be used to remove noise from the point cloud data. For example, a PointNet++ model can be trained, taking noisy point cloud data as input and outputting denoised point cloud data.

[0031] Completing the denoised point cloud data can be done, for example, by filling in missing points using nearest neighbor interpolation or surface fitting methods. For example, in areas with a large number of missing points, local surface fitting methods can be used to reconstruct the surface. Alternatively, point clouds can be completed using generative adversarial networks (GANs) or autoencoders. For example, a point cloud generative adversarial network can be trained and fed with incomplete point cloud data to generate complete point cloud data.

[0032] For example, a voxel filtering algorithm is used to suppress noise in the initial 3D point cloud data. The point cloud data is divided into a 1cm³ voxel grid, and the points within each voxel are averaged to remove noise points. Then, an interpolation method based on surface fitting is used to complete the denoised point cloud data, filling in missing points and ultimately determining the 3D point cloud data for the target product.

[0033] Step S130 extracts 3D geometric features based on the 3D point cloud data, and extracts surface texture features based on the multi-view RGB image data. Specifically, the process of extracting 3D geometric features can be divided into three subsets. The first subset includes the point cloud's curvature, normal vector distribution, and local surface roughness. The second subset describes the local geometric structure of the target product. The third subset describes the overall shape characteristics of the target product. Curvature is a geometric quantity that describes the degree of curvature of an object's surface. In a 3D point cloud, curvature is used to identify sharp edges and smooth areas on the surface. The curvature of each point is calculated by fitting a local surface (such as a quadratic surface). For example, for each point in the point cloud, several points in its neighborhood are selected, a quadratic surface is fitted, and the curvature value of the point is calculated using the principal curvature of the surface. The normal vector is a vector perpendicular to the surface and is used to describe the directionality of the surface, such as determining surface concavity or texture direction. The normal vector of each point is calculated by fitting a local plane or using principal component analysis (PCA). For example, for each point in a point cloud, several points within its neighborhood are selected, the covariance matrix of these points is calculated, and then the principal direction is obtained through PCA, which is the normal vector of the point. Local surface roughness is a geometric quantity that describes the degree of microscopic unevenness of an object's surface and reflects the microscopic texture characteristics of the surface. High-frequency noise in the point cloud is removed through Gaussian filtering, and the difference between the filtered point cloud and the original point cloud is calculated as a measure of roughness. Alternatively, the variance of the point cloud can be calculated within the neighborhood of each point. The larger the variance, the rougher the surface. Alternatively, the gradient of the point cloud can be calculated, and the mean or standard deviation of the gradient can be used as a measure of roughness.

[0034] Multi-view RGB image data is mapped to a high-precision 3D point cloud dataset to generate texture mapping coordinates. For example, a texture mapping algorithm is used to map RGB image data to point cloud data, generating a point cloud model with texture information. Shallow texture feature extraction is performed, using the Local Binary Pattern (LBP) algorithm and the Gray Level Co-occurrence Matrix (GLCM) to analyze the local contrast and directional characteristics of the texture map. For example, the LBP algorithm is used to extract local pattern features of the texture, and the GLCM is used to analyze the directionality and contrast of the texture. Deep texture feature extraction is performed, using a pre-trained residual network (ResNet) to extract deep texture semantic features and fuse them with shallow texture features to generate a surface texture feature vector. For example, a pre-trained ResNet model is used to extract deep semantic features of the texture, and then the deep features are fused with the shallow features to form a complete texture feature vector.

[0035] Step S140: Fuse the three-dimensional geometric features with the surface texture features to construct the multimodal feature vector group. Specifically, weights are assigned to the different features based on their importance, and weighted fusion is performed. For example, based on experimental results, a weight of 0.6 is assigned to the geometric features and a weight of 0.4 is assigned to the texture features to generate the multimodal feature vector group. Alternatively, a deep learning model (such as a multimodal fusion network) is used to fuse the geometric and texture features. For example, a multimodal fusion network is trained, which inputs the geometric and texture features and outputs the fused multimodal feature vector group.

[0036] This implementation method can better capture the detailed features of the target product by extracting multi-dimensional geometric features and surface texture features, thereby improving the accuracy and reliability of defect detection.

[0037] In one possible implementation, the three-dimensional geometric features are fused with the surface texture features to construct the multimodal feature vector group, and step S140 further includes step S141, normalizing the three-dimensional geometric features and the surface texture features to generate normalized data. Specifically, the three-dimensional geometric features and the surface texture features have different dimensions and numerical ranges. In order to make these features comparable in subsequent processing, they are normalized. Normalization scales the eigenvalues to a uniform range (such as [0, 1] or [-1, 1]) to avoid certain features from dominating in terms of numerical value. Among them, the normalization method can select minimum-maximum normalization or Z-score normalization. Minimum-maximum normalization scales the eigenvalues to the range of [0, 1], and Z-score normalization scales the eigenvalues to a distribution with a mean of 0 and a standard deviation of 1.

[0038] In step S142, an attention mechanism is used to dynamically assign fusion weights between the three-dimensional geometric features and the surface texture features to generate a weighted fused feature vector. Specifically, different features have different importance in different scenarios. The attention mechanism dynamically assigns weights to each feature to better utilize feature information. A neural network (such as a Transformer or its variants) is used to learn the importance of features. For example, a simple attention module is used to calculate the weight of each feature. Based on the calculated weights, a weighted sum is performed on the normalized features to generate a weighted fused feature vector. For example, a simple attention module is used to input the normalized three-dimensional geometric features and surface texture features. The neural network calculates the attention score for each feature, and then dynamically assigns weights based on these scores. Assuming the calculated weights are α1 = 0.6 (geometric features) and α2 = 0.4 (texture features), the normalized features are weighted summed to generate a weighted fused feature vector.

[0039] Step S143 performs joint dimensionality reduction on the weighted fused feature vectors. The normalized data is matched based on the dimensionality reduction results to construct the multimodal feature vector set. Specifically, dimensionality reduction techniques are used to reduce the feature dimensionality while retaining the most important information, thereby improving computational efficiency and reducing the risk of overfitting. Dimensionality reduction employs principal component analysis (PCA) and linear discriminant analysis (LDA). Features are projected onto principal component directions through linear transformation, retaining directions with the highest variance. For example, PCA is performed on the weighted fused feature vectors to extract the top k principal components. Simultaneously, dimensionality reduction aims to maximize inter-class distances and minimize intra-class distances. For example, LDA is performed on the weighted fused feature vectors to extract the most discriminative features. Based on the dimensionality reduction results, the normalized data is matched to construct the multimodal feature vector set. For example, PCA is performed on the weighted fused feature vectors to extract the top 10 principal components. LDA is then performed on these principal components to further extract the most discriminative features. Finally, based on the dimensionality reduction results, the normalized data is matched to construct the multimodal feature vector set.

[0040] Step S200: Perform three-dimensional visual defect detection based on the multimodal feature vector group and generate a defect detection quality report.

[0041] Specifically, a convolutional neural network (CNN) or point cloud processing network (such as PointNet or PointCNN) is used to analyze multimodal feature vectors and identify defective areas. For example, a PointNet model is trained, inputting the multimodal feature vectors of point cloud data and outputting a classification result indicating whether each point is a defect. Alternatively, the detection results are classified based on preset thresholds to distinguish between normal and defective areas. For example, if the curvature of a point exceeds a set threshold and the texture roughness is abnormal, it is determined to be a defect point. Features such as the size, shape, and location of the detected defect areas are extracted, and detailed defect description information is generated, including the location, type (such as scratches, cracks, holes, etc.), severity (such as defect area and depth, etc.), and overall quality score. The defect detection results are displayed in the form of tables or visual charts to guide subsequent control decisions. For example, a color-coded 3D model is used to mark the defect locations, with red indicating severe defects and yellow indicating minor defects.

[0042] In one possible implementation, 3D visual defect detection is performed based on the multimodal feature vector group to generate a defect detection quality report. Step S200 further includes step S210, where a normalization process is performed based on the multimodal feature vector group to generate a standard feature matrix. Specifically, each feature in the multimodal feature vector group is normalized to generate a standard feature matrix. For example, the mean and standard deviation of each feature in the multimodal feature vector group (such as curvature, normal vector, texture feature, etc.) are calculated. Then, the Z-score normalization method is used to convert each feature value into a distribution with a mean of 0 and a standard deviation of 1 to generate a standard feature matrix.

[0043] In step S220, a three-dimensional visual defect analysis of the target product is performed according to the standard feature matrix to generate defect detection results. Specifically, a convolutional neural network (CNN) or a point cloud processing network (such as PointNet) is used to analyze the standard feature matrix and identify defect areas. For example, a PointNet model is trained, inputting the standard feature matrix and outputting a classification result indicating whether each point is a defect. Alternatively, the detection results are classified according to a preset threshold to distinguish between normal and defective areas. For example, if the feature value of a point exceeds a set threshold, it is determined to be a defect point. The defect detection results include the location and type of the defect (such as scratches, cracks, holes, etc.), as well as the severity of the defect.

[0044] For example, a trained PointNet model can be used to analyze a standard feature matrix to detect scratches and cracks on the surface of a target product. The resulting defect detection results include the defect's location coordinates, type label (such as "scratch" or "crack"), and defect severity (such as "minor" or "severe").

[0045] Step S230: Perform defect density analysis based on the defect detection results to obtain defect density data and generate a first quality score. Specifically, the number of defects per unit area is calculated as a measure of defect density. For example, the target product surface is divided into several small areas, the number of defects in each area is counted, and then the number of defects per unit area is calculated. The first quality score is generated based on the defect density; for example, the lower the defect density, the higher the quality score.

[0046] For example, the surface of the target product is divided into small areas of 1 cm², and the number of defects in each area is counted. If the defect density in a certain area is 0.5 defects per square centimeter, the first quality score generated is 80 points (out of 100 points) according to the preset scoring rules.

[0047] Step S240 performs defect size analysis based on the defect detection results, obtains size distribution data, and generates a second quality score. Specifically, the dimensions of each defect (e.g., length, width, depth, etc.) are measured and the size distribution is statistically analyzed. For example, the defect size is calculated using the three-dimensional coordinates in the point cloud data. The second quality score is generated based on the defect size distribution. For example, if the defect size distribution deviates from the preset process tolerance range, it is marked as a process abnormality defect and the quality score is lowered.

[0048] For example, the length and depth of scratches on the surface of the target product are measured and the distribution of the scratch length is analyzed. If the scratch length distribution deviates from the preset process tolerance range (for example, the maximum length does not exceed 2mm), it is marked as a process abnormality defect and a secondary quality score of 70 points (out of a maximum of 100 points) is generated.

[0049] Step S250 analyzes the defect location based on the defect detection results, obtains position offset data, and generates a third quality score. Specifically, the defect location is compared with the product assembly coordinate system, and the offset of the defect location is calculated. For example, the coordinate information in the point cloud data is used to calculate the deviation between the defect location and the designed assembly position. The third quality score is generated based on the defect location offset. For example, if the defect location offset exceeds a preset threshold and affects a critical functional area, the quality score is lowered.

[0050] For example, the defect location on the target product surface is compared with the assembly coordinate system to calculate the offset of the defect location. If the position offset of a defect is 1mm and it is located in a critical functional area (such as the sealing surface of the engine block), the third quality score generated according to the preset scoring rules is 60 points (out of a maximum of 100 points).

[0051] In step S260, a defect risk level is determined based on the defect density data, the size distribution data, and the position offset data. A quality assessment is performed based on the defect risk level in combination with the first, second, and third quality scores to generate comprehensive quality score data. Specifically, the comprehensive quality score data is generated based on the defect density data, size distribution data, and position offset data, combined with the first, second, and third quality scores. For example, the comprehensive quality score is calculated using a weighted summation method: comprehensive quality score = w1 × first quality score + w2 × second quality score + w3 × third quality score, where w1, w2, and w3 are weight coefficients that can be adjusted based on actual needs. The defect risk level is determined based on the comprehensive quality score. For example, the comprehensive quality score can be divided into three levels: high risk (0-40 points), medium risk (41-70 points), and low risk (71-100 points).

[0052] For example, assuming the first quality score is 80, the second quality score is 70, and the third quality score is 60. Based on the preset weight coefficients (e.g., w1=0.4, w2=0.3, w3=0.3), the comprehensive quality score is calculated as: Comprehensive quality score = 0.4×80+0.3×70+0.3×60=71. Based on the comprehensive quality score, the defect risk level of the target product is classified as low risk.

[0053] Step S270: Add the comprehensive quality score data to the defect detection quality report. Specifically, the defect detection quality report includes the defect location, type, size, density, position offset, risk level, and comprehensive quality score. The report format can be a table or a visual chart to facilitate a quick understanding of product quality status. For example, the generated defect detection quality report is shown in Table 1.

[0054] Table 1: Example of defect detection quality report

[0055] This approach combines defect density, size distribution, and position offset to generate a comprehensive quality score, providing a comprehensive assessment of product quality. Defect risk levels are assigned based on this comprehensive quality score, providing clear guidance for subsequent handling.

[0056] In one possible implementation, a 3D visual defect analysis of a target product is performed according to the standard feature matrix to generate defect detection results. Step S220 further includes step S221, wherein the standard feature matrix is segmented using a sliding window to extract local feature sub-blocks. Multi-scale feature fusion is performed on the local feature sub-blocks to generate defect candidate regions. Specifically, the standard feature matrix is segmented using a sliding window to extract local feature sub-blocks, which are used to capture feature information in local regions and identify small-scale defects. A fixed-size window (e.g., 3×3, 5×5, etc.) is selected and slid across the standard feature matrix, moving a fixed step size (e.g., 1 pixel) at a time, to extract local feature sub-blocks within the window. Feature sub-blocks of different scales can capture defects of different sizes. Multi-scale feature fusion enables more comprehensive defect identification. Features are extracted for each local feature sub-block, such as calculating the mean, variance, and gradient of the local sub-block. The features of the feature sub-blocks of different scales are fused to generate a comprehensive feature vector. For example, fusion is performed using weighted summation or a deep learning model (e.g., a multi-scale feature fusion network). Based on the fused feature vectors, candidate defect regions are generated. For example, regions that may contain defects are identified through threshold judgment or classifiers (such as support vector machines or deep learning classifiers).

[0057] For example, the standard feature matrix is segmented by sliding windows with window sizes of 3×3 and 5×5 and a step size of 1. The mean and variance of each local feature sub-block are extracted, and then the features of different scales are fused through a multi-scale feature fusion network to finally generate defect candidate regions, which are marked as areas that may contain defects.

[0058] In step S222, boundary screening is performed based on the defect candidate area to determine bounding box coordinates. These bounding box coordinates are then back-projected onto the 3D point cloud dataset for 3D visual analysis to obtain 3D defect information. Specifically, boundary screening is used to determine the bounding box coordinates of the defect to more accurately locate the defect. Boundary detection is performed on the defect candidate area using an edge detection algorithm (such as Canny edge detection or Sobel filtering). Based on the detected boundaries, bounding box coordinates are generated. For example, the minimum bounding box's bounding box coordinates (x_min, y_min, x_max, y_max) are calculated. The bounding box coordinates are then back-projected onto the 3D point cloud dataset to obtain 3D defect information, such as the actual size and depth. Specifically, the 2D bounding box coordinates are mapped onto the 3D point cloud data, the corresponding 3D point cloud area is found, and the actual size (such as length and width) and depth of the defect in 3D space are calculated. For example, the actual size of the defect is obtained by calculating the point-to-point distance in the point cloud data.

[0059] For example, we use the Canny edge detection algorithm to detect the boundaries of defect candidate areas and generate bounding box coordinates. We then back-project the bounding box coordinates onto a 3D point cloud dataset to calculate the actual size and depth of the defect. Assume the detected defect is 2 mm long, 1 mm wide, and 0.5 mm deep.

[0060] Step S223: Add the 3D defect information to the defect detection results. Specifically, add the 3D defect information (such as actual size and depth) to the defect detection results to generate a complete defect detection report. A data structure for the defect detection results is designed to include information such as the defect's location, type, 2D and 3D dimensions, and depth. The calculated 3D defect information is then populated into the defect detection results.

[0061] This implementation uses sliding window segmentation to capture local feature information, helping to identify small-scale defects. Multi-scale feature fusion enables more comprehensive identification of defects of varying sizes, improving the accuracy and robustness of defect detection. Boundary screening allows for precise location of the defect's bounding box coordinates. Backprojection and 3D visual analysis allow for the acquisition of the defect's actual size and depth, providing more comprehensive data support for quality assessment.

[0062] Step S300: formulating control decision instructions according to the defect detection quality report, and automatically controlling the target product through the control decision instructions.

[0063] Specifically, based on the defect type and severity in the defect inspection quality report, corresponding control decision instructions are formulated to control the automated equipment to process the target product. For example, if a severe defect is detected, a command is issued to the robotic arm to move the product to the defective product area. If a minor defect is detected, process parameters are adjusted to reduce the defect rate of subsequent products. Control decisions are dynamically adjusted by combining historical data and real-time feedback. For example, if a certain type of defect occurs frequently, the system can automatically adjust production parameters to optimize the production process. Instructions are sent through industrial robot control systems (such as ABB and KUKA) to control the robotic arm to sort or process the target product. For example, the robotic arm, following the control instructions, grabs defective products and places them in a designated location; automated sorting equipment (such as conveyor belts and chutes) is controlled to sort and process the products. For example, products are sorted into different channels based on the defect detection results. Instructions are sent through communication interfaces with production equipment (such as PLCs and industrial Ethernet) to adjust production process parameters. For example, the temperature or pressure of the injection molding machine can be adjusted to reduce the defect rate.

[0064] In one possible implementation, control decision instructions are formulated according to the defect detection quality report, and the target product is automatically controlled by the control decision instructions. Step S300 further includes step S310, which parses the defect detection quality report to obtain defect type data, and constructs a control decision tree based on the defect type data and the defect risk level. Specifically, the data structure in the defect detection quality report is parsed to extract defect type data (such as scratches, cracks, holes, etc.) and defect risk levels (such as high risk, medium risk, and low risk). A control decision tree is designed, with different defect types and risk levels as decision nodes. Specific control rules are set for each node. For example, high-risk defects are directly marked as unqualified products; medium-risk defects are further inspected or repaired; and low-risk defects are recorded and released.

[0065] For example, suppose a defect inspection quality report contains the following information: Defect Type: Scratches; Risk Level: Medium; Overall Quality Score: 65. Based on this information, the corresponding node in the control decision tree is found and the initial control strategy is determined: adjust process parameters and retest.

[0066] In step S320, the control decision tree is traversed by retrieving the real-time production line status information of the target product, extracting an initial set of control instructions, and performing conflict resolution on this set based on the product constraints to determine an executable control instruction queue. Specifically, the current status information of the target product on the production line, including its location, status, and executed operations, is obtained through an industrial automation system (such as a PLC or MES system). Based on the rules of the decision tree and the real-time status information, a series of control instructions are generated. For example, if the decision tree indicates that process parameters need to be adjusted, an instruction to adjust the parameters is generated. The instruction set is then checked for conflicts. For example, two instructions may require the same device to perform different operations at the same time. Conflicts are resolved through prioritization or rescheduling. For example, higher-priority instructions are executed first, or the execution order of instructions is adjusted to ensure conflict-free control instructions and safe and efficient execution. For example, if two instructions require the same robot arm to grasp different products at the same time, the instruction order is adjusted to execute the higher-priority instruction first.

[0067] In step S330, the drive trajectory of the device to be controlled is analyzed according to the executable control instruction queue to generate the control decision instruction. Specifically, the motion trajectory of the device to be controlled (such as a robotic arm, conveyor belt, etc.) is planned based on the executable control instruction queue. For example, the grasping path of the robotic arm and the movement speed of the conveyor belt are planned. The motion trajectory is checked to see if it will cause collision or interference between devices. For example, the path of the robotic arm to grasp the product from the inspection station and move to the repair station is planned, and the path is checked to see if it will collide with other equipment on the production line. Based on the motion planning and control logic, specific control instructions are generated and sent to the automation equipment. For example, the motion instructions of the robotic arm and the speed instructions of the conveyor belt are generated, and the control instructions are sent to the automation equipment via industrial communication protocols (such as Modbus, Profibus, etc.).

[0068] This implementation develops precise control strategies based on defect type and risk level, ensuring rational and effective defect handling. Integrating real-time production line status information, it generates a control instruction set that matches the current production status. Conflict resolution ensures conflict-free control instructions, avoiding collisions or interference between devices and improving production safety. Drive trajectory analysis ensures efficient and safe execution of control instructions, improving production efficiency. Planned motion trajectories and control logic are converted into specific control instructions, enabling precise control of automated equipment.

[0069] In one possible implementation, a control decision tree is constructed based on the defect type data and the defect risk level. Step 310 further includes step S311, where a defect type-control action mapping rule base is constructed based on the mapping of the defect type data to control actions. Specifically, all defect types are classified, such as scratches, cracks, and holes. Control actions corresponding to each defect type are defined, including physical actions (such as robotic arm grasping and conveyor movement) and non-mechanical actions (such as issuing instructions and adjusting parameters). Mapping defect types to corresponding control actions creates a rule base for quickly finding and executing control actions.

[0070] For example, if the defect type is "scratch," the corresponding control actions include: recording the defect information in a database; adjusting the injection molding machine's process parameters (such as temperature and pressure); and re-sending the product to the inspection station for a second inspection. These mappings are stored in a defect type-control action mapping rule library.

[0071] Step S312: Response intensity parameters are constructed based on the defect risk level and control action response. Specifically, defect risk levels are categorized as high, medium, and low. Response intensity parameters are defined for each risk level, such as sorting speed, robotic arm gripping force, and parameter adjustment amplitude. Based on the defect risk level, the response intensity of the control action is defined to ensure that the force or speed of the control action matches the severity of the defect.

[0072] For example, assuming the defect risk level is "medium risk," the corresponding response intensity parameters include: sorting speed: medium speed (e.g., 10 pieces per minute); robot arm gripping force: moderate (e.g., 50% of maximum force); parameter adjustment range: moderate (e.g., temperature adjustment ±5°C). These response intensity parameters are stored in a response intensity parameter library.

[0073] In step S313, multiple batches of simulated control are performed on the target product to generate execution results. A decision tree structure is then constructed based on these results. Specifically, a virtual production environment is built to simulate the operation of the production line. Multiple batches of simulated control are performed on the target product, and the execution results of each simulation are recorded. Based on the simulation results, the effectiveness of the control strategy is evaluated, such as the accuracy of defect handling and production efficiency. Based on the simulation results, the weights of each branch in the decision tree are adjusted to make the decision tree more inclined to select the more effective control strategy. Based on the simulation results, the decision tree structure is dynamically adjusted, inserting or deleting certain nodes to optimize the decision path.

[0074] For example, suppose a multi-batch simulation reveals that adjusting process parameters before retesting is more effective for scratch defects. Based on these simulation results, the decision tree structure is adjusted to include a branch that prioritizes adjusting process parameters before retesting. This optimized decision tree structure enables more effective scratch defect handling, improving production efficiency and quality.

[0075] Step S314 integrates the defect type-control action mapping rule base with the response strength parameter based on the decision tree structure to construct the control decision tree. Specifically, the rules in the defect type-control action mapping rule base are embedded into each node of the decision tree. The response strength parameter is embedded into each branch of the decision tree to ensure that the execution strength of the control action matches the defect risk level. Dynamic constraint nodes are inserted into the decision tree, taking into account production plan priorities and equipment load data to ensure that control instructions match the production cycle, thus forming a complete control decision tree.

[0076] For example, suppose a decision tree for "scratch" defects includes embedded rules for recording defect information, adjusting process parameters, and retesting. For "medium-risk" defects, response intensity parameters are embedded, including moderate sorting speed, moderate robotic gripping force, and moderate parameter adjustment amplitude. Dynamic constraint nodes are inserted into the decision tree to ensure that control instructions align with the production cycle, avoiding equipment overload or production stagnation. The resulting control decision tree dynamically generates control instructions based on defect type and risk level, ensuring an efficient and stable production process.

[0077] This implementation method, through a mapping rule base, can quickly find and execute control actions corresponding to defect types, thereby improving production efficiency. The rule base covers all possible defect types and control actions, ensuring that all defects can be effectively handled. Response intensity parameters are defined based on the defect risk level to ensure that the force or speed of the control action matches the severity of the defect, thereby improving control accuracy. The effectiveness of the control strategy was verified through multi-batch simulation control to ensure that the structure and weights of the decision tree can adapt to actual production needs. The defect type-control action mapping rule base and response intensity parameters are integrated into the decision tree to form a complete control decision tree, ensuring that the generation of control instructions is more scientific and reasonable. Dynamic constraint nodes are inserted into the decision tree, taking into account production plan priorities and equipment load data, ensuring that control instructions match the production rhythm and avoiding equipment overload or production stagnation. The optimized control decision tree can dynamically generate control instructions to ensure the efficiency and stability of the production process.

[0078] In one possible implementation, the drive trajectory of the target device is analyzed according to the executable control instruction queue to generate the control decision instruction. Step S330 further includes step S331, parsing the executable control instruction queue to extract 3D point cloud data of the target device. Specifically, the executable control instruction queue is parsed to extract the target device, motion type, target pose, and motion constraints for each instruction. The extracted data is organized into a structured form to facilitate subsequent motion analysis.

[0079] For example, suppose the executable control instruction sequence is: a robotic arm grabs a product from the inspection station; the robotic arm moves to the repair station; and a conveyor adjusts its speed. After parsing, the extracted information is: target device: robotic arm; motion type: grab, move; target pose: from the inspection station to the repair station; and constraints: collision avoidance and speed limit.

[0080] Step S332 performs motion analysis based on the 3D point cloud data of the target device and constructs a drive trajectory simulation plan. Specifically, the 3D point cloud data of the target device is processed to extract the device's geometry and range of motion. Based on the extracted geometry and range of motion, the device's motion path is planned. For example, a path planning algorithm (such as the A* algorithm or the RRT algorithm) is used to generate a path from the starting point to the end point. The planned path is then converted into a simulation plan to simulate the device's motion.

[0081] Step S333: Collision detection is performed on the drive trajectory simulation plan to screen a set of candidate trajectories. Multi-objective optimization is then performed based on the candidate trajectories to generate a target execution trajectory. Specifically, a collision detection algorithm (such as a BVH tree or GJK algorithm) is used to detect collision points on the path and determine whether the planned motion path will cause collisions or interference between devices. A set of candidate trajectories that are conflict-free and meet kinematic constraints is selected. A multi-objective optimization algorithm (such as the NSGA-II algorithm) is then used to optimize the candidate trajectories, considering multiple optimization objectives, including path length, motion time, and energy consumption. The target execution trajectory is generated based on the optimization results.

[0082] For example, assuming there are multiple trajectories in the candidate trajectory set, a BVH tree is used to detect the collision points of each trajectory and screen out the candidate trajectory set that is conflict-free and satisfies the kinematic constraints. The NSGA-II algorithm is then used to optimize the candidate trajectory set, taking into account path length and motion time, to generate the optimal target execution trajectory.

[0083] Step S334 performs control encoding based on the target execution trajectory to generate the control decision instructions. Specifically, each point in the target execution trajectory is converted into a specific control instruction, including position, velocity, acceleration, etc. The control instructions are generated based on the trajectory encoding to ensure that the device moves according to the planned trajectory. The control instructions are then sent to the target device via an industrial communication protocol (such as Modbus or Profibus).

[0084] This implementation simulates the movement of devices through a simulation solution, detecting whether the planned motion path will cause collisions or interference between devices, ensuring the safety of the motion path. The optimal trajectory is selected from a set of candidate trajectories, improving motion efficiency.

[0085] The embodiment of the present application adopts the method of obtaining a three-dimensional point cloud dataset of the target product, performing multidimensional feature analysis based on the three-dimensional point cloud dataset, constructing a multimodal feature vector group, and then performing three-dimensional visual defect detection based on the multimodal feature vector group, generating a defect detection quality report, and finally formulating control decision instructions according to the defect detection quality report, and automatically controlling the target product through the control decision instructions. These technical means solve the technical problems of insufficient detection accuracy and comprehensiveness in existing three-dimensional visual inspection for industrial applications, and achieve the technical effect of improving detection accuracy and comprehensiveness.

[0086] In the above, refer to Figure 1 A three-dimensional visual inspection method for industrial applications according to an embodiment of the present invention is described in detail. Figure 2 A three-dimensional visual inspection system for industrial applications according to an embodiment of the present invention is described.

[0087] According to an embodiment of the present invention, a 3D visual inspection system for industrial applications is designed to address the technical issues of insufficient accuracy and comprehensiveness in existing 3D visual inspection systems for industrial applications, thereby achieving the technical effect of improving detection accuracy and comprehensiveness. The 3D visual inspection system for industrial applications includes a multimodal feature vector group construction module 10, a defect detection module 20, and an automation control module 30.

[0088] A multimodal feature vector group construction module 10 is used to obtain a three-dimensional point cloud data set of a target product, perform multidimensional feature analysis based on the three-dimensional point cloud data set, and construct a multimodal feature vector group; a defect detection module 20 is used to perform three-dimensional visual defect detection based on the multimodal feature vector group and generate a defect detection quality report; an automation control module 30 is used to formulate control decision instructions according to the defect detection quality report and perform automated control of the target product through the control decision instructions.

[0089] The specific configuration of the multimodal feature vector group construction module 10 will be described in detail below. As described above, a three-dimensional point cloud dataset of the target product is obtained, and a multi-dimensional feature analysis is performed based on the three-dimensional point cloud dataset to construct a multimodal feature vector group. The multimodal feature vector group construction module 10 may further include: a collaborative scanning unit for collaboratively scanning the target product through multiple sensors to obtain a scanning dataset, wherein the scanning dataset includes initial three-dimensional point cloud data and multi-view RGB image data; a point cloud completion unit for performing noise suppression on the initial three-dimensional point cloud data, performing point cloud completion on the initial three-dimensional point cloud data based on the suppression result, and determining the three-dimensional point cloud data of the target product; a feature extraction unit for extracting three-dimensional geometric features based on the three-dimensional point cloud data and extracting surface texture features based on the multi-view RGB image data; and a feature fusion unit for fusing the three-dimensional geometric features with the surface texture features to construct the multimodal feature vector group.

[0090] Among them, the three-dimensional geometric features and the surface texture features are fused to construct the multimodal feature vector group. The feature fusion unit may further include: a normalization processing subunit for normalizing the three-dimensional geometric features and the surface texture features to generate normalized data; a weighted fusion subunit for dynamically allocating the fusion weights of the three-dimensional geometric features and the surface texture features using an attention mechanism to generate a weighted fusion feature vector; a joint dimensionality reduction subunit for performing joint dimensionality reduction based on the weighted fusion feature vector, matching the normalized data according to the dimensionality reduction result, and constructing the multimodal feature vector group.

[0091] The specific configuration of the defect detection module 20 will be described in detail below. As described above, the defect detection module 20 performs three-dimensional visual defect detection based on the multimodal feature vector group and generates a defect detection quality report. The defect detection module 20 may further include: a standardization processing unit for performing standardization processing based on the multimodal feature vector group to generate a standard feature matrix; a defect analysis unit for performing three-dimensional visual defect analysis on the target product according to the standard feature matrix to generate a defect detection result; a defect density analysis unit for performing defect density analysis based on the defect detection result to obtain defect density data and generate a first quality score; a defect size analysis unit for performing defect size analysis based on the defect detection result to obtain size distribution data and generate a second quality score; a defect position analysis unit for performing defect position analysis based on the defect detection result to obtain position offset data and generate a third quality score; and a quality assessment unit for determining a defect risk level based on the defect density data, the size distribution data, and the position offset data, performing a quality assessment based on the defect risk level in combination with the first quality score, the second quality score, and the third quality score to generate comprehensive quality score data, and adding the comprehensive quality score data to the defect detection quality report.

[0092] Among them, a three-dimensional visual defect analysis is performed on the target product according to the standard feature matrix to generate a defect detection result. The defect analysis unit may further include: a defect candidate area generation subunit for performing sliding window segmentation on the standard feature matrix, extracting local feature sub-blocks, performing multi-scale feature fusion based on the local feature sub-blocks, and generating a defect candidate area; a three-dimensional defect information acquisition subunit for performing boundary screening based on the defect candidate area, determining the bounding box coordinates, reversely projecting the bounding box coordinates to the three-dimensional point cloud data set for three-dimensional visual analysis, obtaining three-dimensional defect information, and adding the three-dimensional defect information to the defect detection result.

[0093] The specific configuration of the automation control module 30 will be described in detail below. As described above, control decision instructions are formulated according to the defect detection quality report, and the target product is automatically controlled by the control decision instructions. The automation control module 30 may further include: a control decision tree construction unit for parsing based on the defect detection quality report, obtaining defect type data, and constructing a control decision tree based on the defect type data and the defect risk level; an executable control instruction queue determination unit for retrieving the real-time production line status information of the target product to traverse the control decision tree, extract the initial control instruction set, perform conflict resolution on the initial control instruction set according to the product constraints, and determine the executable control instruction queue; a control decision instruction generation unit for performing drive trajectory analysis on the controlled device according to the executable control instruction queue to generate the control decision instruction.

[0094] Among them, a control decision tree is constructed according to the defect type data and the defect risk level, and the control decision tree construction unit may further include: a mapping subunit for mapping the defect type data and the control action based on the defect type data, and constructing a defect type-control action mapping rule library; a response intensity parameter construction subunit for responding based on the defect risk level and the control action, and constructing a response intensity parameter; a decision tree structure construction subunit for performing multi-batch simulation control on the target product, generating an execution effect, and constructing a decision tree structure according to the execution effect; a control decision tree construction subunit for integrating the defect type-control action mapping rule library with the response intensity parameter based on the decision tree structure to construct the control decision tree.

[0095] Among them, the driving trajectory analysis of the device to be controlled is performed according to the executable control instruction queue to generate the control decision instruction. The control decision instruction generation unit may further include: a three-dimensional point cloud data extraction subunit for parsing the executable control instruction queue to extract the three-dimensional point cloud data of the target device; a motion analysis subunit for performing motion analysis based on the three-dimensional point cloud data of the target device to construct a driving trajectory simulation scheme; a target execution trajectory generation subunit for performing collision detection on the driving trajectory simulation scheme, screening a candidate trajectory set, performing multi-objective optimization based on the candidate trajectory set, and generating a target execution trajectory; a control coding subunit for performing control coding based on the target execution trajectory to generate the control decision instruction.

[0096] A three-dimensional visual inspection system for industrial applications provided by an embodiment of the present invention can execute a three-dimensional visual inspection method for industrial applications provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0097] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules may be used and run on the user terminal and / or server, and the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other and are not used to limit the scope of protection of the present invention.

[0098] Based on the foregoing embodiments, an embodiment of the present application further provides an electronic device. Figure 3 1 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention, showing a block diagram of an exemplary electronic device suitable for implementing an embodiment of the present invention. Figure 3The electronic device shown is merely an example and should not limit the functionality and scope of use of the embodiments of the present invention. The electronic device is implemented as a general-purpose computing device, and its components may include, but are not limited to, an input device 301, a processor 302, a memory 303, and an output device 304. The processor 302 may be one or more; the memory 303 may include a computer-readable medium and at least one program product, which has a set (at least one) of program modules configured to perform the functions of the various embodiments of the present application.

[0099] The memory 303 shown in the embodiment of the present invention may adopt any combination of one or more computer-readable media; the computer-readable storage medium may be, but is not limited to, an infrared, semiconductor system, device or component, or any combination of the above, for storing software programs, computer executable programs and modules, such as the program instructions / modules corresponding to a three-dimensional visual detection method for industrial applications in an embodiment of the present invention. The processor 302 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 303, thereby realizing the above-mentioned three-dimensional visual detection method for industrial applications.

[0100] The above specific embodiments do not constitute a limitation to the scope of protection of this application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of this application should be included in the scope of protection of this application. In some cases, the actions or steps recorded in this application can be performed in an order different from that in the embodiments and can still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A three-dimensional visual inspection method for industrial applications, characterized in that: The method comprises: Acquire a three-dimensional point cloud dataset of a target product, perform multi-dimensional feature analysis based on the three-dimensional point cloud dataset, and construct a multi-modal feature vector group; Performing three-dimensional visual defect detection based on the multimodal feature vector group and generating a defect detection quality report; A control decision instruction is formulated according to the defect detection quality report, and the target product is automatically controlled by the control decision instruction.

2. A three-dimensional visual inspection method for industrial applications according to claim 1, characterized in that: Acquire a three-dimensional point cloud dataset of a target product, perform multidimensional feature analysis based on the three-dimensional point cloud dataset, and construct a multimodal feature vector group, the method comprising: Collaboratively scan the target product using multiple sensors to obtain a scan data set, wherein the scan data set includes initial three-dimensional point cloud data and multi-view RGB image data; performing noise suppression on the initial three-dimensional point cloud data, and completing the point cloud of the initial three-dimensional point cloud data according to the suppression result to determine the three-dimensional point cloud data of the target product; Extracting three-dimensional geometric features based on the three-dimensional point cloud data, and extracting surface texture features based on the multi-view RGB image data; The three-dimensional geometric features are fused with the surface texture features to construct the multimodal feature vector group.

3. A three-dimensional visual inspection method for industrial applications according to claim 2, characterized in that: The three-dimensional geometric features are fused with the surface texture features to construct the multimodal feature vector group, the method comprising: Normalizing the three-dimensional geometric features and the surface texture features to generate normalized data; Using an attention mechanism to dynamically assign fusion weights of the three-dimensional geometric features and the surface texture features to generate a weighted fusion feature vector; Joint dimensionality reduction is performed based on the weighted fusion feature vectors, and the normalized data is matched according to the dimensionality reduction result to construct the multimodal feature vector group.

4. The three-dimensional visual inspection method for industrial applications according to claim 1, characterized in that: Performing three-dimensional visual defect detection based on the multimodal feature vector group and generating a defect detection quality report, the method comprising: Performing standardization processing based on the multimodal feature vector group to generate a standard feature matrix; Performing a three-dimensional visual defect analysis on the target product according to the standard feature matrix to generate a defect detection result; Perform defect density analysis based on the defect detection results to obtain defect density data and generate a first quality score; performing defect size analysis based on the defect detection results to obtain size distribution data and generate a second quality score; performing defect position analysis based on the defect detection results, obtaining position offset data, and generating a third quality score; Determine a defect risk level according to the defect density data, the size distribution data, and the position offset data, and perform a quality assessment based on the defect risk level in combination with the first quality score, the second quality score, and the third quality score to generate comprehensive quality score data; The comprehensive quality score data is added to the defect detection quality report.

5. The three-dimensional visual inspection method for industrial applications according to claim 4, characterized in that: Performing a three-dimensional visual defect analysis on a target product according to the standard feature matrix to generate a defect detection result, the method comprising: Performing sliding window segmentation on the standard feature matrix to extract local feature sub-blocks, performing multi-scale feature fusion based on the local feature sub-blocks to generate defect candidate regions; Perform boundary screening based on the defect candidate area, determine bounding box coordinates, and reversely project the bounding box coordinates onto the three-dimensional point cloud dataset for three-dimensional visual analysis to obtain three-dimensional defect information; The three-dimensional defect information is added to the defect detection result.

6. The three-dimensional visual inspection method for industrial applications according to claim 4, characterized in that: Formulate a control decision instruction according to the defect detection quality report, and automatically control the target product through the control decision instruction, the method comprising: Analyze the defect detection quality report to obtain defect type data, and construct a control decision tree based on the defect type data and the defect risk level; Retrieving the real-time production line status information of the target product to traverse the control decision tree, extracting the initial control instruction set, resolving conflicts of the initial control instruction set according to product constraints, and determining an executable control instruction queue; The driving trajectory of the device to be controlled is analyzed according to the executable control instruction queue to generate the control decision instruction.

7. A three-dimensional visual inspection method for industrial applications according to claim 6, characterized in that: Constructing a control decision tree based on the defect type data and the defect risk level, the method includes: Based on the mapping of the defect type data and the control action, a defect type-control action mapping rule library is constructed; Responding based on the defect risk level and control action, and constructing a response intensity parameter; Performing multiple batches of simulated control on the target product to generate execution results, and constructing a decision tree structure based on the execution results; The defect type-control action mapping rule base is integrated with the response intensity parameter based on the decision tree structure to construct the control decision tree.

8. The three-dimensional visual inspection method for industrial applications according to claim 6, characterized in that: The method includes: analyzing the driving trajectory of the device to be controlled according to the executable control instruction queue to generate the control decision instruction; Parsing the executable control instruction queue to extract three-dimensional point cloud data of the target device; Perform motion analysis based on the three-dimensional point cloud data of the target device and construct a driving trajectory simulation solution; Performing collision detection on the driving trajectory simulation scheme, screening a candidate trajectory set, performing multi-objective optimization based on the candidate trajectory set, and generating a target execution trajectory; Control encoding is performed based on the target execution trajectory to generate the control decision instruction.

9. A three-dimensional visual inspection system for industrial applications, characterized in that: The system is used to implement the three-dimensional visual inspection method for industrial applications according to any one of claims 1 to 8, and the system includes: A multimodal feature vector group construction module is used to obtain a three-dimensional point cloud dataset of a target product, perform multidimensional feature analysis based on the three-dimensional point cloud dataset, and construct a multimodal feature vector group; a defect detection module, configured to perform three-dimensional visual defect detection based on the multimodal feature vector group and generate a defect detection quality report; The automation control module is used to formulate control decision instructions according to the defect detection quality report, and automatically control the target product through the control decision instructions.

10. An electronic device, characterized in that: The electronic device comprises: a memory for storing executable instructions; The processor is configured to implement the three-dimensional visual inspection method for industrial applications as described in any one of claims 1 to 8 when executing the executable instructions stored in the memory.

Images