A method, device, medium, and product for detecting mis-packaged or missing parts based on digital model feature recognition.

By generating dedicated detection templates and constructing a unified spatial coordinate system, combined with parameter adjustment and contour comparison analysis of image acquisition equipment, the efficiency and accuracy problems of detecting mis-assembly, omission, and over-assembly in industrial production have been solved, achieving efficient and accurate detection of mis-assembly and omission.

CN123071306APending Publication Date: 2026-11-10SHANGHAI POINT SCI & TECH LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611327905.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-28
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing technologies suffer from low efficiency and unstable accuracy in detecting misassembly, omissions, and overassembly in industrial production assembly processes, making it difficult to meet the needs of large-scale automated production. Furthermore, the matching accuracy between the 3D digital model and the actual assembly scenario is insufficient, and the reliability of detection needs to be improved.

Method used

By acquiring a 3D digital model of the assembly scene to be inspected, a dedicated inspection template is generated. A coded grating pattern is projected using a projection device. Combined with parameter adjustments of the image acquisition device, a unified spatial coordinate system is constructed. The optimal theoretical contour information of the parts is extracted and compared with the contour of the actual assembled parts. Inspection is performed by combining category consistency and posture deviation.

Benefits of technology

It enables comprehensive control over all parts that need to be inspected in the assembly process, improves the comprehensiveness and accuracy of inspection, reduces false positives and missed positives, and enhances the reliability and efficiency of inspection, making it suitable for large-scale industrial production.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123071306A_ABST
    Figure CN123071306A_ABST
Patent Text Reader

Abstract

This application discloses a method, device, medium, and product for detecting omissions and errors in assembly based on digital model feature recognition, relating to the fields of machine vision and intelligent inspection technology. The method includes: acquiring a three-dimensional digital model and determining a dedicated detection template corresponding to the group of parts for which omission and error detection is required on the workpiece; projecting an coded grating pattern onto a target area using a projection device and adjusting the parameters of an image acquisition device to acquire a two-dimensional image; transforming each element in the part model group to the coordinate system of the image acquisition device based on a spatial transformation matrix, and extracting the optimal theoretical contour information of the parts under the current pose based on the intrinsic parameter matrix of the image acquisition device; determining the corresponding real assembly part contour information based on the acquired real assembly image of the workpiece and comparing it with the optimal theoretical contour information; and performing omission and error detection based on the contour comparison results and the dedicated detection template. This application can achieve efficient and accurate omission and error detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the fields of machine vision and intelligent inspection technology, and in particular to a method, device, medium and product for detecting mis-assembly and omission based on digital model feature recognition. Background Technology

[0002] In the assembly process of industrial production, misassembly, omission, and overassembly of workpieces are key factors affecting product quality. Traditional inspection methods rely heavily on manual visual inspection, which has problems such as low inspection efficiency, unstable accuracy, and susceptibility to subjective factors, making it difficult to meet the needs of large-scale automated production.

[0003] Currently, some automated inspection technologies use machine vision recognition, but most of them only detect the features of a single part and cannot achieve overall control over all parts that need to be inspected in the assembly scene. In addition, some technologies have not achieved a precise coordinate system consistency between the 3D digital model and the real assembly scene, resulting in insufficient matching accuracy between the theoretical contour and the real contour, which easily leads to false detection and missed detection, and the reliability of inspection needs to be improved.

[0004] Furthermore, when product models change, angles change, or correct samples are unavailable, the inspection benchmarks need to be re-collected and re-maintained. Currently, the processes for contour extraction, deviation calculation, and defect identification are not sufficiently refined, making it difficult to achieve efficient and accurate detection of mis-assembly and omissions. Summary of the Invention

[0005] The purpose of this application is to provide a method, device, medium, and product for detecting errors and omissions based on digital model feature recognition, which can achieve efficient and accurate detection of errors and omissions.

[0006] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a method for detecting mis-packing or missing parts based on digital model feature recognition, including: A three-dimensional digital model of the workpiece to be tested in the assembly scene to be tested is obtained, and a special inspection template is determined for the part model group of the workpiece to be tested that needs to be inspected for errors or omissions. The special inspection template includes the preset standard assembly position coordinates and preset standard posture parameters of each element in the part model group. A coded grating pattern is projected onto the target area using a projection device, and the parameters of the image acquisition device are adjusted to acquire a two-dimensional image. The target area is a region in the assembly scene where target points with known spatial coordinates are arranged. The two-dimensional image contains the target points. Based on the two-dimensional image, the two-dimensional pixel coordinates of each target point are extracted. Based on the three-dimensional coordinates of the target point in the world coordinate system, combined with the two-dimensional pixel coordinates of the image acquisition device and the two-dimensional pixel coordinates of the projection device, the spatial transformation matrix between the three-dimensional digital model and the assembly scene is determined. The two-dimensional pixel coordinates of the projection device are the two-dimensional pixel coordinates of the target point in the projection device plane obtained based on the decoding result of the coded grating pattern. Each element in the part model group is transformed to the coordinate system of the image acquisition device based on the spatial transformation matrix, and the optimal theoretical contour information of the part under the current pose is extracted according to the intrinsic parameter matrix of the image acquisition device. Based on the acquired real assembly image of the workpiece to be tested, the corresponding real assembly part contour information is determined, and compared and analyzed with the optimal theoretical contour information to obtain the contour comparison result. Based on the contour comparison results, the system performs mis-assembly and omission detection according to the dedicated detection template to obtain the detection result information.

[0007] In one embodiment, a three-dimensional digital model of the workpiece to be inspected in the assembly scene is acquired, and a dedicated inspection template is determined for the group of parts modeling the workpiece to be inspected for errors or omissions. Specifically, this includes: A three-dimensional digital model of the workpiece to be tested in the assembly scene is generated using 3D scanning or preset software. The three-dimensional digital model is preprocessed to obtain a standardized three-dimensional digital model; Based on a standardized three-dimensional digital model, a feature extraction algorithm is used to extract the geometric structural features of the workpiece under test, as well as to perform category determination and structural analysis, in order to extract the model group of parts that need to be inspected for errors and omissions. Based on the pose information of the 3D digital model in the world coordinate system, the preset standard assembly position coordinates and preset standard posture parameters of the workpiece to be tested are extracted, and the category information, position coordinates and posture parameters are associated to generate a corresponding special inspection template.

[0008] In one embodiment, an image is acquired by projecting an coded grating pattern onto a target area using a projection device and adjusting the parameters of an image acquisition device to obtain a two-dimensional image, including: Multiple target points are arranged in the assembly scene, and the correspondence between the target points and the world coordinate system is established; the spatial three-dimensional coordinates of the target points are known values ​​in the world coordinate system. The projector is controlled to project an coded grating pattern onto the target area. The parameters of the image acquisition device are adjusted to acquire images of the scene containing the target points, thereby obtaining a two-dimensional image containing the target points.

[0009] In one embodiment, the two-dimensional pixel coordinates of each target point are extracted based on the two-dimensional image, and based on the three-dimensional coordinates of the target points in the world coordinate system, combined with the two-dimensional pixel coordinates of the image acquisition device and the two-dimensional pixel coordinates of the projection device, the spatial transformation matrix between the three-dimensional digital model and the assembly scene is determined, including: The two-dimensional image is preprocessed, and a threshold segmentation algorithm is used for binarization. Additionally, a contour extraction algorithm is used to extract the two-dimensional pixel coordinates of each target point. Based on the correspondence between the three-dimensional coordinates of the target point in the world coordinate system and the two-dimensional pixel coordinates of the image acquisition device, the PnP algorithm is used to solve for the extrinsic parameters of the image acquisition device relative to the world coordinate system; the extrinsic parameters include rotation matrix and translation vector. Based on the two-dimensional pixel coordinates of the projection device, and combined with the geometric constraints between the image acquisition device and the projection device, the spatial mapping relationship between the projection device coordinate system and the world coordinate system is determined. Based on the extrinsic parameters of the image acquisition device relative to the world coordinate system and the spatial mapping relationship between the projection device coordinate system and the world coordinate system, the spatial transformation matrix between the 3D digital model and the assembly scene is determined so as to transform the 3D digital model into the unified world coordinate system corresponding to the assembly scene.

[0010] In one embodiment, each element in the part model group is transformed to the coordinate system of the image acquisition device based on the spatial transformation matrix, and the optimal theoretical contour information of the part under the current pose is extracted according to the intrinsic parameter matrix of the image acquisition device, including: Based on the spatial transformation matrix, the part model group is transformed into the coordinate system of the image acquisition device, and combined with the intrinsic parameter matrix of the image acquisition device, the three-dimensional part model group is projected into a two-dimensional projected image. The Canny edge detection and contour extraction algorithm is used to extract the optimal theoretical contour information of the part.

[0011] In one embodiment, the contour information of the corresponding actual assembled parts is determined based on the acquired actual assembly image of the workpiece to be tested, and compared and analyzed with the optimal theoretical contour information to obtain the contour comparison result, including: The acquired real assembly image of the workpiece to be tested is converted to grayscale, denoised by Gaussian filtering, and edge enhancement is performed to obtain the preprocessed real assembly image. The Canny edge detection algorithm is used to perform edge detection on the preprocessed real assembly image and filter out noisy contours to extract the contour information of the real assembly parts. The contour information of the actual assembled parts is matched one by one with the optimal theoretical contour information for overlap and edge deviation to obtain the contour comparison results.

[0012] In one embodiment, based on the contour comparison result, misassembly / missing parts detection is performed according to the dedicated detection template to obtain detection result information, including: Based on the contour comparison results, the system performs category consistency judgment and misassembly detection according to the dedicated detection template, and judges missing and extra assembly detection based on contour information matching. The judgment results are then verified by combining coordinate position and posture deviation threshold to obtain the detection result information.

[0013] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method for detecting errors and omissions based on digital model feature recognition.

[0014] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned method for detecting errors and omissions based on digital model feature recognition.

[0015] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned error / missing item detection method based on digital model feature recognition.

[0016] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a method, device, medium, and product for detecting omissions and errors in assembly based on digital model feature recognition. By determining a dedicated detection template, it enables comprehensive control over all parts to be inspected in an assembly scenario, avoiding the limitations of single-part inspection and improving the comprehensiveness of the inspection. By transforming each element in the part model group to the coordinate system of the image acquisition device based on a spatial transformation matrix, the contour matching accuracy can be effectively improved. Then, the optimal theoretical contour information of the part under the current pose is extracted according to the intrinsic parameter matrix of the image acquisition device. Based on the acquired real assembly image of the workpiece to be tested, the corresponding real assembly part contour information is determined and compared with the optimal theoretical contour information to obtain the contour comparison result. Omission and error detection is then performed according to the dedicated detection template, which can further improve the reliability and accuracy of the inspection. Therefore, this application can achieve efficient and accurate detection of omissions and errors. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a method for detecting mis- or missing parts based on digital model feature recognition; Figure 2 A flowchart illustrating the design and operation of a method for detecting errors and omissions based on digital model feature recognition; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The purpose of this application is to solve the problems of low efficiency of manual inspection, insufficient accuracy of machine vision inspection, and imperfect process in the current technology, and to realize automated and high-precision inspection of defects such as misassembly, omission, and overassembly of assembled workpieces.

[0021] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] This application can be applied to the field of industrial assembly inspection technology, and is especially suitable for the automated detection of defects such as misassembly, omission, and multiple assembly during the assembly process of workpieces in various aerospace, automotive, and machinery fields.

[0023] This method can be implemented using an industrial camera and a projector. First, a 3D digital model of the workpiece to be inspected corresponding to the assembly scene is acquired. Individual part model groups requiring error / omission detection are extracted, and the standard assembly position coordinates and standard posture parameters of each element in the part model group are determined, generating a dedicated inspection template containing this information. Then, target points with known spatial coordinates are placed in the assembly scene, and the industrial camera parameters are adjusted for image acquisition, obtaining a 2D image containing the target points. Based on the 3D coordinates of the target points in the world coordinate system, combined with the 2D pixel coordinates acquired by the industrial camera, the spatial transformation matrix between the industrial camera and the real assembly scene is calculated, achieving the construction of a unified spatial coordinate system. Furthermore, each element in the part model group is transformed to the industrial camera coordinate system, and the optimal theoretical contour of the part under the current pose is extracted based on the industrial camera's intrinsic parameter matrix. Next, the industrial camera captures a real assembly image of the workpiece, and the contour of the real assembled parts is extracted using image processing methods, comparing and analyzing it with the optimal theoretical contour. Finally, considering category consistency, position deviation, and posture deviation, a comprehensive judgment is made regarding workpiece misassembly, omission, and multiple assembly defects, and the inspection results are output.

[0024] In one exemplary embodiment, a method for detecting errors and omissions based on digital model feature recognition is provided. This method is executed by a computer device, specifically by a computer device such as a terminal or a server alone, or by both a terminal and a server. In this embodiment, the method is described using a server as an example.

[0025] like Figure 1 As shown, this application provides a method for detecting errors and omissions in assembly based on digital model feature recognition, including: Step 100: Obtain the 3D digital model of the workpiece to be inspected in the assembly scene, and determine the dedicated inspection template corresponding to the part model group that needs to be inspected for omissions or errors. The dedicated inspection template includes the preset standard assembly position coordinates and preset standard posture parameters of each element in the part model group.

[0026] This includes acquiring a 3D digital model of the workpiece to be inspected in the assembly scene, and determining the dedicated inspection template corresponding to the part model group of the workpiece that needs to be inspected for errors or omissions. Specifically, this includes: A 3D digital model of the workpiece to be tested in the assembly scene is generated using 3D scanning or preset software; the 3D digital model is preprocessed to obtain a standardized 3D digital model; based on the standardized 3D digital model, a feature extraction algorithm is used to extract the geometric structural features of the workpiece to be tested, as well as to perform category determination and structural analysis, in order to extract the model group of parts that need to be inspected for errors or omissions.

[0027] Based on the pose information of the 3D digital model in the world coordinate system, the preset standard assembly position coordinates and preset standard posture parameters of the workpiece to be tested are extracted, and the category information, position coordinates and posture parameters are associated to generate a corresponding special inspection template.

[0028] Specifically, such as Figure 2 S1: Obtain the 3D digital model of the workpiece to be tested corresponding to the assembly scene to be tested, extract the individual part model group that needs to be inspected for errors and omissions, clarify the standard assembly position coordinates and standard posture parameters of each element in the part model group, and generate a special inspection template containing the above information.

[0029] S11: Obtain the three-dimensional digital model of each workpiece to be tested in the assembly scene to be tested, and preprocess the three-dimensional digital model to obtain standardized three-dimensional model data (standardized three-dimensional digital model).

[0030] S12: Based on the preprocessed 3D digital model, a feature extraction algorithm is used to extract the geometric structural features of the workpiece. The features include key point features, boundary contour features, or surface structure features. Based on the features, the workpiece is classified and its structure is analyzed.

[0031] S13: Based on the pose information of the 3D digital model in the world coordinate system, extract the standard assembly position coordinates and standard posture parameters of the workpiece to be tested, and associate the category information, position coordinates and posture parameters to generate the corresponding special inspection template.

[0032] Specifically, by 3D scanning or importing CAD models, including STP / CAD geometric input (i.e., importing STEP format files or 3D models generated by CAD software), a 3D digital model of the assembly scene and workpiece to be inspected is obtained, ensuring that it contains complete geometric and assembly information of all parts to be inspected. A region-growing-based model segmentation algorithm is then used to extract individual part model groups that require error / omission inspection. , recorded as Manual verification ensures complete segmentation; This represents the number of part models contained in the part model group.

[0033] At the same time, standard parameters are defined, and the standard assembly position coordinates of each part model are measured using the world coordinate system as a reference. Extract standard attitude parameters (rotation matrix) Translation vector Record part category information; finally, integrate part model groups, categories, and standard parameters to construct an inspection template. It is stored in the database in JSON format for easy access later.

[0034] Step 200: Project an coded grating pattern onto the target area using a projection device, and adjust the parameters of the image acquisition device to acquire an image, thereby obtaining a two-dimensional image. The target area is the region in the assembly scene where target points with known spatial coordinates are arranged; the two-dimensional image contains the target points.

[0035] This process involves projecting a coded grating pattern onto a target area using a projection device and adjusting the parameters of an image acquisition device to acquire a two-dimensional image, including: Multiple target points are placed in the assembly scene, and the correspondence between the target points and the world coordinate system is established; the three-dimensional spatial coordinates of the target points are known values ​​in the world coordinate system.

[0036] The projector is controlled to project an coded grating pattern onto the target area. The parameters of the image acquisition device are adjusted to acquire images of the scene containing the target points, thereby obtaining a two-dimensional image containing the target points.

[0037] Projection equipment can be a projector, and image acquisition equipment can be an industrial camera. For example... Figure 2S2: By arranging target points with known spatial coordinates in the assembly scene, controlling the projector to project an coded grating pattern onto the target area, and adjusting the parameters of the industrial camera to acquire images, a two-dimensional image containing the target points is obtained. S2 includes: S21: Arrange several target points in the assembly scene. The three-dimensional spatial coordinates of the target points are known values ​​in the world coordinate system. Establish the correspondence between the target points and the world coordinate system.

[0038] S22: Control the projector to project a preset coded grating pattern onto the area of ​​the workpiece to be tested, adjust the internal parameters and exposure parameters of the industrial camera, acquire images of the scene containing the target point, and obtain two-dimensional image data corresponding to the target point.

[0039] S23: Process the acquired image, extract the two-dimensional pixel coordinates of the target point in the imaging plane of the industrial camera, and obtain the two-dimensional pixel coordinates of the target point in the imaging plane of the projector based on the decoding result of the coded grating pattern.

[0040] Specifically, target points are placed around the assembly scene to be inspected and at key locations (such as the four corners of the assembly table and the edges of the workpiece fixture). These target points are high-contrast circular markers with a diameter of 5mm (black center, white ring, contrast ≥80%) to facilitate identification and extraction by the industrial camera. At least four target points are placed evenly to ensure coverage of the entire field of view of the industrial camera, avoiding target point obstruction or blind spots. After placement, the three-dimensional coordinates of each target point in the world coordinate system are measured using a 3D coordinate measuring machine (accuracy 0.01mm) to ensure accurate coordinate data. Next, the industrial camera parameters are adjusted: a 1920×1080 resolution industrial camera is selected, and the lens focal length is set to 1... The camera is set to 6mm to ensure image clarity; the pixel size is set to 2.2μm; the exposure time is adjusted according to the scene's light intensity (usually 10ms-50ms); and the ISO is set to 100 to avoid overexposure or underexposure. Simultaneously, the camera's white balance parameters are adjusted to ensure accurate color reproduction, facilitating subsequent extraction of target points and part contours. After parameter adjustments, the camera is calibrated to obtain its intrinsic parameter matrix, providing a basis for subsequent projection transformations. Images of the assembly scene with target points are acquired using an industrial camera, and the frame with the highest clarity and unobstructed target points is selected as the valid image. The clear, unobstructed valid image is transmitted to the detection system, providing a reliable image basis for subsequent coordinate system transformations.

[0041] Step 300: Extract the two-dimensional pixel coordinates of each target point based on the two-dimensional image, and determine the spatial transformation matrix between the three-dimensional digital model and the assembly scene based on the three-dimensional coordinates of the target points in the world coordinate system, combined with the two-dimensional pixel coordinates of the image acquisition device and the projection device. The two-dimensional pixel coordinates of the projection device are the two-dimensional pixel coordinates of the target points in the projection device plane obtained based on the decoding result of the coded raster pattern.

[0042] Specifically, the spatial transformation matrix between the 3D digital model and the assembly scene is determined based on the extraction of 2D pixel coordinates of each target point from the 2D image, and the 3D coordinates of the target points in the world coordinate system, combined with the 2D pixel coordinates of the image acquisition device and the projection device. This includes: The two-dimensional image is preprocessed, and a threshold segmentation algorithm is used for binarization. Then, a contour extraction algorithm is used to extract the two-dimensional pixel coordinates of each target point.

[0043] Based on the correspondence between the three-dimensional coordinates of the target point in the world coordinate system and the two-dimensional pixel coordinates of the image acquisition device, the PnP algorithm is used to solve for the extrinsic parameters of the image acquisition device relative to the world coordinate system; the extrinsic parameters include the rotation matrix and the translation vector.

[0044] Based on the two-dimensional pixel coordinates of the projection device, and combined with the geometric constraints between the image acquisition device and the projection device, the spatial mapping relationship between the projection device coordinate system and the world coordinate system is determined.

[0045] Based on the extrinsic parameters of the image acquisition device relative to the world coordinate system and the spatial mapping relationship between the projection device coordinate system and the world coordinate system, the spatial transformation matrix between the 3D digital model and the assembly scene is determined so as to transform the 3D digital model into the unified world coordinate system corresponding to the assembly scene.

[0046] like Figure 2 S3: Based on the 3D coordinates of the target point in the world coordinate system, combined with the 2D pixel coordinates acquired by the industrial camera and the 2D pixel coordinates of the projector, the spatial transformation matrix between the digital model and the real assembly scene is calculated to achieve the construction of a unified spatial coordinate system. S3 includes: S31: Based on the correspondence between the 3D coordinates of the target point in the world coordinate system and its 2D pixel coordinates in the imaging plane of the industrial camera, the PnP algorithm is used to solve for the extrinsic parameters of the industrial camera relative to the world coordinate system, including the rotation matrix. Translation vector .

[0047] S32: Based on the two-dimensional pixel coordinates of the target point in the projector's imaging plane, and combined with the geometric constraints between the industrial camera and the projector, establish a spatial mapping relationship between the projector coordinate system and the world coordinate system.

[0048] S33: Based on the external parameters of the industrial camera and the spatial mapping relationship of the projector, construct a spatial transformation matrix between the 3D digital model and the actual assembly scene, and transform the 3D digital model into the unified world coordinate system corresponding to the actual assembly scene.

[0049] Specifically, the acquired 2D images are preprocessed. First, the images are converted to grayscale to reduce data volume and improve processing speed. Then, Gaussian filtering (filter radius 1.5mm) is used to remove noise from the images and eliminate environmental interference. Finally, a threshold segmentation algorithm (threshold set to 127) is used to binarize the images to highlight the target point contours. A contour extraction algorithm is used to extract the 2D pixel coordinates of each target point, with an extraction accuracy of 0.1 pixels to ensure the accuracy of the pixel coordinate data. The extrinsic parameters are solved based on the correspondence between the 3D coordinates of the target points in the world coordinate system and the 2D pixel coordinates. The PnP algorithm is used to solve the extrinsic parameters of the industrial camera relative to the world coordinate system, including a 3×3 rotation matrix and a 3×1 translation vector. By solving for the extrinsic parameters, a spatial mapping relationship between the industrial camera coordinate system and the world coordinate system is established, and a 4×4 homogeneous spatial transformation matrix M between the industrial camera and the real assembly scene is calculated. The spatial transformation matrix is ​​verified to ensure its orthogonality and invertibility, providing a guarantee for subsequent coordinate system transformation. Finally, a unified spatial coordinate system is constructed: using the spatial transformation matrix M, the world coordinate system of the 3D digital model and the industrial camera coordinate system are unified to construct a unified spatial coordinate system adapted to the real assembly scene. Three verification points are selected to verify the coordinate error under the unified coordinate system, ensuring that the error is ≤0.05mm, eliminating the detection error caused by the coordinate system difference, and laying the foundation for the accurate matching of the theoretical contour and the real contour in the future.

[0050] Step 400: Transform each element in the part model group to the coordinate system of the image acquisition device based on the spatial transformation matrix, and extract the optimal theoretical contour information of the part under the current pose according to the intrinsic parameter matrix of the image acquisition device.

[0051] This process involves transforming each element in the part model group to the coordinate system of the image acquisition device based on a spatial transformation matrix, and extracting the optimal theoretical contour information of the part under the current pose based on the intrinsic parameter matrix of the image acquisition device. This includes: Based on the spatial transformation matrix, the part model group is transformed into the coordinate system of the image acquisition device, and combined with the intrinsic parameter matrix of the image acquisition device, the three-dimensional part model group is projected into a two-dimensional projected image; the Canny edge detection and contour extraction algorithm is used to extract the optimal theoretical contour information of the part.

[0052] like Figure 2 S4 in the equation: Transform each element in the part model group to the industrial camera coordinate system, and extract the optimal theoretical contour of the part under the current pose based on the intrinsic parameter matrix of the industrial camera; wherein, S4 includes: S41: Transform the part model group to the camera coordinate system using the spatial transformation matrix M, and verify the accuracy of the transformation.

[0053] S42: Combine the camera intrinsic parameter matrix to project the 3D part model into a 2D projected image.

[0054] S43: Employs Canny edge detection, contour extraction, and simplification algorithms to extract the optimal theoretical contour of the part, ensuring a complete and distortion-free contour.

[0055] Specifically, the model coordinate system transformation process, based on the unified spatial coordinate system constructed in S3, transforms the three-dimensional digital models of each element in the part model group generated in S1 through a spatial transformation matrix. Perform a homogeneous transformation to the industrial camera coordinate system. The transformation formula is as follows: ,in This is a model of the part in the industrial camera coordinate system. This is a model of the part in the world coordinate system. This represents matrix multiplication and dot product. After the transformation, the accuracy of the model transformation is verified by comparing the pose of the part model in the industrial camera coordinate system with the pose of the part in the actual assembly scene, ensuring no pose deviation. Projection transformation, combined with the intrinsic parameter matrix K of the industrial camera, performs projection transformation on each element of the transformed part model group, simulating the imaging process of the industrial camera, projecting the 3D part model onto the 2D image plane to obtain a 2D projected image of the part. During the projection process, it is ensured that the projection ratio is consistent with the camera imaging ratio to avoid projection distortion. Optimal theoretical contour extraction: The Canny edge detection algorithm is used (low threshold set to 50, high threshold set to 1). 50) Edge detection is performed on the 2D projection image to obtain the edge image of the part; then, a contour extraction algorithm is used to extract the initial contour of the part, and redundant contour points are removed by a contour simplification algorithm (Douglas-Puk algorithm, with a simplification threshold set to 0.5 pixels). Taking into account the geometric features of the part, edge clarity, and adaptability to the assembly scene, the optimal theoretical contour of each element of the part model group under the current pose is extracted; the extracted optimal theoretical contour is verified to ensure that the contour is complete and without distortion, and can accurately reflect the real shape of the part under the current pose, providing a precise theoretical reference for subsequent comparison with the real part contour.

[0056] Step 500: Based on the acquired real assembly image of the workpiece to be tested, determine the corresponding real assembly part contour information, and compare and analyze it with the optimal theoretical contour information to obtain the contour comparison result.

[0057] Specifically, based on the acquired actual assembly image of the workpiece to be tested, the corresponding actual assembly part contour information is determined, and compared with the optimal theoretical contour information to obtain contour comparison results, including: The acquired real assembly image of the workpiece to be tested is converted to grayscale, denoised by Gaussian filtering, and edge enhanced to obtain a preprocessed real assembly image. The Canny edge detection algorithm is used to detect edges in the preprocessed real assembly image and filter out noise contours to extract the contour information of the real assembly parts. The contour information of the real assembly parts is matched and analyzed one by one with the optimal theoretical contour information for overlap and edge deviation to obtain the contour comparison results.

[0058] like Figure 2 S5 in the process involves: using an industrial camera to capture a real image of the workpiece under test; extracting the real part contour using image processing methods; and comparing and analyzing this real part contour with the optimal theoretical contour. S5 includes: S51: Acquire real assembly images of the workpiece to be tested, preprocess the real images, convert the images to grayscale images, and use Gaussian filtering algorithm to remove noise and perform edge enhancement and other preprocessing.

[0059] S52: The Canny edge detection algorithm is used to perform edge detection on the preprocessed real image to obtain the edge image of the part. Then, the algorithm consistent with the theoretical contour extraction is used to extract the real part contour and filter out the noisy contour.

[0060] S53: Match the actual contour with the optimal theoretical contour one by one, calculate parameters such as overlap and edge deviation, and generate a matching report.

[0061] Specifically, the real image acquisition and preprocessing are as follows: Using the same acquisition parameters as S2, real assembly images of the workpiece under test are captured, and the frame with the highest clarity is selected as the real image. The real image is preprocessed as follows: First, the image is converted to grayscale to reduce data volume and improve subsequent processing efficiency. Then, a Gaussian filter algorithm (filter radius 1.5mm) is used to remove noise from the image and eliminate interference from ambient light, dust, etc. Finally, the Sobel operator (edge ​​detection threshold set to 100) is used to enhance the edge features of the part, improving edge clarity and facilitating subsequent contour extraction. Real part contour extraction is performed: The Canny edge detection algorithm (low threshold 50, high threshold 150) consistent with S4 is used to perform edge detection on the preprocessed real image to obtain the edge image of the part. Then, a contour extraction algorithm is used to extract the contour of the part. In a real assembly scenario, the true contours of each part are recorded, including pixel coordinates, contour area, perimeter, and other feature parameters. The extracted true contours are then filtered to remove noisy contours with an area less than 50 pixels, ensuring that the extracted contours are valid part contours and avoiding noise interference in subsequent comparative analysis. Contour comparison analysis: The extracted true part contours are matched one by one with the optimal theoretical contours obtained in S4, using a contour matching algorithm based on Hu moment invariants (matching threshold set to 0.8). Parameters such as contour overlap (the proportion of overlapping area to the theoretical contour area) and edge deviation (the average distance between corresponding points of the true and theoretical contours) are calculated. The matching status of each part's contour is recorded, and a contour matching report is generated. The report includes matching parameters for each part, contour comparison diagrams, and other information, providing data support for subsequent defect identification.

[0062] Step 600: Based on the contour comparison results, perform mis-assembly and omission detection according to the dedicated detection template to obtain the detection result information.

[0063] Among them, based on the contour comparison results, mis-assembly and omission detection is performed according to a dedicated detection template to obtain detection result information, including: Based on the contour comparison results, the system performs category consistency judgment and misassembly detection according to the dedicated detection template. It also judges missing and extra assembly based on the contour information matching. The judgment results are verified by combining the coordinate position and attitude deviation threshold to obtain the detection result information.

[0064] like Figure 2 S6 in the code: Combining category consistency, positional deviation, and posture deviation, it comprehensively judges workpiece misassembly, omission, and excess assembly defects, and outputs the detection result information; S6 includes: S61: Combining the contour comparison results, mis-assembly is determined by category consistency, and missing or extra assembly is determined by contour matching. The determination results are verified by combining the position and posture deviation thresholds.

[0065] S62: The final defect identification and inspection report includes information such as the inspection process, contour comparison data, and defect details. The defect location is visualized on the system interface through the output module of the inspection system, and the inspection results are stored in the system database for subsequent traceability and analysis, so as to realize the automated and accurate detection of defects such as misassembly and omission of assembled parts.

[0066] Specifically, consistency judgment: The category information of the actual part is identified by contour features and size parameters and compared with the standard category information in the detection template. If they are inconsistent, it is directly judged as misassembly, and the position and category information of the misassembly part are recorded. Missing part judgment: All part elements in the detection template are traversed. If the actual contour corresponding to a certain part element is not extracted and the contour overlap is 0, it is judged as missing part, and the category and standard assembly position of the missing part are recorded. Multiple assembly judgment: If a part contour not included in the detection template is extracted in the actual assembly scene, and the contour area and size match the part characteristics, it is judged as... For multiple assembly, record the category and actual assembly position of the multiple assembled parts; Deviation verification: For parts with the same category and existing contours, calculate the positional deviation (deviation values ​​in the X, Y, and Z axes, calculated through the coordinate difference of corresponding points on the contour) and attitude deviation (rotation angle deviation around the X, Y, and Z axes, calculated through the directional features of the contour) between the actual part contour and the optimal theoretical contour. Combined with the preset deviation thresholds (positional deviation threshold set to 0.2mm, attitude deviation threshold set to 0.5°), if the deviation value exceeds the threshold, further confirm whether there is misassembly or improper assembly, and correct the defect judgment result.

[0067] Beneficial effects: The misassembly and omission detection scheme based on 3D digital model contour matching provided in this application achieves automated and high-precision detection of misassembly, omission, and multiple assembly defects in assembled workpieces by generating dedicated detection templates, constructing a unified spatial coordinate system, extracting the optimal theoretical contour and comparing it with the real contour. By extracting part model groups from the 3D digital model and generating detection templates, it enables overall control of all parts to be inspected in the assembly scenario, avoiding the limitations of single-part inspection and improving the comprehensiveness of the inspection. Coordinate system unification based on target points ensures that the matching of theoretical and real contours is based on the same coordinate system, effectively improving contour matching accuracy and reducing false detections and omissions. The scheme improves the contour extraction, deviation calculation, and defect discrimination processes, combining category consistency, positional deviation, and posture deviation for comprehensive discrimination, further enhancing the reliability and accuracy of the inspection. The entire inspection process is highly automated, requiring no manual intervention, significantly improving inspection efficiency and adapting to the needs of large-scale industrial production.

[0068] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 3As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores error detection data based on digital model feature recognition. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the error detection method based on digital model feature recognition.

[0069] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0070] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0071] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0072] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0073] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0074] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0075] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logic devices, etc., and are not limited to these.

[0076] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0077] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting mis- or missing parts based on digital model feature recognition, characterized in that, The error / missing package detection method based on digital model feature recognition includes: A three-dimensional digital model of the workpiece to be tested in the assembly scene to be tested is obtained, and a special inspection template is determined for the part model group of the workpiece to be tested that needs to be inspected for errors or omissions. The special inspection template includes the preset standard assembly position coordinates and preset standard posture parameters of each element in the part model group. A coded grating pattern is projected onto the target area using a projection device, and the parameters of the image acquisition device are adjusted to acquire a two-dimensional image. The target area is a region in the assembly scene where target points with known spatial coordinates are arranged. The two-dimensional image contains the target points. Based on the two-dimensional image, the two-dimensional pixel coordinates of each target point are extracted. Based on the three-dimensional coordinates of the target point in the world coordinate system, combined with the two-dimensional pixel coordinates of the image acquisition device and the two-dimensional pixel coordinates of the projection device, the spatial transformation matrix between the three-dimensional digital model and the assembly scene is determined. The two-dimensional pixel coordinates of the projection device are the two-dimensional pixel coordinates of the target point in the projection device plane obtained based on the decoding result of the coded grating pattern. Each element in the part model group is transformed to the coordinate system of the image acquisition device based on the spatial transformation matrix, and the optimal theoretical contour information of the part under the current pose is extracted according to the intrinsic parameter matrix of the image acquisition device. Based on the acquired real assembly image of the workpiece to be tested, the corresponding real assembly part contour information is determined, and compared and analyzed with the optimal theoretical contour information to obtain the contour comparison result. Based on the contour comparison results, the system performs mis-assembly and omission detection according to the dedicated detection template to obtain the detection result information.

2. The method for detecting mis-assembly and omissions based on digital model feature recognition according to claim 1, characterized in that, Obtain the 3D digital model of the workpiece to be inspected in the assembly scene, and determine the dedicated inspection template corresponding to the part model group of the workpiece that needs to be inspected for errors or omissions. Specifically, this includes: A three-dimensional digital model of the workpiece to be tested in the assembly scene is generated using 3D scanning or preset software. The three-dimensional digital model is preprocessed to obtain a standardized three-dimensional digital model; Based on a standardized three-dimensional digital model, a feature extraction algorithm is used to extract the geometric structural features of the workpiece under test, as well as to perform category determination and structural analysis, in order to extract the model group of parts that need to be inspected for errors and omissions. Based on the pose information of the 3D digital model in the world coordinate system, the preset standard assembly position coordinates and preset standard posture parameters of the workpiece to be tested are extracted, and the category information, position coordinates and posture parameters are associated to generate a corresponding special inspection template.

3. The method for detecting mis-assembly and omissions based on digital model feature recognition according to claim 1, characterized in that, A coded grating pattern is projected onto the target area using a projection device, and the parameters of the image acquisition device are adjusted to acquire a two-dimensional image, including: Multiple target points are arranged in the assembly scene, and the correspondence between the target points and the world coordinate system is established; the spatial three-dimensional coordinates of the target points are known values ​​in the world coordinate system. The projector is controlled to project an coded grating pattern onto the target area. The parameters of the image acquisition device are adjusted to acquire images of the scene containing the target points, thereby obtaining a two-dimensional image containing the target points.

4. The method for detecting mis-assembly and omissions based on digital model feature recognition according to claim 1, characterized in that, Based on the two-dimensional image, the two-dimensional pixel coordinates of each target point are extracted. Then, based on the three-dimensional coordinates of the target points in the world coordinate system, combined with the two-dimensional pixel coordinates of the image acquisition device and the projection device, the spatial transformation matrix between the three-dimensional digital model and the assembly scene is determined, including: The two-dimensional image is preprocessed, and a threshold segmentation algorithm is used for binarization. Additionally, a contour extraction algorithm is used to extract the two-dimensional pixel coordinates of each target point. Based on the correspondence between the three-dimensional coordinates of the target point in the world coordinate system and the two-dimensional pixel coordinates of the image acquisition device, the PnP algorithm is used to solve for the extrinsic parameters of the image acquisition device relative to the world coordinate system; the extrinsic parameters include rotation matrix and translation vector. Based on the two-dimensional pixel coordinates of the projection device, and combined with the geometric constraints between the image acquisition device and the projection device, the spatial mapping relationship between the projection device coordinate system and the world coordinate system is determined. Based on the extrinsic parameters of the image acquisition device relative to the world coordinate system and the spatial mapping relationship between the projection device coordinate system and the world coordinate system, the spatial transformation matrix between the 3D digital model and the assembly scene is determined so as to transform the 3D digital model into the unified world coordinate system corresponding to the assembly scene.

5. The method for detecting mis-assembly and omissions based on digital model feature recognition according to claim 1, characterized in that, Each element in the part model group is transformed to the coordinate system of the image acquisition device based on the spatial transformation matrix. The optimal theoretical contour information of the part under the current pose is extracted based on the intrinsic parameter matrix of the image acquisition device, including: Based on the spatial transformation matrix, the part model group is transformed into the coordinate system of the image acquisition device, and combined with the intrinsic parameter matrix of the image acquisition device, the three-dimensional part model group is projected into a two-dimensional projected image. The Canny edge detection and contour extraction algorithm is used to extract the optimal theoretical contour information of the part.

6. The method for detecting mis-assembly and omissions based on digital model feature recognition according to claim 1, characterized in that, Based on the acquired actual assembly image of the workpiece to be tested, the corresponding actual assembly part contour information is determined, and compared and analyzed with the optimal theoretical contour information to obtain the contour comparison result, including: The acquired real assembly image of the workpiece to be tested is converted to grayscale, denoised by Gaussian filtering, and edge enhancement is performed to obtain the preprocessed real assembly image. The Canny edge detection algorithm is used to perform edge detection on the preprocessed real assembly image and filter out noisy contours to extract the contour information of the real assembly parts. The contour information of the actual assembled parts is matched one by one with the optimal theoretical contour information for overlap and edge deviation to obtain the contour comparison results.

7. The method for detecting mis-assembly and omissions based on digital model feature recognition according to claim 1, characterized in that, Based on the contour comparison results, misassembly and omission detection is performed according to the dedicated detection template to obtain detection result information, including: Based on the contour comparison results, the system performs category consistency judgment and misassembly detection according to the dedicated detection template, and judges missing and extra assembly detection based on contour information matching. The judgment results are then verified by combining coordinate position and posture deviation threshold to obtain the detection result information.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the error / missing part detection method based on digital model feature recognition as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the error / missing assembly detection method based on digital model feature recognition as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the error / missing assembly detection method based on digital model feature recognition as described in any one of claims 1-7.