Three-dimensional positioning and visualization method and device for surface defects based on multi-view fusion

Abstract

The application discloses a kind of surface defect three-dimensional positioning and visualization method and device based on multi-view fusion;The application realizes the accurate positioning of defect in world coordinate system by the back projection calculation of two-dimensional pixel coordinates and camera pose combination;Secondly, the DBSCAN clustering algorithm is introduced to automatically remove the defect points observed by multi-view, so that the same defect from different angles can be effectively identified and merged, the accuracy of defect counting is improved, and the clustering algorithm does not need to specify the number of clusters in advance, has strong anti-noise interference ability, can improve the clustering accuracy, and can accurately distinguish different defect points;Finally, the application innovatively generates defect distribution model graph and detection coverage display model graph, and carries out visual display;Based on this, not only the real three-dimensional distribution of defect on the surface of object can be intuitively presented, which can assist quality analysis, but also the coverage range of single scanning can be accurately quantified and displayed.

Description

Technical Field

[0001] This invention belongs to the field of industrial automation inspection technology, specifically relating to a method and device for three-dimensional localization and visualization of surface defects based on multi-view fusion. Background Technology

[0002] Currently, most industrial surface defect detection systems adopt a single-view detection method, which involves scanning the target surface with a fixed camera. Defect localization is based solely on the pixel coordinates of a single image, without considering the impact of camera pose changes on the defect position. Therefore, existing technologies typically suffer from the following defects: (1) Low defect localization accuracy, as camera pose changes are not considered, leading to inaccurate calculation of the actual position of the defect on the object surface. Furthermore, there is a problem of repeated defect identification, meaning that when the same defect appears in multiple photos, it cannot be effectively identified as the same defect, resulting in errors in defect count statistics. (2) Traditional clustering algorithms (such as K-means) require pre-specifying the number of clusters, are sensitive to noise, and cannot accurately distinguish different defect points. (3) Lack of spatial distribution visualization; it cannot intuitively display the actual distribution of defects on the object surface, which is not conducive to defect analysis and quality assessment. (4) It cannot intuitively assess the actual coverage of a single scan. Therefore, based on the aforementioned shortcomings, how to provide a multi-view fusion surface defect three-dimensional localization and visualization method with high defect localization accuracy and the ability to intuitively display the actual distribution of defects and the coverage of a single scan has become an urgent problem to be solved. Summary of the Invention

[0003] The purpose of this invention is to provide a method and apparatus for three-dimensional localization and visualization of surface defects based on multi-view fusion, in order to solve the problems of low defect localization accuracy, repeated defect identification, inability to accurately distinguish different defect points, lack of spatial distribution visualization, and inability to intuitively evaluate the actual coverage of a single scan in the prior art.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a method for three-dimensional localization and visualization of surface defects based on multi-view fusion is provided, including: Acquire several images captured by an industrial camera along a preset trajectory to detect defects in a target object, as well as the camera pose corresponding to each image; Each image is processed for defect detection to obtain the pixel coordinates of the defect points in each image; Based on the camera pose corresponding to each image, the pixel coordinates of the defect points in each image are mapped to the world coordinate system to obtain the three-dimensional coordinates of each defect point in the world coordinate system. Based on the three-dimensional coordinates of each defect point, the DBSCAN clustering algorithm is used to cluster all defect points to obtain several defect point clusters. Among them, multiple defect points in any defect point cluster correspond to the same defect location, and the images corresponding to multiple defect points are taken from different perspectives. Based on several defect point clusters, the actual three-dimensional coordinates of different defect locations are determined; Based on the actual three-dimensional coordinates of each defect location, each defect location is mapped to the view plane of the masking model of the target object in the world coordinate system to obtain a defect distribution model diagram. Using several images, the scanning range of the industrial camera on the mask model during each defect detection is determined, and a detection coverage display model map is generated based on the scanning range during each defect detection. The defect distribution model and the detection coverage display model are visualized to complete the three-dimensional visualization and localization of surface defects of the target object.

[0005] Based on the aforementioned disclosure, this invention achieves precise localization of defects in the world coordinate system by combining two-dimensional pixel coordinates with camera pose for back projection calculation, thus fundamentally overcoming the localization distortion problem caused by changes in viewing angle. Secondly, it introduces the DBSCAN clustering algorithm to automatically deduplicate defect points observed from multiple perspectives. This effectively identifies and merges the same defect from different angles, improving the accuracy of defect counting. Moreover, this clustering algorithm does not require pre-specifying the number of clusters, has strong anti-noise interference capabilities, and can improve clustering accuracy, thereby accurately distinguishing different defect points. Finally, this invention innovatively generates a defect distribution model diagram and a detection coverage display model diagram, and visualizes them. Based on this, not only can the true three-dimensional distribution of defects on the object surface be intuitively presented, which can assist in quality analysis, but it can also accurately quantify and display the coverage of a single scan. Thus, by fusing multi-view images and precise camera pose information, this invention achieves full-process optimization of precise localization, intelligent deduplication, and intuitive visualization of surface defects from two-dimensional detection to three-dimensional space, effectively solving the inherent defects of traditional technologies, thereby significantly improving the intelligence level and decision support capabilities of automated detection.

[0006] In one possible design, based on the camera pose corresponding to each image, the pixel coordinates of the defect points in each image are mapped to the world coordinate system, including: Obtain the intrinsic parameter matrix of the industrial camera and the mask model corresponding to the target object; For any image, the pixel coordinates of the defect points in the image are converted into normalized camera coordinates according to the intrinsic parameter matrix. Construct the first ray equation of the normalized camera coordinates in the camera coordinate system; Based on the camera pose corresponding to any image, determine the rotation matrix and translation matrix between the camera coordinate system and the world coordinate system, and based on the rotation matrix and the translation matrix, transform the first ray equation to the world coordinate system to obtain the second ray equation; The masking surface of any image in the masking model is determined, and the coordinates of the intersection point between the second ray equation and the masking surface are calculated. The coordinates of the intersection point are then used as the three-dimensional coordinates of the defect point in any image in the world coordinate system.

[0007] In one possible design, based on the intrinsic parameter matrix, the pixel coordinates of defect points in any image are converted into normalized camera coordinates, including: The pixel coordinates of defect points in any of the images are converted into normalized camera coordinates according to the following formula; ; In the formula, For the camera coordinate matrix, ,in, Represents normalized camera coordinates. This represents the transpose operator. Represents the intrinsic parameter matrix. The x-coordinate and y-coordinate of the pixel representing the defect point in any image.

[0008] In one possible design, the first ray equation of the normalized camera coordinates in the camera coordinate system is constructed, including: The first ray equation is constructed according to the following formula; ; In the formula, This represents the three-dimensional coordinates of a defect point in any image in the camera coordinate system. For depth parameters, The camera coordinate matrix; Accordingly, based on the rotation matrix and the translation matrix, the first ray equation is transformed to the world coordinate system to obtain the second ray equation, which includes: Transform the first ray equation to the world coordinate system according to the following formula; ; In the formula, Let the three-dimensional coordinates of the defect point in any of the images be given in the world coordinate system. For rotation matrix, It is a translation matrix. This indicates the transpose operation.

[0009] In one possible design, the coordinates of the intersection point between the second ray equation and the masking surface are calculated, including: Obtain the planar unit normal vector of the masking surface, and the projection distance from the industrial camera to the masking surface when the industrial camera captures any of the images; Based on the projection distance and the unit normal vector of the plane, the plane equation of the masking surface is constructed; The coordinates of the intersection point are obtained by using the plane equation and the second ray equation.

[0010] In one possible design, using several images, the scanning range of the industrial camera on the masked model during each defect inspection is determined, including: Obtain the coordinates of the four corner points of the camera image corresponding to each image, where the coordinates of the four corner points of any image are the coordinates of the four endpoints of that image; Transform the coordinates of the four corner points of each image to the world coordinate system to obtain the projection point of each corner point in each image onto the masking model; The area enclosed by the projection points corresponding to the four corner points in each image is taken as the scanning area of ​​each image on the mask model, and each scanning area is taken as the scanning range of the industrial camera on the mask model when capturing each image. Accordingly, based on the scanning range captured during each defect detection, a detection coverage display model diagram is generated, which includes: Several images are classified to obtain multiple image subsets, where each image subset corresponds to a masking surface on the masking model; For any masking surface, obtain each image in the image subset corresponding to that masking surface, and the scanning area on that masking surface, as the detection coverage area; Each detection coverage area is added to any of the masking surfaces to obtain the detection coverage surface corresponding to any of the masking surfaces, and after all the masking surfaces have been polled, the detection coverage surfaces corresponding to all the masking surfaces are obtained. The detection coverage display model diagram is generated using the detection coverage surfaces corresponding to all masking surfaces.

[0011] In one possible design, after obtaining the three-dimensional coordinates of each defect point in the world coordinate system, the method further includes: Obtain the image metadata corresponding to each image, including timestamp and task ID; The image metadata corresponding to each image and the three-dimensional coordinates of the defect points in each image are stored in the local SQLite database.

[0012] In one possible design, the method further includes: Acquire an image set captured by an industrial camera while performing multiple defect detection tasks, wherein the movement trajectory of the industrial camera while performing each defect detection task is the preset trajectory; Based on the camera imaging center point corresponding to each image in each image set, generate the scanning trajectory corresponding to each defect detection task; A comparison chart of scan trajectories is generated based on each scan trajectory.

[0013] In one possible design, based on the camera imaging center point corresponding to each image in each image set, the scanning trajectory corresponding to each defect detection task is generated, including: For any image set, the camera imaging center points corresponding to each image in the image set are transformed to the world coordinate system to obtain the projection center points of each camera imaging center point on the mask model. According to the shooting order of each image in any image set, the projection center points corresponding to the camera imaging center points of each image are connected sequentially to generate the scanning trajectory of the defect detection task corresponding to any image set. Accordingly, based on each scan trajectory, a scan trajectory comparison map is generated, which includes: For the projection center point at any same detection position in each scanning trajectory, calculate the scanning error distance between each projection center point at any same detection position, and after polling all projection center points in each scanning trajectory, obtain several scanning error distances. The mean error distance is obtained by calculating the average value among several scanning error distances. Each scan trajectory is colored and added to the mask model, and the average error distance is added to the mask model as a trajectory comparison index to generate the scan trajectory comparison map, wherein different scan trajectories correspond to different colors.

[0014] Secondly, a device for three-dimensional localization and visualization of surface defects based on multi-view fusion is provided, including: The acquisition unit is used to acquire several images captured by the industrial camera when it performs defect detection on the target object along a preset trajectory, as well as the camera pose corresponding to each image. The defect detection unit is used to perform defect detection processing on each image to obtain the pixel coordinates of the defect points in each image; The projection unit is used to map the pixel coordinates of the defect points in each image to the world coordinate system based on the camera pose corresponding to each image, so as to obtain the three-dimensional coordinates of each defect point in the world coordinate system. The clustering unit is used to cluster all defect points based on the three-dimensional coordinates of each defect point and using the DBSCAN clustering algorithm to obtain several defect point clusters. Among them, multiple defect points in any defect point cluster correspond to the same defect location, and the multiple defect points correspond to different shooting angles of the images. The defect location determination unit is used to determine the actual three-dimensional coordinates of different defect locations based on several defect point clusters; The defect analysis unit is used to map each defect location to the view plane of the masking model of the target object in the world coordinate system based on the actual three-dimensional coordinates of each defect location, so as to obtain a defect distribution model diagram. The defect analysis unit is used to determine the scanning range of the industrial camera on the mask model during each defect detection using several images, and to generate a detection coverage display model map based on the scanning range during each defect detection. The defect analysis unit is also used to visualize the defect distribution model diagram and the detection coverage display model diagram to complete the three-dimensional visualization and positioning of surface defects of the target object.

[0015] Thirdly, another surface defect three-dimensional localization and visualization device based on multi-view fusion is provided. Taking the device as an electronic device as an example, it includes a memory, a processor and a transceiver connected in sequence. The memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer programs and execute the surface defect three-dimensional localization and visualization method based on multi-view fusion as described in the first aspect or any possible design of the first aspect.

[0016] Fourthly, a storage medium is provided, on which instructions are stored, which, when executed on a computer, perform the multi-view fusion-based three-dimensional localization and visualization method for surface defects as described in the first aspect or any possible design of the first aspect.

[0017] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, causes the computer to perform the multi-view fusion-based three-dimensional localization and visualization method for surface defects as described in the first aspect or any possible design of the first aspect.

[0018] Beneficial effects: (1) This invention achieves precise positioning of defects in the world coordinate system by combining two-dimensional pixel coordinates with camera pose for back projection calculation, thereby fundamentally overcoming the positioning distortion problem caused by changes in viewing angle. Secondly, the DBSCAN clustering algorithm is introduced to automatically deduplicate defect points observed from multiple perspectives. This effectively identifies and merges the same defect from different angles, improving the accuracy of defect counting. Moreover, this clustering algorithm does not require pre-specifying the number of clusters, has strong anti-noise interference capability, and can improve clustering accuracy, thereby accurately distinguishing different defect points. Finally, this invention innovatively generates a defect distribution model diagram and a detection coverage display model diagram, and displays them visually. Based on this, not only can the real three-dimensional distribution of defects on the surface of an object be presented intuitively, which can assist in quality analysis, but it can also accurately quantify and display the coverage of a single scan. Thus, by integrating multi-view images and precise camera pose information, this invention achieves full-process optimization of precise positioning, intelligent deduplication, and intuitive visualization of surface defects from two-dimensional detection to three-dimensional space, effectively solving the inherent defects of traditional technologies, thereby significantly improving the intelligence level and decision support capability of automated detection.

[0019] (2) This invention supports the comparison of multiple tasks in the same space. That is, this invention generates the scanning trajectory corresponding to each defect detection task and calculates the difference distance between each scanning point at the same detection position in each scanning trajectory. Based on this, the average error distance between each scanning trajectory is obtained and used as the trajectory comparison index. In this way, this invention can achieve accurate quantitative evaluation of trajectory overlap rate, solve the inherent defect of traditional technology that cannot evaluate the consistency of multiple scanning behaviors, and thus facilitate the discovery of differences and problems in scanning strategies. Attached Figure Description

[0020] Figure 1 A flowchart illustrating the steps of a method for three-dimensional localization and visualization of surface defects based on multi-view fusion provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the effect of the defect distribution model provided in an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the effect of the detection coverage display model diagram provided in an embodiment of the present invention; Figure 4 A schematic diagram of a scanning trajectory comparison provided in an embodiment of the present invention; Figure 5 A structural diagram of the surface defect three-dimensional localization and visualization device based on multi-view fusion provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0022] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0023] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0024] Example: See Figure 1 As shown, the surface defect three-dimensional localization and visualization method based on multi-view fusion provided in this embodiment can be executed, but is not limited to, by a computer device with certain computing resources and a communication connection to an industrial camera. For example, it can be executed by electronic devices such as servers, edge computers, personal computers (PCs, which are multi-purpose computers of a size, price and performance suitable for personal use; desktop computers, laptops to mini-laptops and tablets and ultrabooks are all personal computers), smartphones or personal digital assistants (PDAs). It is understood that the aforementioned execution entities do not constitute a limitation on the embodiments of this application. Accordingly, the operation steps of this method can be, but are not limited to, the steps S1 to S8 below.

[0025] S1. Acquire several images captured by the industrial camera along a preset trajectory when performing defect detection on the target object, as well as the camera pose corresponding to each image; In specific implementation, before performing defect detection, this embodiment will pre-establish an electronic map of the environment where the target object (such as a vehicle in production) is located, and calibrate the intrinsic parameter matrix (focal length, principal point, distortion coefficient, etc.) of the industrial camera; at the same time, construct a cuboid mask model of the target object (such as a cuboid model of a vehicle, which may include the spatial pose (rotation and translation parameters) and size information of each mask surface), and calibrate the extrinsic parameter relationship between the lidar and the industrial camera.

[0026] Thus, after the aforementioned initialization, defect detection of the target object can be performed. For example, if an industrial camera is installed on a robot, the robot can be controlled to move along a preset trajectory, and the industrial camera can be triggered to take pictures simultaneously during the movement, thereby obtaining images from several different perspectives. At the same time, the camera pose relative to the electronic map at each shooting moment is obtained through LiDAR to obtain the camera pose corresponding to each image.

[0027] After obtaining several captured images and the camera pose corresponding to each image, defect detection can be performed, as shown in step S2 below.

[0028] S2. Perform defect detection processing on each image to obtain the pixel coordinates of the defect points in each image. In this embodiment, each image is input into a pre-trained visual defect detection model (such as a CNN model, residual network model, etc.) to obtain the pixel coordinates of the center of the defect points in each image. At the same time, the model will also output the coordinates of the two-dimensional bounding box and the confidence score of the defect detection for subsequent data retention. Thus, after the defect identification of each image is completed, coordinate transformation can be performed, that is, the pixel coordinates of the defect points in each image are projected onto the mask model of the target object. The process is shown in step S3 below.

[0029] S3. Based on the camera pose corresponding to each image, the pixel coordinates of the defect points in each image are mapped to the world coordinate system to obtain the three-dimensional coordinates of each defect point in the world coordinate system. In this embodiment, the intrinsic parameter matrix of the industrial camera and the camera pose corresponding to each image are used to back-project the pixel coordinates of each defect point onto the corresponding masking surface on the masking model, thereby calculating the three-dimensional position of the defect in the world coordinate system. The coordinate projection can be, but is not limited to, as shown in steps S31 to S35 below.

[0030] S31. Obtain the intrinsic parameter matrix of the industrial camera and the mask model corresponding to the target object; In this embodiment, the intrinsic parameter matrix and the mask model can be stored in advance and called when needed; Thus, after obtaining the camera intrinsic parameters and the mask model of the target object, the pixel coordinates can be transformed to the camera coordinate system, as shown in step S32 below.

[0031] S32. For any image, according to the intrinsic parameter matrix, convert the pixel coordinates of the defect points in the image into normalized camera coordinates; in specific implementation, for example, but not limited to, the following formula can be used to convert the pixel coordinates of the defect points in the image into normalized camera coordinates.

[0032] ; In the formula, For the camera coordinate matrix, ,in, Represents normalized camera coordinates. This indicates the transpose operation. This represents the intrinsic parameter matrix (which is a 3×3 matrix). The x-coordinate and y-coordinate of the pixel representing the defect point in any image.

[0033] Thus, based on the aforementioned formula, after obtaining the normalized camera coordinates corresponding to the defect points in any image, the ray in the camera coordinate system can be constructed, as shown in step S33 below.

[0034] S33. Construct the first ray equation of the normalized camera coordinates in the camera coordinate system; in specific applications, for example, but not limited to, the first ray equation can be constructed according to the following formula.

[0035] ; In the formula, This represents the three-dimensional coordinates of a defect point in any image in the camera coordinate system. For depth parameters, Let be the camera coordinate matrix; in this embodiment, the formula as a whole represents the first ray equation.

[0036] Thus, after constructing the ray equation of the defect point in the camera coordinate system, it is necessary to use the camera pose of any image to determine the rotation matrix and translation matrix between the camera coordinate system and the world coordinate system, so as to transform the first ray equation to the world coordinate system. The transformation process is shown in step S34 below.

[0037] S34. Based on the camera pose corresponding to any of the images, determine the rotation matrix and translation matrix between the camera coordinate system and the world coordinate system, and based on the rotation matrix and the translation matrix, transform the first ray equation to the world coordinate system to obtain the second ray equation; in specific implementation, it has been explained above that the extrinsic parameter relationship between the lidar and the industrial camera is pre-calibrated, and this extrinsic parameter relationship is essentially a 4×4 matrix, which contains rotation and translation vectors; therefore, in actual detection, when a defect point is obtained, the pixel coordinates of the defect point are transformed to the lidar coordinate system using the calibrated extrinsic parameter matrix T1 (i.e., multiplying the pixel coordinates by the extrinsic parameter matrix to obtain the defect coordinates in the lidar coordinate system, which can be represented as P1=T1×(u,v)); then, based on any of the aforementioned images... The camera pose T_w corresponding to the image is used to transform the defect coordinates in the lidar coordinate system to the world coordinate system (that is, multiply the two, P2=T_w×P1=T_w×[T1×(u,v)]). Finally, according to the associative law of matrix multiplication, the composite transformation from the camera coordinate system to the world coordinate system can be obtained, that is, P2=(T_w×T1)×(u,v). At this time, (T_w×T1) is the actual extrinsic parameter matrix of any image transformed to the world coordinate system. That is, the 3x3 submatrix in the upper left corner of the actual extrinsic parameter matrix is ​​the rotation matrix, and the 3x1 vector in the upper right corner is the translation matrix. Of course, the aforementioned method of calculating the rotation and translation matrices between the two coordinate systems based on the camera pose is a common technique for coordinate transformation, and its principle will not be elaborated here.

[0038] After obtaining the rotation and translation matrices, the equation of the first ray can be transformed to the world coordinate system. The transformation formula can be, but is not limited to, the following: ; In the formula, Let the three-dimensional coordinates of the defect point in any of the images be given in the world coordinate system. For rotation matrix, It is a translation matrix. This represents the transpose operator; in this embodiment, the formula as a whole represents the second ray equation.

[0039] Thus, after obtaining the second ray equation, the coordinates of the intersection point between the second ray equation and the corresponding masking surface of any image can be obtained, and the coordinates of the intersection point are the three-dimensional coordinates of the defect point in the world coordinate system in any image; wherein, the calculation process of the intersection point coordinates is as shown in step S35 below.

[0040] S35. Determine the masking surface of any image in the masking model, and calculate the intersection coordinates between the second ray equation and the masking surface, so as to use the intersection coordinates as the three-dimensional coordinates of the defect point in the world coordinate system in any image; in specific implementation, for example, but not limited to, the following steps S35a to S35c can be used to calculate the intersection coordinates.

[0041] S35a. Obtain the planar unit normal vector of the masking surface, and the projection distance from the camera to the masking surface when the industrial camera captures any of the images; in specific implementation, the projection distance from the camera to the masking surface when the industrial camera captures any of the images is essentially: the projection distance of the ray from the center point of the camera's optical center (the center point of any image) to the projection point of the corresponding masking surface in any image, in the direction of the planar unit normal vector; wherein, the center point of the camera's optical center can be first transformed to the world coordinate system to obtain its corresponding world coordinates; then, the signed distance between the masking surface and the origin of the world coordinate system (usually the center point of the bottom surface of the cuboid (the center of the car chassis)) is obtained; next, the dot product of the world coordinates and the planar unit normal vector is calculated, and the difference between the dot product and the signed distance is calculated; finally, the absolute value of the difference is taken to obtain the aforementioned projection distance.

[0042] After obtaining the projection distance and the unit normal vector of the plane, the plane equation of the masking surface can be constructed, as shown in step S35b below.

[0043] S35b. Based on the projection distance and the unit normal vector of the plane, construct the plane equation of the masking surface; in specific implementation, the plane equation is: In the formula, These represent the plane unit normal vector and the projected distance, respectively.

[0044] Thus, after constructing the plane equation of the masking surface, the coordinates of the intersection point of the two can be obtained by combining the second ray equation, as shown in step S35c below.

[0045] S35c. Using the plane equation and the second ray equation, solve for the coordinates of the intersection point; in specific implementation, first substitute the plane equation into the second ray equation to solve for the depth parameter, that is... Substitute into At this point, the depth parameter can be expressed as: ; After calculating the depth parameter based on the aforementioned formula, this depth parameter can be substituted back into the second ray equation, i.e.: Substitute into the equation: At this point, we can obtain for: ; Thus, the three-dimensional coordinates of the defect point in any image in the world coordinate system can be calculated using the aforementioned formula.

[0046] Therefore, through the aforementioned steps S31 to S35, the three-dimensional coordinates of the defect points in each image in the world coordinate system can be calculated; then, the data can be persistently stored, the process of which is as follows: S36. Obtain the image metadata corresponding to each image, where the image metadata includes timestamp and task ID.

[0047] S37. Store the image metadata corresponding to each image and the three-dimensional coordinates of the defect points in each image to the local SQLite database.

[0048] In this embodiment, a separate data storage architecture can be used, but is not limited to, to store the aforementioned data, i.e., a separate design of a main table and sub-tables. The main table stores basic metadata such as timestamps, task IDs, the aforementioned camera intrinsic parameter matrix, and extrinsic parameter matrix (including rotation and translation matrices), while the sub-table stores specific defect detection results, i.e., the two-dimensional bounding box coordinates, confidence scores, and three-dimensional coordinates of each defect point. At the same time, the extrinsic and intrinsic parameter matrices are stored in JSON format to achieve flexible storage and retrieval of complex geometric coordinates under multiple coordinate systems.

[0049] After data storage is completed, defect points can be clustered to group the defect points corresponding to the same defect from different perspectives into a single cluster. The process is shown in step S4 below.

[0050] S4. Based on the 3D coordinates of each defect point, the DBSCAN clustering algorithm is used to cluster all defect points to obtain several defect point clusters. Multiple defect points within any cluster correspond to the same defect location, and the images corresponding to these defect points are taken from different viewing angles. In practice, the 3D coordinates of all defect points are read from the database. The DBSCAN clustering algorithm is applied to cluster the defect points using a preset distance threshold ε (e.g., 5mm). Each cluster is considered as multiple observations of the same physical defect from multiple perspectives. The specific implementation logic may include, but is not limited to: dynamically calculating all defect data within the most recent time window (within a specified time window range) based on the task timestamp attribute, using this as clustering input to achieve intelligent data filtering based on time range; then, implementing the DBSCAN algorithm and configuring neighborhood radius and minimum sample number parameters to support real-time parameter adjustment to adapt to different detection accuracy requirements; finally, establishing a many-to-many mapping relationship between the clustering results and the original detection items through a mapping table to ensure the traceability of the clustering process and the ability to verify the results.

[0051] Of course, the DBSCAN algorithm is a commonly used clustering technique, and its principle will not be elaborated here.

[0052] After clustering the defect points, the actual three-dimensional coordinates of each defect location can be determined, as shown in step S5 below.

[0053] S5. Based on several defect point clusters, determine the actual three-dimensional coordinates of different defect locations; in specific implementation, for example, but not limited to, the three-dimensional coordinate center point of each cluster can be calculated as the actual three-dimensional coordinates, and the cluster size and cluster unique identifier can be stored; further, for any defect point cluster, the cluster center or the first occurrence point of any defect point cluster can be used as the representative position (i.e., actual three-dimensional coordinates) of the defect corresponding to any defect point cluster.

[0054] After obtaining the actual three-dimensional coordinates of each defect location, multi-view visualization of the defects can be performed, as shown in steps S6 to S8 below.

[0055] S6. Based on the actual three-dimensional coordinates of each defect location, map each defect location onto the view plane of the mask model of the target object in the world coordinate system to obtain a defect distribution model diagram; in specific implementation, it is possible, but not limited to, to calculate the accurate projection of the actual three-dimensional coordinates of the defect location onto different view planes on the mask model based on the actual size parameters (length, width, height) and center position coordinates (i.e., origin) of the target object, such as supporting dynamic switching of four views (front, back, left, right).

[0056] The masking model of the target object has been constructed, and the spatial plane equations of each masking surface on the masking model, namely the six outer surfaces (usually focusing on the four vertical surfaces of front, back, left, and right), can be determined. That is, the equation of each plane can be determined by its normal vector and the signed distance to the origin. This information can be directly calculated or derived from the known dimensions, center position, and pose of each surface. Then, for the actual 3D coordinates of each defect location, the view plane to which the defect location belongs is first determined, such as whether it belongs to the left side, right side, or front side. Specifically, the actual 3D coordinates of each defect location and the vertical distance to each view plane can be calculated. The view plane to which the defect location belongs is the view plane with a vertical distance less than a preset threshold (such as 1mm). For defect locations that are close to edges and where two planes intersect, the distance to both planes may be close to 0. Therefore, it can be displayed on either plane.

[0057] Then, the actual three-dimensional coordinates of each defect location are transformed into a two-dimensional plane coordinate system based on the view plane. At this point, each defect location can be projected onto the corresponding view plane. A schematic diagram of the defect distribution model can be found in [reference needed]. Figure 2 As shown; specifically, perspective transformation can be used for projection conversion; of course, the common technique of transforming three-dimensional coordinates to a two-dimensional plane is projection mapping, and its principle will not be elaborated here.

[0058] After the defect distribution analysis is completed, the detection coverage analysis can be performed, as shown in step S7 below.

[0059] S7. Using several images, determine the scanning range of the industrial camera on the mask model during each defect detection, and generate a detection coverage display model based on the scanning range during each defect detection; in specific implementation, for example, but not limited to, the following steps S71 to S73 can be used to determine the scanning range during each defect detection.

[0060] S71. Obtain the coordinates of the four corner points of the camera image corresponding to each image, wherein the coordinates of the four corner points of any image are the coordinates of the four endpoints of that image.

[0061] After obtaining the coordinates of the four corner points corresponding to each image, they can be projected onto the world coordinate system, as shown in step S72 below.

[0062] S72. Transform the coordinates of the four corner points of each image to the world coordinate system to obtain the projection point of each corner point in each image on the mask model; in this embodiment, the transformation process of the corner point coordinates can be referred to the aforementioned step S3, and the process will not be repeated.

[0063] After obtaining the projection points of each corner point in each image onto the mask model, the area enclosed by the projection points can be used as the scanning area, as shown in step S73 below.

[0064] S73. The area enclosed by the projection points corresponding to the four corner points in each image is taken as the scanning area of ​​each image on the mask model, so that each scanning area is taken as the shooting scanning range of the industrial camera on the mask model when shooting each image; in this embodiment, an image corresponds to four corner points, and the four corner points correspond to four projection points in the mask model; thus, the area enclosed by the four projection points is taken as the shooting scanning range of the industrial camera on the mask model when shooting the image.

[0065] Thus, through the aforementioned steps S71 to S73, the scanning range of the industrial camera on the mask model when capturing each image can be determined. Then, based on this, a detection coverage display model diagram can be generated, as shown in the following steps S74 to S77.

[0066] S74. Several images are classified to obtain multiple image subsets, wherein each image subset corresponds to a masking surface on the masking model; in this embodiment, it is equivalent to determining the image corresponding to each masking surface, and then adding the scanning area of ​​the image corresponding to each masking surface to the corresponding masking surface to form a detection coverage surface, the process of which is shown in steps S75 and S76 below.

[0067] S75. For any masking surface, obtain the scanning area of ​​each image in the image subset corresponding to the masking surface as the detection coverage area.

[0068] S76. Add each detection coverage area to any one of the masking surfaces to obtain the detection coverage surface corresponding to the any one of the masking surfaces. After polling all masking surfaces, obtain the detection coverage surfaces corresponding to all masking surfaces. In specific implementation, the coordinates of the projection points of the corner points in each image in the image subset corresponding to the any one of the masking surfaces are known. Therefore, based on the coordinates of the projection points of the four corner points in the image, a coordinate region can be determined on the any one of the masking surfaces, and this region serves as the detection coverage area. Thus, by displaying the detection coverage areas of all images corresponding to the any one of the masking surfaces on the any one of the masking surfaces, the detection coverage surface corresponding to the any one of the masking surfaces can be obtained. See the schematic diagram below. Figure 3 As shown, Figure 3 The blue portion in each face is composed of multiple detection coverage areas.

[0069] Thus, after obtaining the detection coverage surface corresponding to each masking surface, each detection coverage surface can be used to form a detection coverage display model diagram, as shown in step S77 below.

[0070] S77. Using the detection coverage surfaces corresponding to all masking surfaces, generate the detection coverage display model diagram; in this embodiment, the detection coverage display model diagram can be found in [reference needed]. Figure 3 As shown, from Figure 3 The detection coverage of each masking surface on the masking model can be seen intuitively, that is... Figure 3 The blue part in the image.

[0071] After generating the detection coverage display model diagram through the aforementioned steps S71 to S77, it can be visualized together with the defect distribution model diagram, so that staff can intuitively understand the defects on the surface of the target object. The process is shown in step S8 below.

[0072] S8. Visualize the defect distribution model diagram and the detection coverage display model diagram to complete the three-dimensional visualization and positioning of surface defects of the target object.

[0073] In practical implementation, to address the issue of the inability to perform consistency comparisons across multiple scans in traditional technologies, this embodiment also provides a visualization technology for comparing the overlap rate of multi-task scan trajectories. This involves evaluating the trajectory overlap rate of multi-task scans through quantitative analysis of different tasks within the same space. The trajectory consistency comparison process can be, but is not limited to, the steps described below: Step 1: Obtain the image set captured by the industrial camera when performing multiple defect detection tasks, wherein the movement trajectory of the industrial camera when performing each defect detection task is the preset trajectory; in this embodiment, each defect detection task moves along the same trajectory; thus, after obtaining the image set corresponding to multiple defect detection tasks, the task trajectory can be generated, and the process is shown in Step 2 below.

[0074] Step 2: Generate the scanning trajectory corresponding to each defect detection task based on the camera imaging center point corresponding to each image in each image set. In specific applications, take any image set as an example to illustrate the process of generating the scanning trajectory. It can be, but is not limited to: (1) For any image set, convert the camera imaging center point (i.e., image center point) corresponding to each image in the image set to the world coordinate system to obtain the projection center point of each camera imaging center point on the mask model (the conversion process can be referred to the aforementioned step S3, and will not be repeated here); (2) According to the shooting order of each image in the image set, connect the projection center points corresponding to the camera imaging center points of each image in sequence to generate the scanning trajectory of the defect detection task corresponding to the image set.

[0075] Thus, based on the aforementioned method, after generating the scanning trajectory corresponding to each defect detection task, trajectory comparison can be performed, as shown in the third step below.

[0076] Step 3: Generate a scan trajectory comparison chart based on each scan trajectory.

[0077] In this embodiment, the process of generating the scan trajectory comparison map is as follows: (1) For the projection center point at any same detection position in each scanning trajectory, calculate the scanning error distance between each projection center point at any same detection position, and obtain several scanning error distances after polling all projection center points in each scanning trajectory; in specific implementation, each projection center point in each scanning trajectory can be numbered according to the connection order. At this time, the projection center point with the same number in each scanning trajectory is regarded as the projection center point at the same detection position; of course, the shooting time of the image corresponding to each projection center point in the scanning trajectory can also be recorded, and then the projection center point whose difference between the shooting time and the task start time is less than the preset value is regarded as the projection center point at the same detection position. For example, if there are two scanning trajectories, and the difference between the second point in the two scanning trajectories and the corresponding task start time is less than the preset value (such as less than 5ms), then the second point in the two scanning trajectories can be regarded as the projection center point at the same detection position.

[0078] Thus, for any projection center point at the same detection position, the coordinate distance between each projection center point can be calculated, and then the coordinate distance is used as the scanning error distance; where there are more than two scanning trajectories, the average of the coordinate distances between each projection center point is used as the scanning error distance; in this way, after polling all points in each scanning trajectory, several scanning error distances can be obtained; then, based on several scanning error distances, the average error distance can be calculated, and the process is shown in the following step (2).

[0079] (2) Calculate the mean between several scanning error distances to obtain the average error distance; in this embodiment, the average error distance is used as a trajectory comparison index, which is used to measure the trajectory overlap rate between scanning trajectories. In this way, each scanning trajectory and the average error distance can be visualized on the masking model, and a scanning trajectory comparison map can be generated. The process is as shown in the following steps (3).

[0080] (3) Add the color of each scan trajectory to the mask model, and add the average error distance as a trajectory comparison index to the mask model to generate the scan trajectory comparison map; in this embodiment, the smaller the average error distance, the higher the overlap rate, and for example, the coloring corresponding to different scan trajectories is different, the schematic diagram of which can be found in [reference needed]. Figure 4 As shown, this invention enables precise quantitative evaluation of trajectory overlap rate, overcoming the inherent limitation of traditional techniques in assessing the consistency of behavior across multiple scans. This facilitates the discovery of differences and problems in scanning strategies. Therefore, through the multi-view fusion-based three-dimensional localization and visualization method for surface defects described in detail in steps S1 to S8 above, the present invention has the following beneficial effects: (1) By fusing camera intrinsic parameters, pose and object surface (masking surface) geometric information, a precise projection transformation model from two-dimensional pixel coordinates to three-dimensional coordinates of the object surface is constructed, which solves the problem of accurate positioning of defects on the actual object surface and realizes the accurate positioning of defects in a unified world coordinate system.

[0081] (2) It supports intuitive visualization of the distribution of defects across the entire surface, accurately counts the coverage of defect scanning, and provides quantifiable evaluation indicators for detection quality.

[0082] (3) It supports the comparison of multiple tasks in the same space. By calculating the difference between scanning points, it realizes the accurate quantitative evaluation of trajectory overlap rate, which makes it easier to discover the differences and problems of scanning strategies.

[0083] (4) The DBSCAN density clustering algorithm is introduced into the industrial multi-view defect detection process to automatically identify and fuse the same defect point under different views. This method overcomes the limitation of traditional methods (such as K-means) that require a preset number of clusters and realizes automatic and efficient deduplication of repeated defects across views.

[0084] like Figure 5 As shown, the second aspect of this embodiment provides a hardware device for implementing the multi-view fusion-based three-dimensional localization and visualization method for surface defects described in the first aspect of the embodiment, comprising: The acquisition unit is used to acquire several images captured by the industrial camera when it performs defect detection on the target object along a preset trajectory, as well as the camera pose corresponding to each image.

[0085] The defect detection unit is used to perform defect detection processing on each image to obtain the pixel coordinates of the defect points in each image.

[0086] The projection unit is used to map the pixel coordinates of the defect points in each image to the world coordinate system based on the camera pose corresponding to each image, so as to obtain the three-dimensional coordinates of each defect point in the world coordinate system.

[0087] The clustering unit is used to cluster all defect points based on the three-dimensional coordinates of each defect point and using the DBSCAN clustering algorithm to obtain several defect point clusters. In any defect point cluster, multiple defect points correspond to the same defect location, and the images corresponding to multiple defect points are taken from different perspectives.

[0088] The defect location determination unit is used to determine the actual three-dimensional coordinates of different defect locations based on several defect point clusters.

[0089] The defect analysis unit is used to map each defect location to the view plane of the masking model of the target object in the world coordinate system based on the actual three-dimensional coordinates of each defect location, so as to obtain a defect distribution model diagram.

[0090] The defect analysis unit is used to determine the scanning range of the industrial camera on the mask model during each defect detection using several images, and to generate a detection coverage display model map based on the scanning range during each defect detection.

[0091] The defect analysis unit is also used to visualize the defect distribution model diagram and the detection coverage display model diagram to complete the three-dimensional visualization and positioning of surface defects of the target object.

[0092] The working process, working details and technical effects of the device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0093] like Figure 6 As shown, the third aspect of this embodiment provides another surface defect three-dimensional localization and visualization device based on multi-view fusion. Taking the device as an electronic device as an example, it includes: a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the surface defect three-dimensional localization and visualization method based on multi-view fusion as described in the first aspect of the embodiment.

[0094] For specific examples, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.

[0095] In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. For example, the processor may not be limited to microprocessors of the STM32F105 series, reduced instruction set computer (RISC) microprocessors, x86 architecture processors, or processors with integrated neural network processing units (NPUs). The transceiver may be, but is not limited to, a Wi-Fi transceiver, a Bluetooth transceiver, a General Packet Radio Service (GPRS) transceiver, a ZigBee (a low-power LAN protocol based on the IEEE 802.15.4 standard) transceiver, a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. Furthermore, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0096] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0097] The fourth aspect of this embodiment provides a storage medium that stores instructions for the three-dimensional localization and visualization method of surface defects based on multi-view fusion as described in the first aspect of the embodiment. That is, the storage medium stores instructions that, when executed on a computer, perform the three-dimensional localization and visualization method of surface defects based on multi-view fusion as described in the first aspect of the embodiment.

[0098] The storage medium refers to a carrier for storing data, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0099] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0100] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the three-dimensional localization and visualization method for surface defects based on multi-view fusion as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0101] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for three-dimensional localization and visualization of surface defects based on multi-view fusion, characterized in that, include: Acquire several images captured by an industrial camera along a preset trajectory to detect defects in a target object, as well as the camera pose corresponding to each image; Each image is processed for defect detection to obtain the pixel coordinates of the defect points in each image; Based on the camera pose corresponding to each image, the pixel coordinates of the defect points in each image are mapped to the world coordinate system to obtain the three-dimensional coordinates of each defect point in the world coordinate system. Based on the three-dimensional coordinates of each defect point, the DBSCAN clustering algorithm is used to cluster all defect points to obtain several defect point clusters. Among them, multiple defect points in any defect point cluster correspond to the same defect location, and the images corresponding to multiple defect points are taken from different perspectives. Based on several defect point clusters, the actual three-dimensional coordinates of different defect locations are determined; Based on the actual three-dimensional coordinates of each defect location, each defect location is mapped to the view plane of the masking model of the target object in the world coordinate system to obtain a defect distribution model diagram. Using several images, the scanning range of the industrial camera on the mask model during each defect detection is determined, and a detection coverage display model map is generated based on the scanning range during each defect detection. The defect distribution model and the detection coverage display model are visualized to complete the three-dimensional visualization and localization of surface defects of the target object.

2. The method according to claim 1, characterized in that, Based on the camera pose corresponding to each image, the pixel coordinates of the defect points in each image are mapped to the world coordinate system, including: Obtain the intrinsic parameter matrix of the industrial camera and the mask model corresponding to the target object; For any image, the pixel coordinates of the defect points in the image are converted into normalized camera coordinates according to the intrinsic parameter matrix. Construct the first ray equation of the normalized camera coordinates in the camera coordinate system; Based on the camera pose corresponding to any image, determine the rotation matrix and translation matrix between the camera coordinate system and the world coordinate system, and based on the rotation matrix and the translation matrix, transform the first ray equation to the world coordinate system to obtain the second ray equation; The masking surface of any image in the masking model is determined, and the coordinates of the intersection point between the second ray equation and the masking surface are calculated. The coordinates of the intersection point are then used as the three-dimensional coordinates of the defect point in any image in the world coordinate system.

3. The method according to claim 2, characterized in that, Based on the intrinsic parameter matrix, the pixel coordinates of defect points in any image are converted into normalized camera coordinates, including: The pixel coordinates of defect points in any of the images are converted into normalized camera coordinates according to the following formula; ； In the formula, For the camera coordinate matrix, ,in, Represents normalized camera coordinates. This represents the transpose operator. Represents the intrinsic parameter matrix. The x-coordinate and y-coordinate of the pixel representing the defect point in any image.

4. The method according to claim 2, characterized in that, The first ray equation of the normalized camera coordinates in the camera coordinate system is constructed, including: The first ray equation is constructed according to the following formula; ； In the formula, This represents the three-dimensional coordinates of a defect point in any image in the camera coordinate system. For depth parameters, The camera coordinate matrix; Accordingly, based on the rotation matrix and the translation matrix, the first ray equation is transformed to the world coordinate system to obtain the second ray equation, which includes: Transform the first ray equation to the world coordinate system according to the following formula; ； In the formula, Let the three-dimensional coordinates of the defect point in any of the images be given in the world coordinate system. For rotation matrix, It is a translation matrix. This indicates the transpose operation.

5. The method according to claim 2, characterized in that, The coordinates of the intersection point between the second ray equation and the masking surface are calculated, including: Obtain the planar unit normal vector of the masking surface, and the projection distance from the industrial camera to the masking surface when the industrial camera captures any of the images; Based on the projection distance and the unit normal vector of the plane, the plane equation of the masking surface is constructed; The coordinates of the intersection point are obtained by using the plane equation and the second ray equation.

6. The method according to claim 1, characterized in that, Using several images, the scanning range of the industrial camera on the mask model during each defect inspection was determined, including: Obtain the coordinates of the four corner points of the camera image corresponding to each image, where the coordinates of the four corner points of any image are the coordinates of the four endpoints of that image; Transform the coordinates of the four corner points of each image to the world coordinate system to obtain the projection point of each corner point in each image onto the masking model; The area enclosed by the projection points corresponding to the four corner points in each image is taken as the scanning area of ​​each image on the mask model, and each scanning area is taken as the scanning range of the industrial camera on the mask model when capturing each image. Accordingly, based on the scanning range captured during each defect detection, a detection coverage display model diagram is generated, which includes: Several images are classified to obtain multiple image subsets, where each image subset corresponds to a masking surface on the masking model; For any masking surface, obtain each image in the image subset corresponding to that masking surface, and the scanning area on that masking surface, as the detection coverage area; Each detection coverage area is added to any of the masking surfaces to obtain the detection coverage surface corresponding to any of the masking surfaces, and after all the masking surfaces have been polled, the detection coverage surfaces corresponding to all the masking surfaces are obtained. The detection coverage display model diagram is generated using the detection coverage surfaces corresponding to all masking surfaces.

7. The method according to claim 1, characterized in that, After obtaining the three-dimensional coordinates of each defect point in the world coordinate system, the method further includes: Obtain the image metadata corresponding to each image, including timestamp and task ID; The image metadata corresponding to each image and the three-dimensional coordinates of the defect points in each image are stored in the local SQLite database.

8. The method according to claim 1, characterized in that, The method further includes: Acquire an image set captured by an industrial camera while performing multiple defect detection tasks, wherein the movement trajectory of the industrial camera while performing each defect detection task is the preset trajectory; Based on the camera imaging center point corresponding to each image in each image set, generate the scanning trajectory corresponding to each defect detection task; A comparison chart of scan trajectories is generated based on each scan trajectory.

9. The method according to claim 8, characterized in that, Based on the camera imaging center point corresponding to each image in each image set, the scanning trajectory corresponding to each defect detection task is generated, including: For any image set, the camera imaging center points corresponding to each image in the image set are transformed to the world coordinate system to obtain the projection center points of each camera imaging center point on the mask model. According to the shooting order of each image in any image set, the projection center points corresponding to the camera imaging center points of each image are connected sequentially to generate the scanning trajectory of the defect detection task corresponding to any image set. Accordingly, based on each scan trajectory, a scan trajectory comparison map is generated, which includes: For the projection center point at any same detection position in each scanning trajectory, calculate the scanning error distance between each projection center point at any same detection position, and after polling all projection center points in each scanning trajectory, obtain several scanning error distances. The mean error distance is obtained by calculating the average value among several scanning error distances. Each scan trajectory is colored and added to the mask model, and the average error distance is added to the mask model as a trajectory comparison index to generate the scan trajectory comparison map, wherein different scan trajectories correspond to different colors.

10. A device for three-dimensional localization and visualization of surface defects based on multi-view fusion, characterized in that, include: The acquisition unit is used to acquire several images captured by the industrial camera when it performs defect detection on the target object along a preset trajectory, as well as the camera pose corresponding to each image. The defect detection unit is used to perform defect detection processing on each image to obtain the pixel coordinates of the defect points in each image; The projection unit is used to map the pixel coordinates of the defect points in each image to the world coordinate system based on the camera pose corresponding to each image, so as to obtain the three-dimensional coordinates of each defect point in the world coordinate system. The clustering unit is used to cluster all defect points based on the three-dimensional coordinates of each defect point and using the DBSCAN clustering algorithm to obtain several defect point clusters. Among them, multiple defect points in any defect point cluster correspond to the same defect location, and the multiple defect points correspond to different shooting angles of the images. The defect location determination unit is used to determine the actual three-dimensional coordinates of different defect locations based on several defect point clusters; The defect analysis unit is used to map each defect location to the view plane of the masking model of the target object in the world coordinate system based on the actual three-dimensional coordinates of each defect location, so as to obtain a defect distribution model diagram. The defect analysis unit is used to determine the scanning range of the industrial camera on the mask model during each defect detection using several images, and to generate a detection coverage display model map based on the scanning range during each defect detection. The defect analysis unit is also used to visualize the defect distribution model diagram and the detection coverage display model diagram to complete the three-dimensional visualization and positioning of surface defects of the target object.

Images