Method and system for performing three-dimensional flaw detection on industrial product based on multi-source data

Abstract

The invention discloses a method and system for performing three-dimensional flaw detection on industrial products based on multi-source data, and belongs to the technical field of industrial product flaw detection, and the method comprises the following steps: determining the spatial resolution of a training sample and an ultrasonic flaw detection probe; segmenting the world coordinate system based on the spatial resolution to obtain a three-dimensional space containing a plurality of discrete units; configuring discretized point cloud data in the training sample in a three-dimensional space; assigning the discrete units with the point cloud data as a first type of units, and assigning the discrete units without the point cloud data as a second type of units; obtaining an internal flaw detection model based on the assigned discrete unit; acquiring an external flaw detection model through the external image data of the training sample; determining a to-be-detected workpiece, and inputting the internal point cloud data of the to-be-detected workpiece into the internal flaw detection model to obtain an internal flaw detection result; and inputting the external image data of the to-be-detected workpiece into the external flaw detection model to obtain an external flaw detection result.

Description

Technical Field

[0001] This application belongs to the field of industrial flaw detection technology, specifically relating to a method and system for three-dimensional flaw detection of industrial products based on multi-source data. Background Technology

[0002] The mass production of industrial products like automotive mechanical parts, with their solid, sealed, and heterogeneous structures, makes internal quality testing difficult. In many cases, only one-time, destructive sampling inspections are feasible. This not only fails to achieve full inspection but is also destructive, clearly indicating a flaw in the testing method.

[0003] Furthermore, for the problem of external object inspection, three-dimensional damage detection still needs to be performed by human eyes. This requires high standards of human eyesight during the production process and makes it impossible to perform full inspection. Summary of the Invention

[0004] Purpose of the invention: This application develops a method and system for three-dimensional flaw detection of industrial products based on multi-source data, in order to solve the above-mentioned technical problems.

[0005] Technical Solution: In a first aspect, this application provides a method for three-dimensional flaw detection of industrial products based on multi-source data, applied to a flaw detection equipment. The flaw detection equipment includes a robotic arm, an ultrasonic flaw detection probe disposed on the robotic arm for acquiring point cloud data of the target's interior, and a camera for acquiring image data of the target's exterior. The method includes:

[0006] Determine the spatial resolution of the training samples and the ultrasonic flaw detection probe;

[0007] The world coordinate system is segmented based on the spatial resolution to obtain a three-dimensional space containing multiple discrete units;

[0008] The discretized point cloud data within the training samples is configured in the three-dimensional space;

[0009] Discrete units containing the point cloud data are assigned to the first type of unit, and discrete units without the point cloud data are assigned to the second type of unit.

[0010] The internal flaw detection model is obtained based on the assigned discrete units;

[0011] An external flaw detection model is obtained using the external image data of the training samples;

[0012] The workpiece to be inspected is identified, and the internal point cloud data of the workpiece to be inspected is input into the internal flaw detection model to obtain the internal flaw detection results; the external image data of the workpiece to be inspected is input into the external flaw detection model to obtain the external flaw detection results.

[0013] In some embodiments, the training samples include qualified samples, and the step of obtaining the internal flaw detection model based on the assigned discrete units includes:

[0014] In a plurality of qualified samples, the average value of the discrete units at the same position is obtained as the standard value;

[0015] Among the multiple qualified samples, the values ​​of the same discrete units in any two qualified samples are compared, and the average value of the number of discrete units with differences is determined as the standard range value.

[0016] The internal flaw detection model is obtained based on the standard value and the standard range value.

[0017] In some embodiments, the step of inputting the internal point cloud data of the workpiece to be inspected into the internal flaw detection model to obtain the internal flaw detection results includes:

[0018] The discretized point cloud data of the workpiece to be inspected is configured in the three-dimensional space;

[0019] In the workpiece to be inspected, discrete units containing the point cloud data are assigned the first type of unit, and discrete units without the point cloud data are assigned the second type of unit.

[0020] If the number of discrete units in the workpiece to be inspected that differ from the standard value is less than or equal to the standard range value, the workpiece to be inspected is determined to be a qualified product.

[0021] In some embodiments, the training samples include qualified samples, and the step of obtaining the external flaw detection model includes:

[0022] In multiple qualified sample RGB format images, the average value of the R channel, the average value of the G channel, and the average value of the B channel for the same pixel are obtained to construct standard channel values;

[0023] In multiple qualified sample RGB format images, obtain the total difference of the same pixel in the R channel, G channel and B channel in any two qualified samples, and the variance of multiple total differences of the same pixel;

[0024] The average value of the variance of multiple pixels is obtained to construct a standard channel range value;

[0025] The external flaw detection model is obtained based on the standard channel range value.

[0026] In some embodiments, the step of obtaining the internal flaw detection model based on the assigned discrete unit includes:

[0027] The first type of unit is used as the center to encompass multiple discrete units outward to obtain the flaw detection convolution kernel;

[0028] The flaw detection convolution kernel is layered along a preset direction to obtain multiple layered convolutional surfaces;

[0029] Following the layering order along the preset direction, multiple layered convolutional surfaces are sequentially spliced ​​together to obtain spliced ​​convolutional surfaces;

[0030] The internal flaw detection model to be trained is determined, and the internal flaw detection model is trained based on the spliced ​​convolutional surface of the training samples to obtain the trained internal flaw detection model.

[0031] In some embodiments, the internal flaw detection model includes an MLP neural network, the MLP neural network comprising:

[0032] The input layer has the same number of neurons as the number of discrete units in the concatenated convolutional surface.

[0033] The intermediate layer has twice the number of neurons as the concatenated convolutional surface discrete units.

[0034] The output layer comprises three neurons, which are used to represent qualified, unqualified, and pending, respectively.

[0035] In some embodiments, the external flaw detection model constrains features in both dual-temporal feature extraction and feature fusion; the backbone network of the external flaw detection model is equipped with a nonlocal feature pyramid network (NL-FPN) at its center to extract and fuse multi-scale features, so as to construct a densely connected feature fusion module (DFM) to robustly fuse dual-temporal features;

[0036] External flaw detection results are obtained by comparing the external image data of qualified samples and workpieces under the same pose using an external flaw detection model.

[0037] In some embodiments, after assigning a value to the discrete unit, the method further includes:

[0038] Multiple discrete units are enclosed outward from the first type of unit to obtain a filter-enhanced convolution kernel;

[0039] In the filter-enhanced convolution kernel, in response to the existence of two first-class units with an L2 norm distance less than or equal to 2, the discrete unit on the line connecting the two first-class units is assigned as a first-class unit;

[0040] In the filtered and enhanced convolutional kernel, in response to the existence of only one first-type unit, the first-type unit is assigned the value of the second-type unit.

[0041] Secondly, embodiments of this application also provide a system for three-dimensional flaw detection of industrial products based on multi-source data, applied to a flaw detection equipment. The flaw detection equipment includes a robotic arm, an ultrasonic flaw detection probe disposed on the robotic arm for acquiring point cloud data inside the target, and a camera for acquiring image data outside the target. The system includes:

[0042] The training sample configuration module is used to determine the spatial resolution of the training sample and the ultrasonic flaw detection probe;

[0043] A spatial construction module is used to segment the world coordinate system based on the spatial resolution to obtain a three-dimensional space containing multiple discrete units;

[0044] A point cloud configuration module is used to configure the discretized point cloud data within the training samples in the three-dimensional space.

[0045] The assignment module assigns the discrete units containing the point cloud data to the first type of unit, and assigns the discrete units without the point cloud data to the second type of unit;

[0046] The external flaw detection model acquisition module is used to acquire the internal flaw detection model based on the assigned discrete unit.

[0047] The internal flaw detection model acquisition module acquires the external flaw detection model through the external image data of the training samples;

[0048] The flaw detection result module is used to determine the workpiece to be inspected, input the internal point cloud data of the workpiece to be inspected into the internal flaw detection model to obtain the internal flaw detection result; and input the external image data of the workpiece to be inspected into the external flaw detection model to obtain the external flaw detection result.

[0049] Beneficial Effects: Compared with existing technologies, the present application provides a method for three-dimensional flaw detection of industrial products based on multi-source data. This method includes determining the spatial resolution of training samples and ultrasonic flaw detection probes; segmenting the world coordinate system based on the spatial resolution to obtain a three-dimensional space containing multiple discrete units; configuring the discretized point cloud data within the training samples in the three-dimensional space; assigning discrete units with point cloud data as first-class units and discrete units without point cloud data as second-class units; obtaining an internal flaw detection model based on the assigned discrete units; obtaining an external flaw detection model using the external image data of the training samples; determining the workpiece to be inspected and inputting the internal point cloud data of the workpiece to be inspected into the internal flaw detection model to obtain internal flaw detection results; and inputting the external image data of the workpiece to be inspected into the external flaw detection model to obtain external flaw detection results. This application provides ultrasonic flaw detection and visual appearance inspection, performing simultaneous multi-source flaw detection on industrial products using internal and external data. Internal quality inspection can be completed without damaging the workpiece, replacing human eyes for external quality inspection, thus improving detection accuracy and efficiency. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A flowchart illustrating the steps of a method for three-dimensional flaw detection of industrial products based on multi-source data, provided in an embodiment of this application.

[0052] Figure 2 A flowchart illustrating the steps of obtaining the first internal flaw detection model based on the assigned discrete units in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in this application embodiment;

[0053] Figure 3 A flowchart illustrating the steps of obtaining internal flaw detection results using a first internal flaw detection model in a method for three-dimensional flaw detection of industrial products based on multi-source data, as provided in this application embodiment;

[0054] Figure 4 A flowchart illustrating the steps of obtaining a first external flaw detection model in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in this application embodiment;

[0055] Figure 5 A flowchart illustrating the steps of obtaining a second internal flaw detection model based on assigned discrete units in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in this application embodiment;

[0056] Figure 6 A flowchart illustrating the steps of filtering and enhancing discrete units in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in this application embodiment;

[0057] Figure 7 A module connection diagram of a system for three-dimensional flaw detection of industrial products based on multi-source data, provided in an embodiment of this application;

[0058] Figure 8 A schematic diagram showing the location of the flaw detection equipment in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in the embodiments of this application;

[0059] Figure 9 A schematic diagram of the electrical relationships of the flaw detection equipment in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in the embodiments of this application;

[0060] Figure 10A schematic diagram of the appearance of the handheld control and inspection terminal of the flaw detection equipment in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in the embodiments of this application;

[0061] Figure 11 A schematic diagram illustrating the acquisition of the flaw detection convolution kernel in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in the embodiments of this application;

[0062] Figure 12 A flowchart illustrating the steps of acquiring three-dimensional data using a flaw detection device in a method for three-dimensional flaw detection of industrial products based on multi-source data provided in this application embodiment;

[0063] Figure 13 A diagram showing the relationship between different models selected based on the amount of data in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in the embodiments of this application.

[0064] Figure 14 The flowchart of the program for obtaining the first internal flaw detection model based on the assigned discrete unit in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in the embodiments of this application;

[0065] Figure 15 The flowchart of the procedure for internal flaw detection using the first internal flaw detection model MDA-1 in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in the embodiments of this application;

[0066] Figure labels: 10, Training sample configuration module; 20, Spatial construction module; 30, Point cloud configuration module; 40, Assignment module; 50, External flaw detection model acquisition module; 60, Internal flaw detection model acquisition module; 70, Flaw detection result module. Detailed Implementation

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0068] The mass production of industrial products like automotive mechanical parts, with their solid, sealed, and heterogeneous structures, makes internal quality testing difficult. In many cases, only one-time, destructive sampling inspections are feasible. This not only fails to achieve full inspection but is also destructive, clearly indicating a flaw in the testing method.

[0069] There are many existing methods for internal flaw detection of objects, including ultrasonic testing, radiographic testing, magnetic particle testing, penetrant testing, eddy current testing, acoustic emission testing, optical holographic testing, and infrared thermal imaging testing. Among these, ultrasonic testing has become the mainstream internal flaw detection solution due to its minimal environmental impact and relatively relaxed operating environment requirements. It provides an image with depth information; however, for 3D reconstruction, the user needs to obtain the pose of the ultrasonic probe.

[0070] Furthermore, for the problem of external object inspection, three-dimensional damage detection still needs to be performed by human eyes. This requires high standards of human eyesight during the production process and makes it impossible to perform full inspection.

[0071] Furthermore, with increasingly stringent production standards, people not only want to conduct full inspections on every product, but also want to retain relevant information about them before they leave the factory, and in the event of quality problems, be able to use objective digital 3D reconstruction to show the problematic areas to colleagues, thereby facilitating quality management.

[0072] Finally, since collecting fault samples for different types of products is difficult, people hope to start with small samples and use it quickly. During use, they can continuously collect fault samples and improve the accuracy of the algorithm to achieve the effect of adaptive sample quantity.

[0073] Since ultrasonic flaw detection data is an image containing depth information, further 3D reconstruction requires the user to obtain the probe's pose. Therefore, a robotic arm needs to be designed and used to move the probe, accurately acquiring the probe pose corresponding to each ultrasonic image, thus enabling 3D reconstruction. Similarly, for quality inspection of external surfaces, it is also necessary to acquire and match the camera pose corresponding to the images captured by the camera.

[0074] Therefore, there is an urgent need for a system that can easily control the movement of a robotic arm, adjust the pose of the ultrasonic flaw detector and RGB camera on the arm, collect corresponding images, and then perform anomaly detection. Furthermore, the system can adapt to the number of samples and be handheld for convenient on-site communication with colleagues.

[0075] In view of this, embodiments of this application provide a method for three-dimensional flaw detection of industrial products based on multi-source data, including determining the spatial resolution of training samples and ultrasonic flaw detection probes; segmenting the world coordinate system based on the spatial resolution to obtain a three-dimensional space containing multiple discrete units; configuring the discretized point cloud data inside the training samples in the three-dimensional space; assigning discrete units with point cloud data as first-type units and discrete units without point cloud data as second-type units; obtaining an internal flaw detection model based on the assigned discrete units; obtaining an external flaw detection model using the external image data of the training samples; determining the workpiece to be inspected and inputting the internal point cloud data of the workpiece to be inspected into the internal flaw detection model to obtain internal flaw detection results; inputting the external image data of the workpiece to be inspected into the external flaw detection model to obtain external flaw detection results. This application provides ultrasonic flaw detection and visual appearance inspection, performing simultaneous multi-source flaw detection on industrial products using internal and external data, completing internal quality inspection without damaging the workpiece, replacing human eyes for external quality inspection, and improving inspection accuracy and efficiency.

[0076] In some embodiments, please refer to Figure 8 and Figure 9 , Figure 8 This is a schematic diagram showing the location of the flaw detection equipment in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in the embodiments of this application. Figure 9 This is a schematic diagram of the electrical relationship of the flaw detection equipment in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in the embodiments of this application. The method for three-dimensional flaw detection of industrial products based on multi-source data provided in the embodiments of this application is applied to the flaw detection equipment, which includes a handheld control and detection terminal, a robotic arm and its controller, a rotating turntable, a supplementary light, a server, and an ultrasonic flaw detection probe set on the robotic arm for acquiring point cloud data inside the target and a camera for acquiring image data outside the target.

[0077] Specifically, please refer to Figure 10 , Figure 10This diagram illustrates the appearance of a handheld control and inspection terminal for a flaw detection device used in a method for 3D flaw detection of industrial products based on multi-source data, as provided in this application embodiment. The handheld control and inspection terminal includes a display screen, a touchscreen, buttons, an interface, a speaker, a GPU processor, Bluetooth, and WiFi. The display screen shows the user the information they need to observe and the related processing flow. The touchscreen allows the user to interact with the handheld control and inspection terminal and process related flow. The buttons facilitate user interaction with the handheld control and inspection terminal, including power on / off, enter / exit, move left / right, and, when observing images, zoom in / out, zoom out, etc. The interface connects to ultrasonic flaw detection probes, cameras, robotic arms, and rotating disks to control their operation and collect relevant data. The speaker emits prompts to guide the user's actions. The GPU processor performs algorithmic processing on the images, providing computing resources; in this embodiment, a Nano Jetson minimum GPU core motherboard is used. Bluetooth connects to other Bluetooth devices, including transmitting information with other handheld control and inspection terminals. WiFi is used to enable handheld control and detection terminals to upload relevant information about each detected object to a computer server via the network for backup and analysis.

[0078] Specifically, the ultrasonic flaw detector probe is mounted on a robotic arm. As the robotic arm moves, it contacts the surface of the object to be inspected under different known postures, performs flaw detection on its interior, and obtains three-dimensional data with depth information. This data is then transmitted to a handheld control and inspection terminal via an interface for further processing. In this embodiment, the ultrasonic flaw detector probe is an XUT610C digital ultrasonic flaw detector.

[0079] Specifically, the camera is mounted on a robotic arm. As the robotic arm moves, it non-contactly inspects the surface of the object under different known postures, performing surface quality checks and obtaining surface data without depth information. This data is then transmitted via an interface to a handheld control and inspection terminal for further processing. In the embodiments of this application, the camera used is a Hikvision 8K color camera with a circular fill light.

[0080] Specifically, the robotic arm and its controller are standard robotic hands, and their actuators are in their original working mode before the performance enhancements of this application; for example, in a handling robot, this would be its gripper. This application adds an ultrasonic flaw detector and a camera to the robotic arm. The positions of the ultrasonic flaw detector and the camera are relatively fixed, thus enabling them to perform flaw detection work. The robotic arm's controller is connected to a handheld control and detection terminal, receiving motion control parameters from the handheld control and detection terminal. Without a rotating turntable (optional), if only one robotic arm and its controller are installed, the robotic arm structure needs at least four axes to achieve redundant control, and the industrial inspection object must be within its three-dimensional motion space. If this is not met, then at least two or more are needed to enable the ultrasonic flaw detector and camera to perform three-dimensional inspection of the industrial inspection object. In the embodiments of this application, one ABB RB120 robotic arm is used.

[0081] Specifically, the rotating turntable is used to place industrial inspection objects. It is used under the following conditions: if only one robotic arm and its controller are installed, the robotic arm structure has fewer than four axes, or the industrial inspection object is not within the robotic arm's three-dimensional motion space. In this case, the robotic arm's motion is minimal, and the rotating turntable drives the industrial inspection object to rotate, thereby enabling the ultrasonic flaw detector probe and camera to perform three-dimensional inspection of the industrial inspection object. The rotating turntable receives motion control parameters from a handheld control inspection terminal. In the embodiments of this application, the inspection object is the crankshaft connecting rod of a small passenger car, and the rotating turntable is custom-made, allowing only unidirectional counter-clockwise rotation with a controllable rotation angle.

[0082] Specifically, the supplementary light is used to solve the problem of inconsistent lighting in photos taken by the camera from different angles due to uneven external light intensity. It is mounted on the robotic arm and faces the object being inspected. In the embodiments of this application, a ring-shaped diffused supplementary light is used, with its center point coinciding with the center point of the camera and illuminating in the direction of the camera. The specific model is the Nanguan Nanguang R480C 48-watt ring supplementary light.

[0083] Specifically, the server stores data for each sample, the overall dataset, and backups of each model. It can communicate with a handheld control and detection terminal via WiFi. In the embodiments of this application, a Huawei industrial server is used, which communicates with Huawei Cloud.

[0084] In some embodiments, please refer to Figure 12 , Figure 12This application provides a flowchart illustrating the steps of acquiring 3D data using a flaw detection device in a method for 3D flaw detection of industrial products based on multi-source data. When collecting internal point cloud data and external image data of the target using the flaw detection device, the application first constructs a dataset by collecting preliminary data on the same standard product (in this application, a crankshaft connecting rod of a small family car). Ten qualified samples and ten unqualified samples, already manually judged, are collected as training samples, and specific unqualified areas are identified. The unqualified samples are required to include typical unqualified categories. In this embodiment, the typical unqualified categories include five types: internal cracks, internal bubbles, external notches, surface scratches, and surface irregularities, with two samples for each category. Then, 3D reconstruction is performed through steps a to f.

[0085] Step a: Reset the robotic arm and turntable, initializing Rnow=0 and Rde=12. Here, Rnow represents the current angle of the turntable; Rde is the fixed angle of each rotation of the turntable. Using the ground base of the robotic arm as the origin; the Z-axis is the direction perpendicular to the ground base upwards; the X-axis is the direction from the ground base towards the center of the turntable; the Y-axis is deduced using the right-hand rule of the robot coordinate system; establish a world coordinate system in space. ,in, The coordinate system is denoted by , and w represents the world coordinate system. The user first fixes the sample in a uniform pose on the rotating turntable, and the turntable remains stationary.

[0086] Step b: Data acquisition within the robotic arm's movement space. The user initiates the data acquisition process. The software, according to a pre-programmed sequence, sends signals to the robotic arm controller via a handheld control terminal, thereby driving the robotic arm to move according to the program. Within its range of motion, the robotic arm acquires data from multiple angles of the target. The acquired data includes the rotation angle of the turntable, the pose of the robotic arm actuators, surface images captured by the camera, and several internal images with depth information obtained by the ultrasonic flaw detector. After acquisition, the robotic arm returns to its initial pose.

[0087] Step c: Control the rotating turntable to move by a fixed angle Rde, causing the sample fixed on the rotating turntable to move, and then come to rest, Rnow = Rnow + Rde. In the embodiments of this application, the unidirectional counterclockwise rotation Rde is 12 degrees.

[0088] Step d: Determine if Rnow is greater than 360 degrees. If it is, the data acquisition process ends and proceed to step e. If not, repeat steps b to c to repeatedly collect multi-angle data at different rotating turntable angles until Rnow is greater than or equal to 360 degrees.

[0089] Step e: Based on the known rotation angle of the turntable and the pose of the robotic arm actuator, combined with the pose of the ultrasonic flaw detector and camera relative to the robotic arm actuator when the user installed them, and based on fundamental robotics knowledge (refer to the book "Fundamentals of Robotics Technology," edited by Xiong Youlun, Huazhong University of Science and Technology Press), it is easy to know the pose of the camera corresponding to the object being inspected when taking various images each time (surface images taken by the camera, and internal images with depth information obtained by the ultrasonic flaw detector). The ultrasonic flaw detector probe corresponds to the pose of the object being inspected. Where c represents the camera, and cpi represents the cpi-th visible light image captured by the camera; s represents the sonar, and spi represents the spi-th sonar image captured by the sonar. Since both the camera and the sonar are mounted on the robotic arm, cpi and spi have the same number under the same robotic arm pose, but their meanings are different.

[0090] Step f: Based on multi-source data objDatai={obji, , ,cpi,spi}, where obji is the number of the detected object obj. Using existing 3D reconstruction algorithms, the detected object obji is reconstructed in 3D to obtain a 3D reconstruction result obji3D={obji,pcloudi,skini}, where pcloudi represents the pcloudi-th 3D point cloud data, which expresses the intrinsic information of the detected object and is a continuous quantity; skini is the 3D skin of the object, which expresses the surface information of the detected object.

[0091] For details on the reconstruction methods of 3D point clouds, please refer to the paper titled "Multi-Sonar Fusion-Based Precision Underwater 3D Reconstruction for Optimal Scan Path Planning of AUV," published in IEEE ACCESS, Vol. 13, pp. 35157-35173. For details on the reconstruction methods of 3D skin, please refer to the paper titled "FD-TR: Feature detector based on scale invariant feature transform and bidirectional feature regionalization for digital image watermarking," published in Multimed. Tools Appl., Vol. 80, pp. 32197–32217.

[0092] Understandably, through the three-dimensional reconstruction method in steps a to f, three-dimensional reconstruction is performed on 10 qualified samples and 10 unqualified samples that have been manually judged, to obtain a sample set.

[0093] The sample set of qualified products is GoodDatas={objDatai}, where objDatai=1~10. The sample set of unqualified products is BadDatas={objDatai}, where objDatai=1~10.

[0094] Furthermore, this application uses the good data set (GoodDatas) and the bad data set (BadDatas) as model inputs to train the 3D data anomaly detection algorithm designed in this paper, obtaining a trained model MD, which includes two schemes: MDA and MDB. Please refer to [link / reference]. Figure 13 , Figure 13 This application provides a method for 3D flaw detection of industrial products based on multi-source data, illustrating the relationship between different model selection schemes according to data volume. Scheme MDA is the initial scheme, suitable for conditions with very small data volumes, but the results are relatively coarse. Scheme MDB is the upgraded scheme, suitable for later stages when the data volume reaches a certain level, such as more than 500 qualified and 500 unqualified samples each. Each scheme has two data sources, resulting in two analysis channels: an internal flaw detection model (MDA-1, MDB-1) and an external flaw detection model (MDA-2, MDB-2).

[0095] Understandably, when the initial amount of data is insufficient, an MDA model can be used for detection. First, the 3D data of the workpiece to be detected is reconstructed through steps a to f to obtain Tobji3D. Then, the trained MDA model is used to perform anomaly detection on Tobji3D to obtain the result of whether it is abnormal, and the abnormal parts are marked for user observation. In the course of work, the user can modify the result of whether it is abnormal, manually mark it, and add the judgment result to the training sample consisting of the qualified product sample set GoodDatas and the unqualified product sample set BadDatas to retrain the model and obtain a new MDA model. When the number of such manually marked samples accumulates to a certain extent in the later stage, for example, more than 500 qualified product samples and more than 500 unqualified product samples, the present invention will select the new MDA model for analysis.

[0096] In some embodiments, please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a method for three-dimensional flaw detection of industrial products based on multi-source data, as provided in this application. Specifically, the method for three-dimensional flaw detection of industrial products based on multi-source data is implemented through steps 100 to 700:

[0097] Step 100: Determine the spatial resolution of the training samples and the ultrasonic flaw detection probe.

[0098] Step 200: Segment the world coordinate system based on spatial resolution to obtain a three-dimensional space containing multiple discrete units.

[0099] Specifically, based on the spatial resolution sr of the ultrasonic flaw detection probe, the world coordinate system is... The discrete 3D space is divided into 3D square grids of size sr*sr*sr, which is a three-dimensional space containing multiple discrete units.

[0100] Step 300: Configure the discretized point cloud data within the training samples in a three-dimensional space.

[0101] Step 400: Assign the discrete units with point cloud data to the first type of unit, and assign the discrete units without point cloud data to the second type of unit.

[0102] Specifically, each pcloudi in the sample set (representing the pcloudi-th 3D point cloud data) expresses the intrinsic information of the detected object. It is a continuous quantity in space, and its value is essentially either 1 or 0, where 1 indicates change and 0 indicates absence. pcloudi is then projected into a discrete 3D space. Discrete cells containing point cloud data are assigned a value of 1, obtaining the first type of cell; discrete cells without point cloud data are assigned a value of 0, obtaining the second type of cell. For each 3D square grid of size sr*sr*sr, if it contains even one point of pcloudi data, the grid is defined as containing information, i.e., a value of 1, belonging to the first type of cell. This information could be the boundary of the object or internal damage information; if there is no point, it is a value of 0, belonging to the second type of cell. Thus, each pcloudi corresponds to a discrete point cloud dataset dispcloudi with 3D grid properties. From a three-dimensional perspective, the vast majority of areas inside this discrete point cloud dataset gpcloudi are 0, indicating no information; the cells (discrete points) that contain information have a value of 1, which exists at the boundaries and may contain defects inside.

[0103] Understandably, due to the presence of various noises, the discrete point cloud dataset `dispcloudi` may contain salt-and-pepper noise and Gaussian noise, although the noise level is generally not too high. Salt-and-pepper noise may also break the original continuous structure of 1 values. Therefore, this application designs a 3D filtering and enhancement scheme to filter and enhance `dispcloudi`. The idea is to use an operation similar to opening in two-dimensional morphology, which can be used to eliminate small objects, separate objects at fine points, and smooth the boundaries of larger objects without significantly changing their area. Therefore, after assigning values ​​to the discrete units, this application performs filtering and enhancement on the discrete units. Please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a flowchart illustrating the steps of filtering and enhancing discrete units in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in this application embodiment. Specifically, the method for filtering and enhancing discrete units in this application is implemented through steps 410 to 430:

[0104] Step 410: Enclose multiple discrete units outward from the first type of unit to obtain the filter-enhanced convolution kernel.

[0105] Step 420: In the filter-enhanced convolution kernel, in response to the existence of two first-class units with an L2 norm distance less than or equal to 2, the discrete unit on the line connecting the two first-class units is assigned as a first-class unit.

[0106] Step 430: In the filter-enhanced convolution kernel, in response to the existence of only one first-class unit, the first-class unit is assigned the value of the second-class unit.

[0107] Specifically, centered on a point with a value of 1 in dispcloudi, three discrete points are extended in each of the X, Y, and Z directions in the world coordinate system, forming a 7*7*7 3D convolution kernel. Within this kernel, if two points have a value of 1 and their L2 norm distance is less than or equal to 2, then the point at the 3D connection between them is set to 1. This enhances the connectivity of broken lines. For example, if a point with XYZ coordinates 3, 3, 3 has a value of 1, and if there are points with the same value of 1 on the diagonal (3, 3, 1), then their L2 norm distance is equal to 2, and the point at the 3D connection between them (2, 2, 2) will be changed from 0 to 1. If only the center point of this 7*7*7 3D convolution kernel has a value of 1, then that center point is reset to 0. This provides the ability to filter individual noise points in space.

[0108] Step 500: Obtain the internal flaw detection model based on the assigned discrete units.

[0109] In some embodiments, when obtaining the first internal flaw detection model MDA-1, please refer to Figure 2 and Figure 14 , Figure 2 The flowchart illustrates the steps of obtaining the first internal flaw detection model based on assigned discrete units in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in this application embodiment. Figure 14 The flowchart of the program for obtaining the first internal flaw detection model based on the assigned discrete units in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in the embodiments of this application is as follows: The method for obtaining the first internal flaw detection model based on the assigned discrete units in this application is specifically implemented through steps 510 to 530:

[0110] Step 510: In multiple qualified samples, obtain the average value of discrete cells at the same location as the standard value.

[0111] Specifically, the average value of the same point across all samples in `dispcloudi` is rounded to obtain the standard value `Sdispcloud` for that point. Assuming there are 3 samples, `dispcloud` might contain 500*500*500 cells (discrete points). For a given point where sample 1 is 1, sample 2 is 0, and sample 3 is 0, then 1 + 0 + 0 = 1, which is rounded to 0. This point in `Sdispcloud` is then set to 0.

[0112] Step 520: Among multiple qualified samples, compare the values ​​of the same discrete units in any two qualified samples, and determine the average value of the number of discrete units that differ, which is the standard range value.

[0113] Specifically, the `dispcloudi` in the qualified product sample set `GoodDatas` is analyzed and compared to obtain the number of deviations between different samples at the same position. The comparison involves iterating through each point in `dispcloudi`. If the value of this point between consecutive samples (e.g., the i-th qualified product sample and the j-th qualified product sample) is 1 and 0 respectively, then a deviation exists. The total number of deviations between the i-th and j-th qualified product samples, `GoodDev(i,j)`, is then calculated. Finally, the average value `AvgGoodDev` is obtained for the total number of deviations between all i and j samples in the qualified sample set. Assuming there are 3 samples, comparing sample 1 with sample 2 yields `GoodDev(1,2)`; then comparing sample 1 with sample 3 yields `GoodDev(1,3)`; then comparing sample 2 with sample 3 yields `GoodDev(2,3)`. The average of `GoodDev(1,2)`, `GoodDev(1,3)`, and `GoodDev(2,3)` is then taken to obtain the standard range value `AvgGoodDev`.

[0114] Further, the dispcloudi in the nonconforming sample set BadDatas is analyzed and compared with Sdispcloud using the same method as in steps 510 to 520 to obtain the nonconforming range value AvgBadDev.

[0115] Step 530: Obtain the internal flaw detection model based on the standard value and standard range value.

[0116] In some embodiments, please refer to the documentation when obtaining the second internal flaw detection model MDB-1. Figure 5 , Figure 5 This application provides a flowchart illustrating the steps of obtaining a second internal flaw detection model based on assigned discrete units in a method for three-dimensional flaw detection of industrial products based on multi-source data. Specifically, the method for obtaining the second internal flaw detection model based on assigned discrete units is implemented through steps 540 to 570.

[0117] Step 540: Using the first type of unit as the center, encompass multiple discrete units outward to obtain the flaw detection convolution kernel.

[0118] Specifically, please refer to Figure 11 , Figure 11 This diagram illustrates the acquisition of the flaw detection convolution kernel in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in this application embodiment. A 9*9*9 three-dimensional flaw detection convolution kernel is constructed, centered on the point with a value of 1 in dispcloudi, extending four discrete points in each of the X, Y, and Z directions in the world coordinate system, forming a 9*9*9 three-dimensional convolution kernel. The reason for convolving only the point with a value of 1 is that, since the interior of most workpieces to be inspected is sparse with 1s, this reduces the amount of computation.

[0119] Step 550: Divide the flaw detection convolution kernel into layers along a preset direction to obtain multiple layered convolutional surfaces.

[0120] Specifically, the points convolved by the flaw detection convolution kernels are layered along the Z-axis to obtain nine 9*9 two-dimensional layered convolutional surfaces from small to large Z-values.

[0121] Step 560: Following the preset layering order, stitch together multiple layered convolutional surfaces sequentially to obtain the stitched convolutional surface.

[0122] Specifically, following the Z-axis from smallest to largest, layered convolutional surfaces are stitched together sequentially from left to right and top to bottom to form a 27*27 stitched convolutional surface image. Clearly, this image is divided into 3*3 large regions, with a 9*9 small region placed within each large region. The large regions are numbered as follows: the first row from left to right corresponds to Z-axis values ​​1-3; the second row from left to right corresponds to Z-axis values ​​4-6; and the third row from left to right corresponds to Z-axis values ​​7-9. Thus, for each point with a value of 1, a 27*27 image is obtained in space. The discrete points of 1 in this image may form a scar that needs to be identified.

[0123] Step 570: Determine the internal flaw detection model to be trained, and train the internal flaw detection model based on the spliced ​​convolutional surface of the training samples to obtain the trained internal flaw detection model.

[0124] In some embodiments, the second internal flaw detection model includes an MLP neural network, which comprises an input layer, an intermediate layer, and an output layer. The number of neurons in the input layer is the same as the number of discrete units in the concatenated convolutional surface. The number of neurons in the intermediate layer is twice the number of discrete units in the concatenated convolutional surface. The output layer comprises three neurons, used to represent qualified, unqualified, and pending, respectively. Specifically, the input layer has 27*27 neurons, and the input value of each neuron is only 1 or 0, which comes from the 27*27 concatenated convolutional surface image. The intermediate layer has 27*27*2 neurons. Its output consists of three neurons. When the first output neuron is 1, it indicates that the region is qualified. When the second output neuron is 1, it indicates that the region is unqualified. When the third output neuron is 1, it indicates that the region is in the gray area. Only one of the three neurons can output 1 when labeling samples.

[0125] Understandably, when using the MDB-1 scheme, the amount of data has reached a certain level, for example, more than 500 qualified samples and more than 500 unqualified samples. Since each unqualified sample contains several internal points, the amount of data is sufficient to support this training scheme. The corresponding training sample set can be easily obtained through personnel labeling.

[0126] Step 600: Obtain the external flaw detection model using the external image data of the training samples.

[0127] In some embodiments, when obtaining the first external flaw detection model MDA-2, please refer to Figure 4 , Figure 4 This application provides a flowchart of the steps for obtaining a first external flaw detection model in a method for three-dimensional flaw detection of industrial products based on multi-source data. Specifically, the method for obtaining the first external flaw detection model in this application is implemented through steps 610 to 640:

[0128] Step 610: In multiple qualified sample RGB format images, obtain the average value of the R channel, the average value of the G channel, and the average value of the B channel for the same pixel, and construct the standard channel values.

[0129] Specifically, in a qualified sample RGB format image Sskin, each pixel Sskini has 3 RGB channel values. Assuming there are 3 samples, for a certain point Sskini in Sskin, sample 1 is (100, 150, 200), that is, the R channel is 100, the G channel is 150, and the B channel is 200; sample 2 is (110, 160, 210), and sample 3 is (120, 140, 250). Then the standard channel value of this point is (110, 150, 220).

[0130] Step 620: In multiple qualified RGB format images, obtain the total difference of the same pixel in the R, G and B channels of any two qualified samples, as well as the variance of the multiple total differences of the same pixel.

[0131] Specifically, assuming sample i is (100, 150, 200) and sample j is (110, 160, 210), then the total difference is |100-110|+|150-160|+|200-210|=30.

[0132] Step 630: Obtain the average value of the variance of multiple pixels and construct the standard channel range value.

[0133] Specifically, a manually adjustable threshold Td1 is set, with its initial value being the average variance of the differences among all points in the qualified sample. For example, assuming there are 3 samples, for a certain point Sskini in Sskin, there is a difference of 1 between samples 1 and 2, a difference of 2 between samples 1 and 3, and a difference of 3 between samples 2 and 3. Then, for the position of this point, the variance of its difference, DevSskini, is calculated; then, the average of this DevSskini for all points in Sskini is taken to obtain AvgGoodDevSskin. This value can be adjusted by the user as needed.

[0134] Furthermore, the Sskin of the non-conforming sample set is analyzed using the same method as in steps 610 to 630 to obtain the non-standard channel range value AvgBadDevSskin.

[0135] Step 640: Obtain the external flaw detection model based on the standard channel range value.

[0136] In some embodiments, this application also provides a second external flaw detection model, MDA-2, which uses image data to represent the surface information of the object being detected. The values ​​are 0-255 values ​​of the three RGB channels of color. Since image data is not sparse, and people can simulate many surface anomalies by externally damaging a few samples, the amount of data for this task can be very large, fully supporting the number of network parameters for deep learning.

[0137] The approach adopted in this application (paper titled FCCDN: Feature constraint network for VHR image change detection, published in ISPRS Journal of Photogrammetry and Remote Sensing, 2022, Vol. 187, pp. 101-119) yields the MDB-2 model for external surface anomaly detection, which is then trained using simulated surface anomaly data. The MDB-2 model constrains features in both dual-temporal feature extraction and feature fusion. At the center of the backbone network, a Non-Local Feature Pyramid Network (NL-FPN) is designed to extract and fuse multi-scale features, and a Densely Connected Feature Fusion Module (DFM) is constructed to robustly fuse dual-temporal features. This construction integrates multiple features in each stream and makes better decisions, thereby increasing the model's robustness and enabling it to cope with instability caused by various noises.

[0138] Step 700: Determine the workpiece to be inspected, input the internal point cloud data of the workpiece to be inspected into the internal flaw detection model, and obtain the internal flaw detection results; input the external image data of the workpiece to be inspected into the external flaw detection model, and obtain the external flaw detection results.

[0139] Specifically, when using the first internal flaw detection model, MDA-1, please refer to [link / reference needed]. Figure 3 , Figure 3 The flowchart illustrates the steps of obtaining internal flaw detection results using a first internal flaw detection model in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in this application embodiment. Specifically, the method for obtaining internal flaw detection results in this application is implemented through steps 710 to 730:

[0140] Step 710: Configure the discretized point cloud data of the workpiece to be inspected in three-dimensional space.

[0141] Step 720: In the workpiece to be inspected, the discrete units with point cloud data are assigned the first type of unit, and the discrete units without point cloud data are assigned the second type of unit.

[0142] Step 730: In response to the fact that the number of discrete units in the workpiece to be inspected that differ from the standard value is less than or equal to the standard range value, the workpiece to be inspected is determined to be a qualified product.

[0143] Specifically, please refer to Figure 15 , Figure 15 The flowchart of the procedure for internal flaw detection using the first internal flaw detection model MDA-1 in the method for three-dimensional flaw detection of industrial products based on multi-source data provided in this application embodiment is as follows: The workpiece to be inspected, dispcloud(j), is compared with each point in this Sdispcloud, and the total number of different points, TDev(j), is calculated. If TDev(j) <= AvgGoodDev, the workpiece to be inspected is an acceptable qualified product. If it exceeds this value, the workpiece to be inspected is likely a defective product. Further, if TDev(j) >= AvgBadDev, it is determined to be a defective product; otherwise, it is a pending sample in an unknown gray area. The user uses human experts to analyze and label the sample, and then resubmits it to the training sample set.

[0144] Specifically, when using the second type of internal flaw detection model MDB-1, multiple discrete units are included outward from the first type of unit in the workpiece to be inspected to obtain a 9*9*9 flaw detection convolution kernel; then, the flaw detection convolution kernel is layered along the preset Z-axis direction to obtain multiple 9*9 layered convolution surfaces; then, along the preset Z-axis direction, according to the layering order from smallest to largest, the multiple layered convolution surfaces are spliced ​​from left to right and from top to smallest to obtain a 27*27*27 spliced ​​convolution surface, which is then input into the trained second type of internal flaw detection model MDA-2 to obtain the flaw detection result.

[0145] Specifically, when using the first external flaw detection model, MDA-2, for the same pixel, the R, G, and B channel values ​​of the pixel in the RGB format image of the workpiece to be inspected are subtracted from the R, G, and B channel values ​​of the standard channel values, respectively. The absolute values ​​are then summed to obtain the difference of the current pixel. The differences of all pixels in the RGB format image of the workpiece to be inspected are obtained, and the average difference is calculated. If the average difference is less than or equal to the standard channel range, the workpiece is judged as a qualified product. If the average difference exceeds the standard channel range, the workpiece is likely to be a defective product. Further, if the average difference is greater than or equal to the non-standard channel range, the workpiece is judged as a defective product; otherwise, it is a pending sample in an unknown gray area. The user uses human experts to analyze and label the sample, and then resubmits it to the training sample set.

[0146] Specifically, when using the second external flaw detection model MDB-2, the RGB format image data of the workpiece to be inspected is input into the external flaw detection model MDB-2. The external flaw detection model MDB-2 compares the RGB format image data of the workpiece to be inspected with the RGB format image data of qualified samples to perform change detection and obtain the detection results.

[0147] Understandably, the method for three-dimensional flaw detection of industrial products based on multi-source data provided in the embodiments of this application includes determining the spatial resolution of the training sample and the ultrasonic flaw detection probe; segmenting the world coordinate system based on the spatial resolution to obtain a three-dimensional space containing multiple discrete units; configuring the discretized point cloud data inside the training sample in the three-dimensional space; assigning the discrete units with point cloud data as first-type units and the discrete units without point cloud data as second-type units; obtaining an internal flaw detection model based on the assigned discrete units; obtaining an external flaw detection model through the external image data of the training sample; determining the workpiece to be inspected and inputting the internal point cloud data of the workpiece to be inspected into the internal flaw detection model to obtain the internal flaw detection result; inputting the external image data of the workpiece to be inspected into the external flaw detection model to obtain the external flaw detection result. This application provides ultrasonic flaw detection and visual appearance inspection, performing simultaneous multi-source flaw detection on industrial products through internal and external data, completing internal quality inspection without damaging the workpiece, replacing the human eye for external quality inspection, and improving inspection accuracy and efficiency.

[0148] In some embodiments, this application also provides a system for three-dimensional flaw detection of industrial products based on multi-source data. Please refer to [link to relevant documentation]. Figure 7 , Figure 7 This is a module connection diagram of a system for three-dimensional flaw detection of industrial products based on multi-source data, provided in an embodiment of this application. The system is applied to a flaw detection device, which includes a robotic arm, an ultrasonic flaw detection probe mounted on the robotic arm for acquiring internal point cloud data of the target, and a camera for acquiring external image data of the target.

[0149] Training sample configuration module 10 is used to determine the spatial resolution of the training sample and the ultrasonic flaw detection probe;

[0150] The spatial construction module 20 is used to segment the world coordinate system based on spatial resolution to obtain a three-dimensional space containing multiple discrete units;

[0151] Point cloud configuration module 30 is used to configure the discretized point cloud data inside the training sample in a three-dimensional space;

[0152] The assignment module 40 assigns the discrete units with point cloud data to the first type of unit and the discrete units without point cloud data to the second type of unit.

[0153] External flaw detection model acquisition module 50 is used to acquire internal flaw detection model based on the assigned discrete unit;

[0154] The internal flaw detection model acquisition module 60 acquires the external flaw detection model through the external image data of the training samples;

[0155] The flaw detection result module 70 is used to determine the workpiece to be inspected, input the internal point cloud data of the workpiece to be inspected into the internal flaw detection model to obtain the internal flaw detection results, and input the external image data of the workpiece to be inspected into the external flaw detection model to obtain the external flaw detection results.

[0156] This application has provided a detailed description of a method and system for three-dimensional flaw detection of industrial products based on multi-source data, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for three-dimensional flaw detection of industrial products based on multi-source data, characterized in that, An application to a flaw detection device, the flaw detection device including a robotic arm, an ultrasonic flaw detection probe disposed on the robotic arm for acquiring point cloud data of the target's interior, and a camera for acquiring image data of the target's exterior, the method comprising: Determine the spatial resolution of the training samples and the ultrasonic flaw detection probe; The world coordinate system is segmented based on the spatial resolution to obtain a three-dimensional space containing multiple discrete units; The discretized point cloud data within the training samples is configured in the three-dimensional space; Discrete units containing the point cloud data are assigned to the first type of unit, and discrete units without the point cloud data are assigned to the second type of unit. The internal flaw detection model is obtained based on the assigned discrete units; An external flaw detection model is obtained using the external image data of the training samples; The workpiece to be inspected is identified, and the internal point cloud data of the workpiece to be inspected is input into the internal flaw detection model to obtain the internal flaw detection results; the external image data of the workpiece to be inspected is input into the external flaw detection model to obtain the external flaw detection results.

2. The method for three-dimensional flaw detection of industrial products based on multi-source data according to claim 1, characterized in that, The training samples include qualified samples, and the step of obtaining the internal flaw detection model based on the assigned discrete units includes: In a plurality of qualified samples, the average value of the discrete units at the same position is obtained as the standard value; Among the multiple qualified samples, the values ​​of the same discrete units in any two qualified samples are compared, and the average value of the number of discrete units with differences is determined as the standard range value. The internal flaw detection model is obtained based on the standard value and the standard range value.

3. The method for three-dimensional flaw detection of industrial products based on multi-source data according to claim 2, characterized in that, The step of inputting the internal point cloud data of the workpiece to be inspected into the internal flaw detection model to obtain the internal flaw detection results includes: The discretized point cloud data of the workpiece to be inspected is configured in the three-dimensional space; In the workpiece to be inspected, discrete units containing the point cloud data are assigned the first type of unit, and discrete units without the point cloud data are assigned the second type of unit. If the number of discrete units in the workpiece to be inspected that differ from the standard value is less than or equal to the standard range value, the workpiece to be inspected is determined to be a qualified product.

4. The method for three-dimensional flaw detection of industrial products based on multi-source data according to claim 1, characterized in that, The training samples include qualified samples, and the step of obtaining the external flaw detection model includes: In multiple qualified sample RGB format images, the average value of the R channel, the average value of the G channel, and the average value of the B channel for the same pixel are obtained to construct standard channel values; In multiple qualified sample RGB format images, obtain the total difference of the same pixel in the R channel, G channel and B channel in any two qualified samples, and the variance of multiple total differences of the same pixel; The average value of the variance of multiple pixels is obtained to construct a standard channel range value; The external flaw detection model is obtained based on the standard channel range value.

5. The method for three-dimensional flaw detection of industrial products based on multi-source data according to claim 1, characterized in that, The steps for obtaining the internal flaw detection model based on the assigned discrete unit include: The first type of unit is used as the center to encompass multiple discrete units outward to obtain the flaw detection convolution kernel; The flaw detection convolution kernel is layered along a preset direction to obtain multiple layered convolutional surfaces; Following the layering order along the preset direction, multiple layered convolutional surfaces are sequentially spliced ​​together to obtain spliced ​​convolutional surfaces; The internal flaw detection model to be trained is determined, and the internal flaw detection model is trained based on the spliced ​​convolutional surface of the training samples to obtain the trained internal flaw detection model.

6. The method for three-dimensional flaw detection of industrial products based on multi-source data according to claim 5, characterized in that, The internal flaw detection model includes an MLP neural network, which comprises: The input layer has the same number of neurons as the number of discrete units in the concatenated convolutional surface. The intermediate layer has twice the number of neurons as the concatenated convolutional surface discrete units. The output layer comprises three neurons, which are used to represent qualified, unqualified, and pending, respectively.

7. The method for three-dimensional flaw detection of industrial products based on multi-source data according to claim 1, characterized in that, The external flaw detection model constrains features in both dual-temporal feature extraction and feature fusion. The backbone network of the external flaw detection model is equipped with a nonlocal feature pyramid network (NL-FPN) to extract and fuse multi-scale features, so as to construct a densely connected feature fusion module (DFM) to robustly fuse dual-temporal features. External flaw detection results are obtained by comparing the external image data of qualified samples and workpieces under the same pose using an external flaw detection model.

8. The method for three-dimensional flaw detection of industrial products based on multi-source data according to claim 1, Its distinctive feature is that, after assigning values ​​to the discrete units, the method further includes: Multiple discrete units are enclosed outward from the first type of unit to obtain a filter-enhanced convolution kernel; In the filter-enhanced convolution kernel, in response to the existence of two first-class units with an L2 norm distance less than or equal to 2, the discrete unit on the line connecting the two first-class units is assigned as a first-class unit; In the filtered and enhanced convolutional kernel, in response to the existence of only one first-type unit, the first-type unit is assigned the value of the second-type unit.

9. A system for three-dimensional flaw detection of industrial products based on multi-source data, characterized in that, An instrument for flaw detection, comprising a robotic arm, an ultrasonic flaw detection probe mounted on the robotic arm for acquiring point cloud data of the target's interior, and a camera for acquiring image data of the target's exterior, the system comprising: Training sample configuration module (10) is used to determine the spatial resolution of the training sample and the ultrasonic flaw detection probe; The space construction module (20) is used to segment the world coordinate system based on the spatial resolution to obtain a three-dimensional space containing multiple discrete units; Point cloud configuration module (30) is used to configure the discretized point cloud data inside the training sample in the three-dimensional space; The assignment module (40) assigns the discrete units that have the point cloud data to the first type of unit and assigns the discrete units that do not have the point cloud data to the second type of unit; The external flaw detection model acquisition module (50) is used to acquire the internal flaw detection model based on the assigned discrete unit; The internal flaw detection model acquisition module (60) acquires the external flaw detection model through the external image data of the training samples; The flaw detection result module (70) is used to determine the workpiece to be inspected, input the internal point cloud data of the workpiece to be inspected into the internal flaw detection model, and obtain the internal flaw detection result; input the external image data of the workpiece to be inspected into the external flaw detection model, and obtain the external flaw detection result.

Images