Weld defect detection systems, methods, apparatuses, electronic devices, media, and products

Abstract

The present disclosure relates to a welding defect detection system, method, device, electronic equipment, medium and product, and relates to the technical field of intelligent manufacturing. The method comprises the following steps: in response to the to-be-inspected product reaching a target detection area, controlling a plurality of image acquisition devices to perform an image acquisition operation to obtain a plurality of to-be-inspected images of the to-be-inspected product at different angles; constructing a to-be-inspected three-dimensional model of the to-be-inspected product according to the plurality of to-be-inspected images at different angles and camera parameters; and performing defect detection judgment according to the to-be-inspected three-dimensional model and a standard three-dimensional model to obtain a defect detection result. The present disclosure can improve the detection accuracy of welding defects.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent manufacturing technology, and in particular to a welding defect detection system, method, apparatus, electronic device, medium, and product. Background Technology

[0002] As the manufacturing industry places increasingly higher demands on product quality, welding defects such as weld beads, pinholes, incomplete solder filling, and overheating may occur during the welding process of some welded products, such as U-shaped tubes. To reduce the defect rate, relevant technologies use manual visual inspection or a combination of manual inspection and an industrial camera to detect welding defects in the products to be inspected. However, this still results in low accuracy due to missed or false detections. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides a welding defect detection system, method, apparatus, electronic equipment, medium, and product to achieve the technical effect of improving the detection accuracy of welding defects.

[0004] According to a first aspect of the present disclosure, a welding defect detection system is provided, comprising: a plurality of image acquisition devices and a controller;

[0005] The plurality of image acquisition devices are all electrically connected to the controller;

[0006] The plurality of image acquisition devices include a group of side-viewing devices deployed diagonally in front of and diagonally behind the product to be inspected, and a group of top-viewing devices deployed above the product to be inspected.

[0007] The image acquisition device is used to capture images of the product under inspection, thereby obtaining images of the product under inspection from different perspectives.

[0008] The controller is used to detect welding defects based on the image to be inspected, and obtain the defect detection result.

[0009] According to a second aspect of the present disclosure, a welding defect detection method is provided, applied to a welding defect detection system as described in the first aspect, the method comprising:

[0010] In response to the product under inspection reaching the target detection area, the multiple image acquisition devices are controlled to perform image acquisition operations to obtain multiple images of the product under inspection from different perspectives;

[0011] Based on the multiple images to be inspected from different perspectives and the camera parameters, a three-dimensional model of the product to be inspected is constructed.

[0012] Defect detection and judgment are performed based on the 3D model to be inspected and the standard 3D model to obtain the defect detection results.

[0013] According to a third aspect of the present disclosure, a welding defect detection apparatus is provided, applied to a welding defect detection system as described in the first aspect, the apparatus comprising:

[0014] The system calibration module is used to control multiple image acquisition devices to perform calibration operations and determine camera parameters;

[0015] The image acquisition module is used to control multiple light sources and multiple image acquisition devices to perform image acquisition operations in response to the product under inspection reaching the target detection area, so as to obtain multiple images of the product under inspection from different perspectives.

[0016] A multi-view 3D reconstruction module is used to construct a 3D model of the product to be inspected based on the image to be inspected and the camera parameters.

[0017] The defect detection module performs defect detection operations based on the three-dimensional model to be inspected and the standard three-dimensional model to obtain the defect detection results.

[0018] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising:

[0019] processor;

[0020] Memory used to store processor-executable instructions;

[0021] The processor is configured to perform a welding defect detection method as described in the second aspect.

[0022] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which, when instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform a welding defect detection method as described in the second aspect.

[0023] According to a fourth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a welding defect detection method as described in the second aspect.

[0024] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: First, multiple images of the product to be inspected from different perspectives are acquired through multiple image acquisition devices to avoid missing welding defects on the weld, making the images of the product to be inspected more comprehensive for subsequent inspection. Then, based on the multiple images of the product to be inspected from different perspectives and camera parameters, a three-dimensional model of the product to be inspected is constructed. This three-dimensional model of the product to be inspected comprehensively restores the welding defects on the actual object. Finally, the three-dimensional model of the product to be inspected is compared with a standard three-dimensional model to obtain accurate defect detection results, thereby improving the accuracy of the detection results.

[0025] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0027] Figure 1 This is a schematic diagram illustrating an application scenario of a welding defect detection method according to some embodiments of this disclosure.

[0028] Figure 2 This is a schematic diagram of the overall structure of a welding defect detection system according to some embodiments of the present disclosure.

[0029] Figure 3 This is a top-view schematic diagram of the deployment of the equipment group according to some embodiments of the present disclosure.

[0030] Figure 4 This is a flowchart illustrating a welding defect detection method according to some embodiments of the present disclosure.

[0031] Figure 5 This is a block diagram of a welding defect detection device according to some embodiments of the present disclosure.

[0032] Figure 6 This is a block diagram of an electronic device according to some embodiments of the present disclosure.

[0033] Figure 7 This is a block diagram of a chip mentality provided according to some of the embodiments shown in this disclosure.

[0034] Reference numerals: 21-Image acquisition device; 22-Controller; 23-Light source. Detailed Implementation

[0035] Some embodiments of this disclosure will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0036] The embodiments described in the following examples of this disclosure are not representative of all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0037] Figure 1 This is a schematic diagram illustrating an application scenario of a welding defect detection method according to some embodiments of this disclosure. For example... Figure 1 As shown, the scenario includes a product assembly line, a welding defect detection section, a first product unloading point, and a second product unloading line.

[0038] like Figure 1 As shown, the product assembly line can be a conveyor belt or other structure used to transport products to be inspected. The products move along the assembly line to a welding defect detection section for inspection. This section can be a testing chamber where images of welding defects are captured manually or using installed industrial cameras, followed by inspection. The first and second product unloading lines can also be conveyor belts used to transport inspected and qualified products to the next production node.

[0039] Continue to refer to Figure 1 In related technologies, welding defect detection schemes combining manual self-inspection with one or two industrial cameras can only capture two-dimensional images of the weld joints of the product under inspection from one or two angles, such as the top or side. The presence of welding defects is then determined by directly analyzing features such as grayscale, contrast, and edge contours in the two-dimensional images. While this scheme is more efficient than manual self-inspection and may improve accuracy in certain scenarios, practical experience has shown that it is heavily influenced by lighting and viewing angles, especially on highly reflective metal surfaces. This results in poor imaging quality and difficulty in feature extraction, leading to inaccurate detection results. Furthermore, two-dimensional images cannot reproduce three-dimensional dimensions, such as for special products like U-shaped tubes with varying diameters and bends, further contributing to the poor accuracy of the detection results.

[0040] To address the aforementioned technical problems, some embodiments of this disclosure provide a welding defect detection method with the following technical concept: A larger number of industrial cameras are used to acquire images of the product under inspection from multiple different perspectives. Based on these images and the parameters of each camera, a three-dimensional model of the product is constructed. Finally, the constructed three-dimensional model is compared with a standard three-dimensional model to determine the defect detection result. This detection process not only avoids problems caused by single-angle lighting but also allows for the determination of the size and specific location of welding defects in the inspected product through the constructed three-dimensional model, thus improving the accuracy of welding defect detection.

[0041] Figure 2This is a schematic diagram of the overall structure of a welding defect detection system according to some embodiments of the present disclosure.

[0042] like Figure 2 As shown, the welding defect detection system includes: multiple image acquisition devices 21 and a controller 22;

[0043] Among them, multiple image acquisition devices 21 are electrically connected to the controller 22;

[0044] The multiple image acquisition devices 21 include a group of side-viewing devices deployed diagonally in front of and diagonally behind the product to be inspected, and a group of top-viewing devices deployed above the product to be inspected.

[0045] The image acquisition device 21 is used to take pictures of the product under inspection to obtain images of the product under inspection from different angles;

[0046] The controller 22 is used to detect welding defects based on the image to be inspected and obtain the defect detection results.

[0047] In this embodiment, the image acquisition device 21 can be an industrial camera or other low-cost shooting equipment; this embodiment does not limit its use. When the product to be inspected is... Figure 2 The U-shaped tube shown moves into the shooting range of the image acquisition device 21 under the action of the conveyor belt, and then is sequentially photographed by the image acquisition device 21 deployed at the corresponding positions to obtain images of the product under inspection from different angles. In this embodiment, the image acquisition device 21 can be installed near the conveyor belt of the production line at the corresponding position by means of screw connection. The installation method is not limited here.

[0048] To obtain multiple images of the product under inspection from different perspectives, several of the image acquisition devices 21 can be deployed at intervals in front of and behind the product under inspection to form a side-viewing device group, for example... Figure 2 The four image acquisition devices 21 located on both sides of the direction of movement of the product to be inspected form a side-viewing device group. Through practice, it has been found that the distance between these image acquisition devices 21 and the conveyor belt is set at 200±100 mm, and the angle between the shooting direction of the image acquisition devices 21 in the side-viewing device group and the direction of conveying the inspected product can be 45° and 135°. The number of image acquisition devices 21 on the left and the number of image acquisition devices 21 on the right are both two, and the angles are different.

[0049] Figure 3 This is a top-view schematic diagram of the deployment of the equipment group according to some embodiments of the present disclosure.

[0050] like Figure 2 and Figure 3As shown, several other images among the multiple image acquisition devices 21 can be deployed in a surrounding manner above the product to be inspected to form a top-view device group. For example... Figure 2 The three image acquisition devices 21 shown are positioned above the product to be inspected. These three image acquisition devices 21 can be arranged in a circular manner above the product to be inspected, for taking pictures of the product from above. The vertical distance between the image acquisition devices 21 in the top-view device group and the product to be inspected is 360±50 mm.

[0051] In this embodiment, the controller 22 can be a server, chip, all-in-one machine, or other device capable of image data acquisition, image data processing, and defect detection. This embodiment does not limit the specific device used. The controller 22 and each image acquisition device 21 can be connected via wired or wireless communication.

[0052] Continue to refer to Figure 2 In an optional embodiment of this disclosure, the welding defect detection system further includes: multiple light sources 23, with a light source deployed next to each image acquisition device, the multiple light sources being used to provide illumination to the surface of the product to be inspected from different angles. All multiple light sources 23 are electrically connected to a controller 2, so that each light source 23 can be turned on or off via the controller 22.

[0053] Continue to refer to Figure 2 and Figure 3 In an optional embodiment of this disclosure, the side-viewing device group includes at least two image acquisition devices 21 respectively deployed on both sides of the conveying direction of the product to be inspected, and the top-viewing device group includes at least three image acquisition devices 21 deployed in a triangle above the product to be inspected.

[0054] In this embodiment, the multiple light sources 21 include multiple shadowless ring light sources installed around each image acquisition device 21 and at least one dome light source installed on the top of the inspection chamber. That is, the image acquisition devices 21 in the side-viewing device group can be surrounded by shadowless ring light sources, while the image acquisition devices 21 in the side-viewing device group are surrounded by dome light sources. When the multiple light sources 21 are turned on, they can provide illumination at different angles to the product under inspection. Among them, the low-angle shadowless ring light sources are installed around the lenses of cameras 1 to 7 to highlight minor surface undulations and defects (scratches and pits will produce light and shadow effects at the edges). The dome light source is installed on the top of the inspection chamber to form a uniform integrating sphere illumination environment, which can greatly eliminate specular reflections on the metal surface and make the surface texture and defects clearly imaged.

[0055] In an optional embodiment of this disclosure, the angle between the image acquisition device 21 and the light source 23 ranges from 0 to 90°, and the angle can be appropriately adjusted according to the different shapes of the product to be inspected or the different requirements for welding defect detection.

[0056] In this embodiment, multiple image acquisition devices in the newly designed welding defect detection system can capture images of the product under inspection from different perspectives, avoiding missed images or images lacking defect detection-related information. This provides a higher quality image foundation for the final defect detection results and improves detection accuracy.

[0057] Figure 4 This is a flowchart illustrating a welding defect detection method according to some embodiments of the present disclosure.

[0058] like Figure 1 As shown, this welding defect detection method is applied to a welding defect detection system as described in the above embodiments. The method includes the following steps:

[0059] S401: In response to the product under inspection reaching the target inspection area, control multiple image acquisition devices to perform image acquisition operations to obtain multiple inspection images of the product from different perspectives.

[0060] In this embodiment, the detection equipment built into the production line, either an infrared sensor or a displacement sensor, can detect whether a product to be inspected has moved to the target detection area. This target detection area can be the starting point area in the direction of the conveyor belt's movement, or it can be a portion of the conveyor belt covered by the field of view of the foremost image acquisition device. Figure 2 The area directly below the equipment group shown in the top view.

[0061] In this embodiment, a control signal can be sent to the image acquisition device to control the device to power on and photograph the product under inspection. Furthermore, this is done to ensure better lighting conditions for photographing.

[0062] In an optional embodiment of this disclosure, specific illumination can also be provided by controlling multiple light sources. The specific light sources have been described in the above system embodiments and will not be repeated here. The multiple different viewpoints may include, but are not limited to, the following viewpoints: top view, left oblique top view and right oblique top view, front side view, left front oblique side view, left rear oblique side view, right front oblique side view and right rear oblique side view.

[0063] S402: Construct a 3D model of the product to be inspected based on multiple images of the product from different perspectives and camera parameters.

[0064] In this embodiment, image features can be extracted from multiple images of the product under inspection from different perspectives. These features are then combined with camera parameters such as the distance between the image acquisition device capturing the current image and the product under inspection, the angle between the image acquisition device and the product under inspection, and the pose parameters of the image acquisition device. Based on this data, the specific three-dimensional position information of each pixel in the image under inspection corresponding to a point on the actual product under inspection can be determined. Furthermore, based on this three-dimensional position information, a three-dimensional model of the product under inspection can be constructed using three-dimensional drawing tools.

[0065] To further reduce the discrepancy between the 3D model of the product to be inspected and the actual product, in an optional embodiment of this disclosure, the image acquisition device can be calibrated before acquiring the image to be inspected. Therefore, the method further includes:

[0066] Step A: Control multiple image acquisition devices to capture images of the calibration plate as it moves and rotates multiple times within the detection field of view, obtaining multiple sets of calibration plate images.

[0067] In this embodiment, a high-precision checkerboard calibration board can be used. This calibration board is moved and rotated multiple times within the detection field of view of each image acquisition device. Simultaneously, all image acquisition devices are controlled to capture multiple sets of images of the calibration board for subsequent calibration.

[0068] Step B: Perform joint solution processing on multiple sets of calibration board images to obtain the internal and external parameters of each image acquisition device.

[0069] In this embodiment, common calibration methods such as Zhang Zhengyou's calibration method can be used to jointly solve the internal parameters of multiple image acquisition devices, such as focal length and distortion coefficient, and external parameters, such as the relative position and orientation between cameras, i.e., the global coordinate system. Finally, the calibration is completed based on these internal and external parameters.

[0070] In this embodiment, the internal and external parameters of each image acquisition device can be obtained by calibrating each image acquisition device. This allows for more accurate construction of the 3D model of the product to be inspected after capturing images of the product to be inspected. It also reduces the differences in size, bending angle, etc., between the 3D model and the product to be inspected, thereby providing a more accurate model to improve the accuracy of defect detection.

[0071] The following will explain how to construct the 3D model to be inspected, using specific examples.

[0072] Based on the above embodiments, in an optional embodiment of this disclosure, step S402 includes:

[0073] S402a: Extract target feature points from the image to be inspected.

[0074] In this embodiment, robust feature points can be extracted from the image to be inspected as target feature points through an artificial intelligence agent or a pre-selected feature extraction model. For example, robust feature points can be scale-invariant feature transform SIFT or transform and rotate ORB features.

[0075] S402b: Determine the feature correspondence between each target feature point based on the target feature points and camera parameters.

[0076] In this embodiment, the internal and external parameters of the calibrated image acquisition device can be used to significantly narrow the matching search range through epipolar constraints, thereby quickly and accurately establishing the correspondence between feature points in all images.

[0077] S402c: Determine the initial sparse point cloud data based on the feature correspondence and target feature points.

[0078] In this embodiment, initial sparse point cloud data can be formed through sparse point cloud reconstruction. This can be achieved by matching the features of target feature points with feature correspondences, and then calculating the three-dimensional coordinates of these target feature points using triangulation principles to form the initial sparse point cloud data.

[0079] S402d: Determine the dense point cloud based on the initial sparse point cloud data and the image to be inspected.

[0080] In this embodiment, a dense point cloud can be generated by dense point cloud reconstruction. The dense point cloud reconstruction method may include using a multi-view stereo (MVS) algorithm to match pixels one by one and generate a dense point cloud containing hundreds of thousands or even millions of three-dimensional points, which can completely restore the surface morphology of the product to be inspected, such as a U-shaped tube and the welding area.

[0081] S402e: Use dense point cloud as the 3D model of the product to be inspected.

[0082] In this embodiment, all dense point clouds are placed in a pre-calibrated coordinate system to obtain the final 3D model to be inspected.

[0083] S403: Defect detection and judgment are performed based on the three-dimensional model to be inspected and the standard three-dimensional model to obtain the defect detection results.

[0084] In this embodiment, defect detection and judgment can be achieved by comparing data points of some welded parts in the 3D model to be inspected with those in the standard 3D model one by one to determine whether welding defects have occurred in these parts. When welding defects are found in the welded parts of the 3D model to be inspected, the similarity depth may differ from that of the standard 3D model.

[0085] In an optional embodiment of this disclosure, all data points of the three-dimensional model to be inspected can be compared one by one with those of the standard three-dimensional model to detect defects other than welding defects, thereby further improving the yield rate of production.

[0086] Based on the above embodiments, in an optional embodiment of this disclosure, the 3D model to be inspected includes multiple 3D points to be matched; step S403 includes:

[0087] S403a: Determine independent defect areas based on each 3D point to be matched and the standard 3D model.

[0088] In this embodiment, each 3D point to be matched can be precisely matched with standard 3D points in a standard 3D model. Then, the regions where the matched data points show discrepancies are located are identified as independent defect regions.

[0089] Specifically, in an optional embodiment of this disclosure, step S403a includes:

[0090] S403a1: Perform a matching calculation operation based on each 3D point to be matched and the standard 3D model to obtain the distance deviation from each 3D point to be matched to the corresponding surface of the standard 3D model.

[0091] In this embodiment, the Iterative Closest Point (ICP) algorithm can be used to accurately register each 3D point to be matched with the standard 3D point in the standard 3D model, and the deviation between the registered 3D point to be matched and the corresponding standard 3D point can be calculated to obtain the deviation value.

[0092] S403a2: Generate a deviation chromatogram based on the distance deviation between each 3D point to be matched and the corresponding surface of the standard 3D model.

[0093] In this embodiment, to more intuitively demonstrate the degree of deviation of each 3D point to be matched, color difference is used to represent the magnitude of the deviation. That is, the intensity of the color is used to indicate the size of the deviation; for example, the darker the color, the greater the distance deviation.

[0094] S403a3: Define the regions in the deviation chromatogram that deviate to a preset deviation threshold as independent defect regions.

[0095] In this embodiment, regions in the deviation chromatogram that deviate to a preset deviation threshold can be identified as independent defect regions through defect segmentation and identification methods. Specifically, algorithms such as region growing, cluster analysis, and detection networks can be used to automatically segment independent defect regions from the deviation map. Defects are then classified according to their geometric characteristics (such as area, depth, volume, and aspect ratio): scratches (slender and deep), pits (local depressions), weld beads (local protrusions), deformation, misinstalled or missing U-shaped tubes, etc.

[0096] S403b: Determine the defect type and the corresponding defect feature quantification value based on the independent defect area.

[0097] In this embodiment, within an independent defect area, based on the intuitive 3D model to be inspected and the independent defect areas already identified in the above embodiments, the system automatically analyzes the geometric features of the specific independent defect area (such as area, depth, volume, and aspect ratio) to classify defects: scratches (slender and deep), pits (local depressions), weld beads (local protrusions), deformation, misinstalled or missing U-shaped tubes, etc., to determine the defect type. Then, based on the depth, aspect ratio, volume, area, and other data of these defect types, the welding defects are quantified into defect feature quantification values. For example, if the maximum pit depth is 1.5 mm, according to pre-defined rules, it can be associated with the corresponding feature quantification value. For example, 1.5 mm is greater than 1 mm, the pit is deeper, and the feature quantification value is 2 (indicating that the defect is relatively serious and cannot be judged as acceptable).

[0098] Furthermore, in an optional embodiment of this disclosure, step S403b, determining the defect type based on the independent defect region, includes:

[0099] S403b1: Perform defect identification operations based on independent defect regions to obtain the defect geometric features of the independent defect regions.

[0100] S403b2: Determine the defect type of an independent defect region based on the defect geometry.

[0101] In this embodiment, algorithms such as region growing, cluster analysis, and detection networks can be used to automatically identify and segment independent defect regions from the deviation map. Then, the defects are separated based on the geometric characteristics of each independent defect region. For example, if an independent defect region is a pit, the welding defect type is a pit or a localized pit. If an independent defect region is a protrusion, the environmental defect type can be a weld bead or a localized weld protrusion.

[0102] S403c: Defect quality is scored based on defect type and defect feature quantification value to obtain defect detection results.

[0103] In this embodiment, a pre-defined scoring model can be used to perform defect cooling scoring on the input defect type and defect feature quantification values, and finally output specific defect detection results.

[0104] Based on the above embodiments, in an optional embodiment of this disclosure, step S403c includes:

[0105] S403c1: Determine the defect score corresponding to each defect type based on the defect feature quantification value and preset weight data.

[0106] S403c2: Determine the overall quality score of the product to be inspected based on the defect score corresponding to each defect type.

[0107] S403c3: The overall quality score and the defect score corresponding to each defect type are used as the defect detection results.

[0108] In this embodiment, a defect score corresponding to each defect type can be determined by scoring or deducting points for each defect. Then, points are deducted for each defect based on pre-set weighted data. For example, defects located at weld points have a higher weight, deep defects are penalized more severely than shallow defects, and points are deducted for defects detected in each independent defect area. The final output comprehensive cooling score can be calculated on a percentage or ten-point scale. This improves the accuracy of the scoring results, thereby increasing the accuracy of defect detection results.

[0109] In summary, the welding defect detection method provided in an optional embodiment of this disclosure firstly involves controlling multiple image acquisition devices to acquire multiple images of the product under inspection from different perspectives when the product reaches the target area. This avoids missing welding defects in the acquired images, resulting in more comprehensive images for subsequent inspection. Then, based on the multiple images from different perspectives and camera parameters, a three-dimensional model of the product under inspection is constructed. This three-dimensional model comprehensively reproduces the welding defects on the actual product. Finally, the three-dimensional model under inspection is compared with a standard three-dimensional model to obtain accurate defect detection results, thus improving the accuracy of the detection results.

[0110] Based on the above embodiments, in an optional embodiment of this disclosure, the welding defect detection system further includes multiple light sources, which are used to provide illumination to the surface of the product under inspection from different perspectives; before controlling multiple image acquisition devices to perform image acquisition operations in step S401 to obtain multiple images of the product under inspection from different perspectives, the system further includes:

[0111] S401a: Controls the activation of multiple light sources to provide illumination to the surface of the product under inspection from different angles.

[0112] In this embodiment, after the product under inspection has reached the target inspection area but before image acquisition, multiple light sources can be activated to provide illumination to the surface of the product under inspection from different perspectives. Then, multiple image acquisition devices are controlled to acquire images of the product under inspection from multiple different perspectives. The resulting images are of higher quality than those acquired before the light sources are activated, reducing shadows and noise caused by insufficient lighting, thus improving image quality. This allows the 3D model constructed during subsequent defect detection to more closely resemble the actual product under inspection, thereby improving the accuracy of welding defect detection results.

[0113] To further improve detection accuracy, in an optional embodiment of this disclosure, before extracting target feature points from the image to be inspected in step S402a of the above embodiment, the method further includes:

[0114] Step S402a1: Preprocess the image to be inspected to subtract the optical dark field, eliminate uneven illumination and reduce noise, and obtain the processed image to be inspected.

[0115] In this embodiment, preprocessing can be achieved through methods such as noise reduction using filtering algorithms and optical parameter correction, subtracting optical dark fields and flattening corrections to eliminate uneven illumination and reduce noise. This results in a clearer and higher-quality image, providing support for subsequent feature extraction and other processes. Ultimately, this contributes to improving the accuracy of welding defect detection. To quickly detect welding defects, prevent defective products from entering the next process, improve product yield, and provide welding suggestions and reference data for welding operations on the production line of the products under inspection, in an optional embodiment of this disclosure, it further includes:

[0116] S404: Send the defect detection results to the welding equipment of the product to be inspected so that the user can adjust the welding equipment according to the defect detection results.

[0117] In this embodiment, defect detection results can be sent to the controller or display device of the welding equipment for the product under inspection via wireless or wired communication, allowing users to view them or for the processor to automatically retrieve them, thus improving the welding quality of subsequent product assembly. In this embodiment, defect detection results may include, but are not limited to, the overall cooling score of the component under inspection, the inspection results indicating whether the product under inspection is qualified, a detailed inspection report, the type of welding defect, and welding recommendations.

[0118] Figure 5 This is a block diagram illustrating a welding defect detection device according to some embodiments of the present disclosure. (Refer to...) Figure 5 The device includes an image acquisition module 51, a multi-view 3D reconstruction module 52, and a defect detection module 53.

[0119] The image acquisition module 51 is used to control multiple image acquisition devices to perform image acquisition operations in response to the product under inspection reaching the target detection area, so as to obtain multiple images of the product under inspection from different perspectives.

[0120] The multi-view 3D reconstruction module 52 is used to construct a 3D model of the product to be inspected based on multiple images of the product to be inspected from different perspectives and camera parameters.

[0121] The defect detection module 53 performs defect detection and judgment based on the three-dimensional model to be inspected and the standard three-dimensional model, and obtains the defect detection results.

[0122] In an optional embodiment of this disclosure, the image acquisition module 51 is specifically used to: control multiple image acquisition devices to capture images of the calibration plate that moves and rotates multiple times within the detection field of view, thereby obtaining multiple sets of calibration plate images; and perform joint solution processing on the multiple sets of calibration plate images to obtain the internal and external parameters of each image acquisition device.

[0123] In an optional embodiment of this disclosure, the multi-view 3D reconstruction module 52 is specifically used for: extracting target feature points from the image to be inspected; determining the feature correspondence between each target feature point based on the target feature points and camera parameters; determining initial sparse point cloud data based on the feature correspondence and the target feature points; determining dense point cloud based on the initial sparse point cloud data and the image to be inspected; and using the dense point cloud as the 3D model of the product to be inspected.

[0124] In an optional embodiment of this disclosure, the three-dimensional model to be inspected includes multiple three-dimensional points to be matched; the defect detection module 53 is specifically used to: determine independent defect regions based on each three-dimensional point to be matched and the standard three-dimensional model; determine the defect type and the corresponding defect feature quantification value based on the independent defect region; and perform defect quality scoring based on the defect type and defect feature quantification value to obtain the defect detection result.

[0125] In an optional embodiment of this disclosure, the defect detection module 53 is specifically used to: perform a matching calculation operation based on each three-dimensional point to be matched and the standard three-dimensional model to obtain the distance deviation from each three-dimensional point to be matched to the corresponding surface of the standard three-dimensional model; generate a deviation chromatogram based on the distance deviation from each three-dimensional point to be matched to the corresponding surface of the standard three-dimensional model; and determine the areas in the deviation chromatogram that deviate to a preset deviation threshold as independent defect areas.

[0126] In an optional embodiment of this disclosure, the defect detection module 53 is specifically used to: perform a defect identification operation based on an independent defect region to obtain the defect geometric features of the independent defect region; and determine the defect type of the independent defect region based on the defect geometric features.

[0127] In an optional embodiment of this disclosure, the defect detection module 53 is specifically used to: score the defect quality based on the defect type and the defect feature quantification value to obtain the defect detection result, including: determining the defect score corresponding to each defect type based on the defect feature quantification value and preset weight data; determining the comprehensive quality score of the product to be inspected based on the defect score corresponding to each defect type; and using the comprehensive quality score and the defect score corresponding to each defect type as the defect detection result.

[0128] In an optional embodiment of this disclosure, before the multi-view 3D reconstruction module 52 extracts target feature points from the image to be inspected, the image acquisition module 51 is further configured to: preprocess the image to be inspected to subtract the optical dark field in the image to be inspected, eliminate uneven illumination and reduce noise, so as to obtain the processed image to be inspected.

[0129] In an optional embodiment of this disclosure, the defect detection module 53 is further configured to: send the defect detection results to the welding equipment of the product to be inspected, so that the user can adjust the welding equipment according to the defect detection results.

[0130] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0131] Figure 6 This is a block diagram of an electronic device according to some embodiments of the present disclosure. For example, the electronic device may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0132] Reference Figure 6 The electronic device may include one or more of the following components: processing component 602, memory 604, power component 606, multimedia component 608, audio component 610, input / output (I / O) interface 612, sensor component 614, and communication component 616.

[0133] Processing component 602 typically controls the overall operation of an electronic device, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0134] Memory 604 is configured to store various types of data to support the operation of the electronic device. Examples of this data include instructions for any application or method operating on the electronic device, contact data, phonebook data, messages, pictures, videos, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0135] Power component 606 provides power to various components of an electronic device. Power component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0136] Multimedia component 608 includes a screen that provides an output interface between an electronic device and a user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the electronic device is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0137] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0138] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0139] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of the electronic device. For example, sensor assembly 614 can detect the on / off state of the electronic device, the relative positioning of components such as the display and keypad of the electronic device, changes in the position of the electronic device or a component of the electronic device, the presence or absence of user contact with the electronic device, the orientation or acceleration / deceleration of the electronic device, and temperature changes of the electronic device. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0140] Communication component 616 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0141] In some embodiments of this disclosure, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0142] In some embodiments of this disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions that can be executed by a processor 620 of an electronic device to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0143] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements a welding defect detection method as described in the above method embodiments.

[0144] Figure 7 This is a block diagram of a chip mentality provided according to some of the embodiments shown in this disclosure.

[0145] Some embodiments of this disclosure also provide a chip system, such as Figure 7 As shown, the chip system includes at least one processor 701 and at least one interface circuit 702. The processor 701 and the interface circuit 702 are interconnected via lines. For example, the interface circuit 702 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 702 can be used to send signals to other devices (e.g., the processor 701). Exemplarily, the interface circuit 702 can read instructions stored in the memory and send those instructions to the processor 701. When the instructions are executed by the processor 701, the electronic device can perform the steps in the above method embodiments. Of course, the chip system may also include other discrete devices, and some embodiments of this disclosure do not specifically limit this.

[0146] In some embodiments of this disclosure, the interface circuit 702 can acquire data, program instructions, and / or information from the internal storage area of ​​the chip system; it can also acquire data, program instructions, and / or information from outside the chip system.

[0147] Optionally, the chip system also includes a memory 703, which is used to store necessary computer programs and data.

[0148] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

[0149] In the above detailed description, reference has been made to the accompanying drawings, which illustrate specific aspects of this disclosure by way of illustration. In this regard, terms indicating direction or positional relationship, such as “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “rear,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” “counterclockwise,” “axial,” “radial,” and “circumferential,” are used with reference to the orientation of the described figures. Since components of the described device can be positioned in multiple different orientations, directional terms are used for illustrative purposes and not for limitation. It should be understood that other aspects can be utilized and structural or logical changes can be made without departing from the concept of this disclosure. Therefore, the following detailed description should not be considered limiting.

[0150] It should be understood that, unless otherwise specifically indicated, features of various embodiments of this disclosure described herein can be combined with each other. As used herein, the term “and / or” includes any one of the relevant listed items and any combination of any two or more; similarly, “at least one of…” includes any one of the relevant listed items and any combination of any two or more.

[0151] It should be understood that, unless otherwise expressly specified and limited, the terms "joining," "attaching," "installing," "connecting," "linking," "fixing," etc., used in the embodiments of this disclosure should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms herein based on the specific circumstances.

[0152] Furthermore, the term "above" as used herein with respect to components, elements, or material layers formed or located "above" a surface may be used to indicate that the component, element, or material layer is "indirectly" positioned (e.g., placed, formed, deposited, etc.) on the surface such that one or more additional components, elements, or layers are arranged between the surface and the component, element, or material layer. However, the term "above" as used with respect to components, elements, or material layers formed or located "above" a surface may also optionally have a specific meaning: that the component, element, or material layer is "directly" positioned (e.g., placed, formed, deposited, etc.) on the surface, for example, in direct contact with the surface.

[0153] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited to these terms. Rather, these terms are used only to distinguish one component, part, region, layer, or section from another. Therefore, without departing from the teachings of the examples described herein, the first component, part, region, layer, or section mentioned in the examples may also be referred to as the second component, part, region, layer, or section. Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include at least one of that feature. In the description herein, “a plurality” means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0154] It should be understood that spatial relative terms, such as “above,” “upper,” “below,” and “lower,” are used herein to describe the relationship between one element and another shown in the figures. In addition to the orientation depicted in the figures, these spatial relative terms are also intended to encompass different orientations of the device in use or operation. For example, if the device in the figures is flipped, an element described as “above” or “upper” relative to another element would be “below” or “lower” relative to that other element. Thus, depending on the spatial orientation of the device, the term “above” encompasses both above and below orientations. Devices may have other orientations (e.g., rotated 90 degrees or in other orientations), and the spatial relative terms used herein should be interpreted accordingly.

[0155] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0156] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0157] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0158] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A welding defect detection system, characterized in that, include: Multiple image acquisition devices and controllers; The plurality of image acquisition devices are all electrically connected to the controller; The plurality of image acquisition devices include a group of side-viewing devices deployed diagonally in front of and diagonally behind the product to be inspected, and a group of top-viewing devices deployed above the product to be inspected. The image acquisition device is used to capture images of the product under inspection, thereby obtaining images of the product under inspection from different perspectives. The controller is used to detect welding defects based on the image to be inspected, and obtain the defect detection result.

2. The welding defect detection system according to claim 1, characterized in that, Also includes: Multiple light sources; Each of the image acquisition devices is accompanied by a light source, which is used to illuminate the surface of the product under inspection from different angles.

3. The welding defect detection system according to claim 1, characterized in that, The side-viewing device group includes at least two image acquisition devices respectively deployed on both sides of the conveying direction of the product to be inspected, and the top-viewing device group includes at least three image acquisition devices deployed in a triangle above the product to be inspected.

4. A method for detecting welding defects, characterized in that, The method, applied to the welding defect detection system as described in claim 1, comprises: In response to the product under inspection reaching the target detection area, the multiple image acquisition devices are controlled to perform image acquisition operations to obtain multiple images of the product under inspection from different perspectives; Based on the multiple images to be inspected from different perspectives and camera parameters, a 3D model of the product to be inspected is constructed. Defect detection and judgment are performed based on the three-dimensional model to be inspected and the standard three-dimensional model to obtain the defect detection results.

5. The method according to claim 4, characterized in that, The method includes: Multiple image acquisition devices are controlled to capture images of the calibration plate, which moves and rotates multiple times within the detection field of view, to obtain multiple sets of calibration plate images; The multiple sets of calibration board images are jointly solved to obtain the internal and external parameters of each image acquisition device.

6. The method according to claim 4, characterized in that, The step of constructing a 3D model of the product to be inspected based on the image to be inspected and camera parameters includes: Extract target feature points from the image to be inspected; Based on the target feature points and the camera parameters, determine the feature correspondence between each target feature point; Based on the feature correspondence and the target feature points, determine the initial sparse point cloud data; Based on the initial sparse point cloud data and the image to be inspected, a dense point cloud is determined; The dense point cloud is used as the three-dimensional model of the product to be inspected.

7. The method according to claim 4, characterized in that, The 3D model to be inspected includes multiple 3D points to be matched; The defect detection operation based on the three-dimensional model to be inspected and the standard three-dimensional model, to obtain the defect detection result, includes: Based on each 3D point to be matched and the standard 3D model, determine the independent defect area; Based on the independent defect regions, the defect type and the corresponding defect feature quantification value are determined; Based on the defect type and the quantitative value of the defect characteristics, a defect quality score is obtained to obtain the defect detection result.

8. The method according to claim 7, characterized in that, The process of determining independent defect regions based on each 3D point to be matched and the standard 3D model includes: A matching calculation operation is performed based on each 3D point to be matched and the standard 3D model to obtain the distance deviation from each 3D point to be matched to the corresponding surface of the standard 3D model; A deviation chromatogram is generated based on the distance deviation between each 3D point to be matched and the corresponding surface of the standard 3D model; The regions in the deviation chromatogram that deviate to a preset deviation threshold are identified as independent defect regions.

9. The method according to claim 7, characterized in that, The step of determining the defect type based on the independent defect region includes: Perform a defect identification operation based on the independent defect region to obtain the defect geometric features of the independent defect region; The defect type of the independent defect region is determined based on the defect geometry.

10. The method according to claim 7, characterized in that, The step of scoring the defect quality based on the defect type and the quantified value of the defect features to obtain the defect detection result includes: Based on the quantified value of the defect features and the preset weight data, the defect score corresponding to each defect type is determined; The overall quality score of the product under inspection is determined based on the defect score corresponding to each defect type. The overall quality score and the defect score corresponding to each defect type are used as the defect detection results.

11. The method according to claim 4, characterized in that, The welding defect detection system also includes multiple light sources, which are used to provide illumination to the surface of the product under inspection from different perspectives; before controlling the multiple image acquisition devices to perform image acquisition operations and obtain multiple images of the product under inspection from different perspectives, the system further includes: The multiple light sources are activated to provide illumination to the surface of the product under inspection from different angles.

12. The method according to claim 6, characterized in that, Before extracting target feature points from the image to be inspected, the process also includes: The image to be inspected is preprocessed to remove the optical dark field, eliminate uneven illumination, and reduce noise, resulting in a processed image to be inspected.

13. The method according to claim 4, characterized in that, Also includes: The defect detection results are sent to the welding equipment of the product to be inspected, so that the user can adjust the welding equipment according to the defect detection results.

14. A welding defect detection device, characterized in that, The device, applied to the welding defect detection system as described in claim 1, comprises: The image acquisition module is used to control the multiple image acquisition devices to perform image acquisition operations in response to the product under inspection reaching the target detection area, so as to obtain multiple images of the product under inspection from different perspectives. The multi-view 3D reconstruction module is used to construct a 3D model of the product under inspection based on the multiple images of the product under inspection from different perspectives and camera parameters. The defect detection module performs defect detection and judgment based on the three-dimensional model to be inspected and the standard three-dimensional model, and obtains the defect detection result.

15. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform a welding defect detection method as described in any one of claims 4 to 13.

16. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the mobile terminal, the mobile terminal is able to perform a welding defect detection method as described in any one of claims 4 to 13.

17. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements a welding defect detection method as described in any one of claims 4 to 13.

Images