Special-shaped forge piece surface defect detection method

Abstract

The invention discloses a special-shaped forge piece surface defect detection method, and belongs to the technical field of machine vision inspection.The method comprises the steps that a depth image and point cloud data of a reference sample piece are obtained to serve as reference data; wherein the reference sample piece refers to a standard forge piece which is the same as a forge piece to be detected in model and has no surface defect; acquiring a grayscale image and a depth image of the to-be-detected forging; based on the reference data and the depth image of the to-be-detected forge piece, obtaining a surface relative depth image of the to-be-detected forge piece; and based on the surface relative depth image and the gray level image of the to-be-detected forge piece, forge piece surface defect detection is realized. The forging surface defect detection efficiency and precision can be greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision detection, in particular to a special-shaped forge piece surface defect detection method. BACKGROUND

[0002] In the production process of forge pieces, due to the initial defects such as inclusions and surface microcracks in raw materials, combined with improper process parameter control in the heating, deformation and cooling stages, various types of defects are easily formed on the surface of the forge pieces. Therefore, a reliable forge piece surface defect detection scheme is crucial for ensuring product quality.

[0003] However, compared with regular-shaped steel materials such as strip steel and wide-thick plates, the automatic detection of special-shaped forge piece surface defects is significantly more difficult. The surface of the forge piece is often in a multi-curved, uneven and heterogeneous shape, which makes it difficult for the detection probe to achieve full coverage and easily forms detection blind spots, posing a serious challenge to the integrity and consistency of imaging. The surface of the forge piece is generally rough, with uneven reflection, and often has interference such as oxide skin and oil stains, resulting in low image signal-to-noise ratio, which seriously affects the recognition accuracy and stability of the visual detection method. SUMMARY

[0004] The present application provides a special-shaped forge piece surface defect detection method to solve the technical problem that the recognition accuracy and stability of the prior art are poor for special-shaped forge piece surface defect detection, which is difficult to meet the actual production needs.

[0005] To solve the above technical problems, the present application provides the following technical solutions: On the one hand, the present application provides a special-shaped forge piece surface defect detection method, comprising: obtaining the depth image and point cloud data of a reference sample as reference data; wherein the reference sample refers to a standard forge piece of the same model as the forge piece to be detected without surface defects; obtaining the gray-scale image and depth image of the forge piece to be detected; based on the reference data and the depth image of the forge piece to be detected, obtaining the surface relative depth image of the forge piece to be detected; based on the surface relative depth image and the gray-scale image of the forge piece to be detected, realizing forge piece surface defect detection.

[0006] Further, the obtaining of the depth image and point cloud data of the reference sample comprises: The reference sample is placed in a detection position, and a 3D surface structured light camera is used to collect a gray image and a depth image of the reference sample; the depth image of the reference sample is sequentially subjected to multi-dimensional denoising, point cloudization and downsampling to obtain point cloud data of the reference sample, and the point cloud data of the reference sample is further segmented into parts according to actual regions of the forging prone to defects to obtain point cloud data corresponding to each part, thereby forming a part point cloud set.

[0007] Further, the gray image and the depth image of the forging to be detected are obtained, including: The forging to be detected is placed in a detection position, and a 3D surface structured light camera is used to collect a gray image and a depth image of the forging to be detected; wherein the gray image and the depth image of the forging to be detected are matrices of the same size.

[0008] Further, based on the reference data and the depth image of the forging to be detected, a surface relative depth image of the forging to be detected is obtained, including: The depth image of the forging to be detected is sequentially subjected to multi-dimensional denoising, point cloudization and downsampling to obtain point cloud data of the forging to be detected; The point cloud data of the forging to be detected is registered with the point cloud data of the reference sample to obtain a set of registration rigid body matrices; Based on the set of registration rigid body matrices, dimension reduction mapping and data difference are realized to obtain a plurality of surface relative depth images corresponding to the forging to be detected.

[0009] Further, the process of multi-dimensional denoising includes: The gray image corresponding to the depth image to be denoised is subjected to global threshold segmentation to extract a preliminary contour of the target object; then the contour boundary is reduced through a morphological erosion operation to retain a stable region closer to the center of the field of view to generate a binary effective region template; finally, the binary effective region template is used for mask operation on the depth image to extract the data within the template as effective depth image data after denoising, and the values of the remaining positions are set to 0; The foreground depth lower limit threshold and the background depth upper limit threshold are set, and the invalid region exceeding the measurement range is located through threshold segmentation; then a morphological erosion operation is performed to optimize the connectivity of the invalid region to obtain an effective region template; finally, the effective region template is used for mask operation on the depth image to be denoised to retain the data within the template as effective depth image data after denoising, and the values of the remaining positions are set to 0; For each pixel point in the depth image to be denoised, the average distance of the K nearest neighbor points is calculated, and the average distances of all points form a Gaussian distribution; then, according to the mean and standard deviation of the Gaussian distribution, a distance threshold is set, and the pixel points with an average distance exceeding the distance threshold are determined as outliers, and their values are set to 0, thereby retaining the depth image data with a required density; wherein K is a preset integer value.

[0010] Further, the point cloud data of the to-be-detected forging is registered with the point cloud data of the reference sample to obtain a registration rigid body matrix set, including: The point cloud data of the complete reference sample and the point cloud data of the to-be-detected forging are registered, including: performing down-sampling processing on the point cloud data to be registered; extracting features of the down-sampled point cloud data through KPConv convolution to obtain rotation equivariant feature description and rotation invariant feature description respectively; realizing point cloud registration based on the rotation equivariant feature description and the rotation invariant feature description to obtain an original rigid body matrix; Based on the original rigid body matrix, the point cloud data of the to-be-detected forging is converted to obtain converted point cloud; the converted point cloud and the point cloud data of the reference sample are registered again, at this time, the registration process adopts an ICP iterative registration method, but only the point cloud in the component point cloud set is considered when calculating the error; the component point clouds in the component point cloud set are iterated in sequence to register, and the rigid body matrix corresponding to the minimum registration error of each component point cloud is reserved to form a registration rigid body matrix set together with the original rigid body matrix.

[0011] Further, based on the registration rigid body matrix set, dimension reduction mapping and data difference are realized to obtain a plurality of surface relative depth images corresponding to the to-be-detected forging, including: The registration rigid body matrix set is traversed, and for the traversed rigid body matrix, the following steps are executed: Step 1, converting the point cloud data of the to-be-detected forging by using the rigid body matrix; Step 2, the values of the point cloud data converted in step 1 in each direction The index value is obtained according to the following formula respectively And the depth value corresponding to the index value , and the value under the condition of is reserved to obtain a set ; ; Wherein, represents the floor function; is the resolution of the camera in the Z direction; is the distance value from the calibration plane to the camera during camera calibration; is the focal length of the camera lens; is the width of the camera target surface; is the height of the camera target surface; is the number of pixels of the depth image data in the width direction; is the number of pixels of the depth image data in the height direction; Step 3, a zero matrix with a size of is constructed, and the set is traversed ,by Set the value of the corresponding coordinate position in the zero matrix to the coordinate. The adjusted depth image of the forging to be tested is obtained, and the projection mapping from the point cloud to the regular mesh is completed. Step 4: Construct an effective domain template based on the depth image of the reference sample and the adjusted depth image of the forging to be tested. : ; in, For valid field templates Middle position The value; Position in the depth image of the reference sample The value, Position in the depth image of the forging under test after adjustment Value; valid field template The regions with a value of 1 are valid regions, and the regions with a value of 0 are invalid regions. Step 5: Perform data difference between the two depth images and multiply them with the effective domain template to obtain the surface relative depth image of the forging to be tested; After traversing the set of registered rigid body matrices, multiple relative depth images of the surfaces of the forging to be inspected are obtained.

[0012] Furthermore, the method of detecting surface defects in forgings based on the relative depth image and grayscale image of the surface of the forging to be inspected includes: The surface relative depth image of the forging to be inspected is input into the preset first defect detection model to obtain the first detection result; The grayscale image of the forging to be inspected is input into the preset second defect detection model to obtain the second detection result; The first detection result and the second detection result are combined using the maximum suppression method to obtain the fused defect detection result, which is used as the surface defect detection result of the forging to be tested.

[0013] In another aspect, the present invention also provides an electronic device comprising a processor and a memory; wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described method.

[0014] In another aspect, the present invention also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the above method.

[0015] The beneficial effects of the technical solution provided by this invention include at least the following: The special-shaped forge piece surface defect detection method provided by the application realizes accurate detection of the surface defects of the forge piece by comprehensively utilizing the depth image and the gray image collected by the camera. The specific processing procedure comprises: multi-dimensional denoising, point cloud processing, depth registration, dimension reduction mapping and data difference of the original depth data, and finally the relative depth information reflecting the surface topography is extracted. The method fully utilizes the three-dimensional features contained in the depth data, and can effectively solve the detection problems caused by the geometric complexity and uneven surface of the special-shaped forge piece. Through the dimension reduction mapping technology, the defect recognition process which is traditionally dependent on three-dimensional registration is converted into efficient analysis on the two-dimensional depth image. This conversion not only significantly improves the detection speed, but also can represent the subtle morphology of the defects with higher density of pixels, thereby comprehensively improving the recognition accuracy and comprehensive detection capability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0017] Figure 1 is a flowchart of the special-shaped forge piece surface defect detection method provided by the embodiment of the application. Figure 2 is a system block diagram of the electronic device provided by the embodiment of the application. DETAILED DESCRIPTION

[0018] In order to make the objects, technical solutions and advantages of the application clearer, the following will further describe the embodiments of the application in detail with reference to the drawings.

[0019] First of all, it should be noted that in the embodiments of the application, the words such as "exemplarily", "for example" are used to represent as an example, illustration or description. Any embodiment or design scheme described as "exemplary" in the application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "exemplarily" is intended to present the concept in a specific way. In addition, in the embodiments of the application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0020] First embodiment

[0021] Aiming at the irregular shape of the special-shaped forgings, the surface imaging is difficult; and since the collected point cloud can only obtain distance value, it cannot directly show the actual quality of the surface, can only reflect the contour structure of the forgings, and it is difficult to realize the defect detection based on the point cloud. The embodiment provides a special-shaped forging surface defect detection method to solve the problem of special-shaped forging surface defect detection. The method sets multiple 3D face structured light cameras around the measured forging to collect multiple key positions on the surface of the special-shaped forging. The data output by the camera includes a grayscale image and a depth image, based on which subsequent defect detection is realized. The method can be realized by an electronic device, which can be a terminal or a server. The execution process of the method includes the following steps: S1, obtaining the depth image and point cloud data of the reference sample as reference data; wherein the reference sample refers to a standard forging without surface defects of the same model as the forging to be detected; It should be noted that, in the embodiment, before the forging surface defect detection, a standard forging without surface defects is placed at the detection position as a reference sample, and a set of measurement data for the reference sample is obtained as a reference for subsequent online detection of the forging surface defects. The depth image of the reference sample is subjected to the steps of multi-dimensional denoising, point cloud, and downsampling to obtain the point cloud data. The point cloud data of the reference sample is further segmented into multiple component point cloud sets according to the areas prone to defects of the actual forging.

[0022] S2, obtaining the grayscale image and depth image of the forging to be detected; It should be noted that, in the embodiment, when the forging surface defect detection is performed, the forging to be detected is placed at the detection position, and multiple 3D face structured light cameras are set around the measured forging to collect multiple key positions on the surface of the special-shaped forging. The data output by the camera includes a grayscale image and a depth image, wherein the grayscale image and the depth image are matrices of the same size.

[0023] S3, obtaining the surface relative depth image of the forging to be detected based on the reference data and the depth image of the forging to be detected; It should be noted that, in the embodiment, after the cameras collect the data of the measured part, the depth images output by the cameras are subjected to multi-dimensional denoising, point cloud, downsampling, depth registration, dimension reduction mapping, and data difference to obtain the surface relative depth image. Wherein, The step of multi-dimensional denoising is to sequentially subject the depth image to noise suppression based on the shooting range, noise suppression based on the measurement depth, and noise suppression based on the point density to obtain the denoised depth data, which is as follows: 1) Noise suppression based on shooting range: Global thresholding is performed on the grayscale image output by the camera to extract the preliminary outline of the target object; then, morphological erosion is used to reduce the outline boundary, retaining a stable region closer to the center of the field of view, and generating a binary effective region template; finally, the template is used to perform a masking operation on the depth image data, and the data within the template is extracted as the effective depth image data after denoising, while the values ​​at other positions are set to 0.

[0024] 2) Noise suppression based on depth of field measurement: Set a lower threshold for the foreground depth of field and an upper threshold for the background depth of field for the depth data. Use threshold segmentation to locate invalid regions that exceed the measurement range. Then perform morphological erosion to optimize the connectivity of invalid regions and obtain an effective region template. Finally, use this template to perform a masking operation on the depth image data, retain the data within the template as the effective depth image data after denoising, and set the values ​​of other positions to 0.

[0025] 3) Noise suppression based on point density: For each pixel, calculate the average distance to its K nearest neighbors. The average distance of all points forms a Gaussian distribution. Then, based on the mean and standard deviation of this distribution, set a distance threshold. Pixels whose average distance exceeds the threshold are identified as outliers and their values ​​are set to 0, thereby retaining depth image data with the required density and improving overall smoothness. Here, K is a preset value.

[0026] Point cloudification is the process of converting depth image data acquired by a camera into point cloud coordinate format; where, it is assumed that the number of pixels in the width direction of the depth image data is... The number of pixels in the height direction of the depth image data is Each pixel corresponds to the width and height coordinates of the image. Then, the data of each pixel in the depth image is converted into point cloud coordinates by the following formula:

[0027] The newly obtained point cloud is , Let Z be the camera's resolution. The distance from the calibration plane to the camera during camera calibration. The focal length of the camera lens. For coordinates The value of the depth image at the location, The dimension in the width direction of the camera target surface. The dimension is the height of the camera target surface. During the point cloudification process, only data with non-zero values ​​in the depth image are retained.

[0028] The depth registration process involves registering the point cloud generated from the depth image of the forging to be tested with the point cloud generated from the depth image of the reference sample. Specific steps include: 1) First, the complete reference sample point cloud and the point cloud of the forging under test output by each camera are registered. The steps include: ① Point cloud downsampling: The input point cloud is downsampled to reduce data density and computational complexity, while retaining key geometric features; ② Feature extraction: Features are extracted by KPConv convolution to obtain rotationally equivalent descriptions and rotationally invariant descriptions; ③ Coarse registration: The overlap of the two point clouds is calculated based on the rotationally invariant feature description. The isovariant description is compared with 60 isovariants of the reference based on the overlapping area. The minimum L2 distance is taken to obtain the basic rotational registration and determine the approximate spatial transformation relationship; ④ Fine registration: The rotationally equivalent feature description is input into the regression network and fine estimation is achieved based on the residual loss. The registration accuracy is optimized to the sub-millimeter level to obtain the rigid body matrix. 2) Multidimensional Auxiliary Registration: Based on the rigid body matrix obtained in the previous step, the point cloud of the forging to be tested is transformed to obtain the transformed point cloud; the transformed point cloud is registered with the point cloud of the reference sample again. At this time, the fine registration process adopts the ICP iterative registration method, but only the point cloud in the part point cloud set is considered when calculating the error; the point clouds in the reference part point cloud set are traversed sequentially for registration, and the rigid body matrix with the smallest registration error corresponding to each part point cloud is retained (each registration parameter can ensure that each local area can be better aligned and characterized, making the local registration more refined, thereby highlighting more subtle defect features), together with the original rigid body matrix to obtain the registration rigid body matrix set.

[0029] The process of dimensionality reduction mapping and data differencing is as follows: Select rigid body matrices sequentially from the set of rigid body matrices obtained from registration, and then proceed with the subsequent steps; 1) Use rigid body matrix to generate the original point cloud of the forging under test. Transform into ; 2) Values ​​of the transformed point cloud in each direction The index values ​​are obtained according to the following formulas. and the depth value corresponding to that index value. and retain The values ​​under the given conditions are obtained as a set. ;

[0030] in, This indicates rounding down. Let Z be the camera's resolution. The distance from the calibration plane to the camera during camera calibration. The focal length of the camera lens. The dimension in the width direction of the camera target surface. The dimension in the height direction of the camera target surface. This represents the number of pixels in the width direction of the depth image data. This represents the number of pixels in the height direction of the depth image data. 3) The size of the construction is The zero matrix, traversing the set ,by Set the value of the corresponding coordinate position in the zero matrix to the coordinate. The adjusted depth image matrix of the forging under test is obtained, and the projection mapping from the point cloud to the regular grid is completed. 4) Construct an effective domain template based on the depth image matrix of the reference piece and the adjusted depth image matrix of the forging to be tested. From the formula, we get:

[0031] in, Position in the depth image matrix of the reference part The value, Position in the adjusted depth image matrix of the forging under test The value is 1 in the template, which is a valid area and 0 is an invalid area; 5) Perform data difference between the two depth image matrices and multiply them with the effective domain template to obtain the surface relative depth image matrix; By traversing the rigid body matrix set and performing steps 1 to 5, multiple surface relative depth image matrices can be obtained for subsequent defect detection.

[0032] S4, based on the relative depth image and grayscale image of the surface of the forging to be inspected, realizes the detection of surface defects of the forging.

[0033] Specifically, in this embodiment, the implementation process of S4 is as follows: S41. The collected grayscale images are labeled to construct a dataset and the detection model is trained to obtain the first defect detection model. This model can be a single-stage model of the YOLO series, such as YOLOv5, YOLOv8 and corresponding variants, or a two-stage model of the RCNN series, such as Faster RCNN and Mask RCNN, to perform defect detection on grayscale images. S42, the surface relative depth image is labeled to construct a dataset, and the detection model is trained to obtain the second defect detection model. This model can use the traditional feature extraction + support vector machine method, or it can use deep learning object detection network, such as FCOS, CenterNet, Yolov5, etc., to perform defect detection on the surface relative depth image. S43. After performing defect detection on the grayscale image and the surface relative depth image, the defect detection results of the two are merged according to the maximum suppression method to obtain the fused detection result.

[0034] In summary, this embodiment provides a method for detecting surface defects in irregularly shaped forgings, such as... Figure 1 As shown, this method for detecting surface defects in irregularly shaped forgings utilizes a magic hand structure to install cameras at key points on the forging to acquire grayscale and depth image data. By registering these data with reference data from a standard part, the registered point cloud is projected back to the depth image data, and then differentially derived to obtain a relative depth map of the surface. The grayscale image and multiple sets of relative depth maps are then used to detect defects using a model, obtaining defect location boxes and category information. This information is then fused to obtain the final defect data. By using multiple auxiliary registration matrices to fully uncover minute defects and by using depth images obtained through dimensionality reduction, the method avoids the insufficient processing speed problem associated with point cloud detection, thus significantly improving the efficiency and accuracy of forging surface defect detection.

[0035] Second Embodiment

[0036] This embodiment provides an electronic device, such as... Figure 2 As shown, the electronic device includes a processor and a memory; wherein the processor and the memory can be connected via a communication bus; the memory stores at least one instruction, which is loaded and executed by the processor to implement the method of the first embodiment described above. Furthermore, the electronic device may also include a transceiver, the processor and the transceiver can be connected via a communication bus, and the transceiver is used to communicate with other devices.

[0037] Below, in conjunction with Figure 2 A detailed introduction to each component of this electronic device is provided below: The processor is the control center of the electronic device. The electronic device may include multiple processors, each of which can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The term "processor" can refer to a single processor or a collective term for multiple processing elements. For example, a processor can be one or more central processing units (CPUs), other general-purpose processors, application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), one or more field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0038] In a specific implementation, as one example, the processor may include one or more CPUs, for example... Figure 2 CPU0 and CPU1 shown are, of course, merely illustrative examples.

[0039] The memory is used to store the software program that executes the solution of the present invention, and the processor controls its execution. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0040] Optionally, the memory may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may be integrated with the processor or exist independently, and may be accessed through the interface circuit of the electronic device ( Figure 2 (Not shown in the image) is coupled to the processor; however, this embodiment of the invention does not impose specific limitations on this.

[0041] The transceiver may include a receiver and a transmitter. Figure 2 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function. The transceiver can be integrated with the processor or exist independently, and can be connected through the interface circuit of the electronic device (…). Figure 2 (Not shown in the image) is coupled to the processor, and this embodiment of the invention does not specifically limit this.

[0042] In addition, it should be noted that, Figure 2 The structure of the electronic device shown is not intended to limit the device. Actual devices may include more or fewer components than shown, or combine certain components, or have different component arrangements. Furthermore, the technical effects achieved by this electronic device when performing the method of the first embodiment described above can be referenced to the technical effects described in the first embodiment; therefore, they will not be repeated here.

[0043] Third Embodiment

[0044] This embodiment provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method of the first embodiment described above. The computer-readable storage medium may be a ROM, random access memory, CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc. The instruction stored therein can be loaded and executed by a processor in a terminal.

[0045] Furthermore, it should be noted that the present invention can be provided as a method, apparatus, or computer program product. Therefore, embodiments of the present invention can take the form of a completely or partially hardware embodiment, a completely or partially software embodiment, or an embodiment combining software and hardware aspects. Moreover, when implemented in software, embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any usable medium accessible to a computer or a data storage device such as a server or data center containing one or more sets of usable media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive (SSD).

[0046] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0047] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0048] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element. Furthermore, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Additionally, the character " / " in this text generally indicates an "or" relationship between the preceding and following objects, but it can also indicate an "AND / OR" relationship. Please refer to the context for specific interpretations. "At least one" refers to one or more items, while "more than" refers to two or more items. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can be represented as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0049] Furthermore, it is understood that in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0050] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0051] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of functional modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Additionally, the functional units in the various embodiments of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0052] If the method is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0053] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments of the present invention have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make several improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A method for detecting surface defects of a profiled forging, characterized by, The method comprises the following steps: obtaining a depth image and point cloud data of a reference sample as reference data; wherein the reference sample refers to a standard forging without surface defects of the same type as the forging to be detected; obtaining a grayscale image and a depth image of the forging to be detected; obtaining a surface relative depth image of the forging to be detected based on the reference data and the depth image of the forging to be detected; realizing the detection of surface defects of the forging based on the surface relative depth image and the grayscale image of the forging to be detected.

2. The method of surface defect detection of a profiled forging according to claim 1, wherein The method for obtaining the depth image and the point cloud data of the reference sample comprises the following steps: placing the reference sample at a detection position and using a 3D surface structured light camera to collect a grayscale image and a depth image of the reference sample; sequentially performing multi-dimensional denoising, point cloudization and downsampling on the depth image of the reference sample to obtain point cloud data of the reference sample; and further segmenting the point cloud data of the reference sample into parts according to regions of the actual forging prone to defects to obtain point cloud data corresponding to each part and form a part point cloud set.

3. The method of claim 1, wherein the surface defect of the profiled forging is detected by the method comprising: The method for obtaining the grayscale image and the depth image of the forging to be detected comprises the following steps: placing the forging to be detected at a detection position and using a 3D surface structured light camera to collect a grayscale image and a depth image of the forging to be detected; wherein the grayscale image and the depth image of the forging to be detected are matrices of the same size.

4. The method of claim 2, wherein the surface defect of the profiled forging is detected by the method comprising: The method for obtaining a surface relative depth image of the forging to be detected based on the reference data and the depth image of the forging to be detected comprises the following steps: sequentially performing multi-dimensional denoising, point cloudization and downsampling on the depth image of the forging to be detected to obtain point cloud data of the forging to be detected; aligning the point cloud data of the forging to be detected with the point cloud data of the reference sample to obtain a set of registration rigid body matrices; realizing dimension reduction mapping and data difference based on the set of registration rigid body matrices to obtain a plurality of surface relative depth images corresponding to the forging to be detected.

5. The method of surface defect detection of a profiled forging according to claim 2 or 4, characterized in that, The process of multi-dimensional denoising comprises the following steps: performing global threshold segmentation on the grayscale image corresponding to the depth image to be denoised to extract the preliminary contour of the target object; subsequently reducing the contour boundary through a morphological erosion operation to retain a stable region closer to the center of the field of view to generate a binary effective region template; and finally performing a mask operation on the depth image using the binary effective region template to extract the data within the template as effective depth image data after denoising, and setting the values of the remaining positions to 0; setting a foreground depth lower threshold and a background depth upper threshold, and positioning the invalid region exceeding the measurement range through threshold segmentation based on the set threshold; subsequently performing a morphological erosion operation to optimize the connectivity of the invalid region to obtain an effective region template; and finally performing a mask operation on the depth image to be denoised using the effective region template to retain the data within the template as effective depth image data after denoising, and setting the values of the remaining positions to 0; for each pixel point in the depth image to be denoised, calculating the average distance of the K nearest neighbor points, and the average distances of all points form a Gaussian distribution; subsequently setting a distance threshold based on the mean and standard deviation of the Gaussian distribution, and determining the pixel points with average distances exceeding the distance threshold as outliers, and setting their values to 0, so as to retain the depth image data with a density meeting the requirements; wherein K is a preset integer value.

6. The method of surface defect detection of a profiled forging according to claim 4, wherein The point cloud data of the to-be-detected forging and the point cloud data of the reference sample are registered to obtain a registration rigid body matrix set, including: The point cloud data of the complete reference sample and the point cloud data of the to-be-detected forging are registered, including: performing down-sampling processing on the point cloud data to be registered; extracting features of the down-sampled point cloud data through KPConv convolution to obtain rotation equivariant feature description and rotation invariant feature description respectively; realizing point cloud registration based on the rotation equivariant feature description and the rotation invariant feature description to obtain an original rigid body matrix; Based on the original rigid body matrix, the point cloud data of the to-be-detected forging is converted to obtain converted point cloud; the converted point cloud and the point cloud data of the reference sample are registered again, at this time, the registration process adopts an ICP iterative registration method, but only the point cloud in the component point cloud set is considered when calculating the error; the component point clouds in the component point cloud set are iterated in sequence to register, and the rigid body matrix corresponding to the minimum registration error of each component point cloud is reserved to form a registration rigid body matrix set together with the original rigid body matrix.

7. The method of surface defect detection of a profiled forging according to claim 4, wherein Based on the registration rigid body matrix set, dimension reduction mapping and data difference are realized to obtain a plurality of surface relative depth images corresponding to the to-be-detected forging, including: The registration rigid body matrix set is iterated, and for the iterated rigid body matrix, the following steps are executed: Step 1, converting the point cloud data of the to-be-detected forging by using the rigid body matrix; Step 2, the values in each direction of the point cloud data converted in Step 1 The index value is obtained according to the following formula respectively The depth value corresponding to the index value And the values under the condition of The set is obtained ; ； wherein, denotes a floor function; is a resolution of the camera in the Z direction; is a distance value from a calibration plane to the camera at the time of camera calibration; is a focal length of the camera lens; is a size of the camera target surface in the width direction; is a size of the camera target surface in the height direction; is a number of pixels of the depth image data in the width direction; is a number of pixels of the depth image data in the height direction; Step 3, construct a structure with a size of The zero matrix, traversing the set ,by Set the value of the corresponding coordinate position in the zero matrix to the coordinate. The adjusted depth image of the forging to be tested is obtained, and the projection mapping from the point cloud to the regular mesh is completed. Step 4, constructing an effective field template according to the depth image of the reference sample and the adjusted depth image of the to-be-tested forging : ； wherein, is the effective field template is the value of the position in the depth image of the reference sample is the value of the position in the depth image of the reference sample is the value of the position in the depth image of the reference sample is the value of the position in the depth image of the reference sample is the value of the position in the depth image of the reference sample is the effective field template is the effective field template Step 5, performing data difference on the two depth images, and multiplying the effective domain template to obtain a surface relative depth image of the to-be-detected forging; After iterating the registration rigid body matrix set, a plurality of surface relative depth images of the to-be-detected forging are obtained.

8. The method of claim 1, wherein The surface relative depth image and the gray image of the to-be-detected forging are used to realize forging surface defect detection, including: The surface relative depth image of the to-be-detected forging is input into a preset first defect detection model to obtain a first detection result; The gray image of the to-be-detected forging is input into a preset second defect detection model to obtain a second detection result; The first detection result and the second detection result are merged by using a maximum value suppression method to obtain a fused defect detection result as a surface defect detection result of the to-be-detected forging.

Images