Defect positioning method and device, storage medium and chip

Abstract

The invention relates to a defect positioning method and device, a storage medium and a chip. Acquiring a plurality of X-ray images of a to-be-detected workpiece at different angles and shooting pose information of each X-ray image; for each X-ray image, performing defect detection on the X-ray image to obtain a defect image corresponding to the X-ray image; according to each defect image and the shooting pose information of the X-ray image corresponding to each defect image, constructing a first reconstruction model of the to-be-detected workpiece; constructing a second reconstruction model of the to-be-detected workpiece according to the design three-dimensional model of the to-be-detected workpiece, the first reconstruction model and the shooting pose information corresponding to each defect image; and generating a defect stereoscopic model according to the first reconstruction model and the second reconstruction model. The defect three-dimensional model not only can visually display the position of the defect, but also can reflect the position correlation of the defect in a three-dimensional space. The three-dimensional defect form of the defect three-dimensional model is beneficial to defect cause analysis, and the production efficiency and the production yield are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a defect location method, device, storage medium and chip. Background Art

[0002] Defect detection refers to the inspection of products during the manufacturing process to find possible defects and repair them in time to ensure that product quality meets standards. With the development of the manufacturing industry, defect detection technology is also constantly improving, but there are still some difficulties and challenges.

[0003] The goal of defect detection is to determine the yield rate of the product and control defective products within the factory to prevent them from affecting the performance of mass-produced parts. However, a more important goal is to find the cause of the defect, improve the process technology, fundamentally solve the problem, and improve the yield rate of production. However, the current defect detection method is basically to detect through two-dimensional images formed by X-ray scanning, which cannot intuitively display the location of the defect and the position correlation of the defect in three-dimensional space. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a defect location method, device, storage medium and chip.

[0005] According to a first aspect of an embodiment of the present disclosure, a defect location method is provided, including:

[0006] Acquire multiple X-ray images of the workpiece to be inspected at different angles and shooting posture information of each of the X-ray images;

[0007] For each of the X-ray images, performing defect detection on the X-ray image to obtain a defect image corresponding to the X-ray image;

[0008] Constructing a first reconstruction model of the workpiece to be inspected according to each of the defect images and the shooting posture information of the X-ray image corresponding to each of the defect images;

[0009] Constructing a second reconstructed model of the workpiece to be inspected according to the designed three-dimensional model of the workpiece to be inspected, the first reconstructed model, and the shooting posture information corresponding to each of the defect images;

[0010] A defect three-dimensional model of the workpiece to be inspected is generated according to the first reconstruction model and the second reconstruction model.

[0011] Optionally, constructing a second reconstructed model of the workpiece to be inspected according to the designed three-dimensional model of the workpiece to be inspected, the first reconstructed model and the shooting posture information corresponding to each of the defect images includes:

[0012] Determine the design component contained in each defect image according to the design three-dimensional model of the workpiece to be inspected, the first reconstructed model, and the shooting posture information corresponding to each defect image;

[0013] According to the design components contained in each of the defect images and the shooting posture information corresponding to each of the defect images, a second reconstruction model of the workpiece to be inspected is constructed by using a pre-constructed modeling model;

[0014] Optionally, determining the design component included in each defect image according to the design three-dimensional model of the workpiece to be inspected, the first reconstructed model and the shooting pose information corresponding to each defect image includes:

[0015] Registering the designed three-dimensional model of the workpiece to be inspected and the first reconstructed model;

[0016] For each defect image, the design component included in the defect image is determined according to the design three-dimensional model obtained after registration and the shooting posture information corresponding to the defect image.

[0017] Optionally, determining the design component included in the defect image according to the design three-dimensional model obtained after registration and the shooting posture information corresponding to the defect image includes:

[0018] Determining, according to the shooting posture information corresponding to the defect image, a ray irradiation angle when shooting an X-ray image corresponding to the defect image;

[0019] Projecting the designed three-dimensional model obtained after registration according to the ray irradiation angle to obtain a projection image;

[0020] The design component in the projection image is determined as the design component included in the defective image.

[0021] Optionally, constructing the second reconstruction model of the workpiece to be inspected by using a pre-constructed modeling model according to the design components contained in each of the defect images and the shooting posture information corresponding to each of the defect images, comprises:

[0022] For each of the defect images, according to the design components contained in the defect image, the defect image is annotated with design components to obtain a target image corresponding to the defect image;

[0023] Input each of the target images and the shooting posture information corresponding to each of the target images into a pre-constructed modeling model to obtain a second reconstructed model of the workpiece to be inspected, wherein the shooting posture information corresponding to the target image is the shooting posture information corresponding to the defect image corresponding to the target image.

[0024] Optionally, the modeling model is trained in the following manner:

[0025] Acquire multiple reference images of a reference workpiece, shooting posture information corresponding to each of the reference images, and a reference stereo model of the reference workpiece, wherein the multiple reference images include multiple reference X-ray images at different angles with components annotated thereon;

[0026] The model is trained by taking the multiple reference images and the shooting posture information corresponding to each of the reference images as the input of the model and taking the reference stereo model as the target output of the model to obtain the model.

[0027] Optionally, generating the defect three-dimensional model of the workpiece to be inspected according to the first reconstruction model and the second reconstruction model includes:

[0028] The outer edge contour of the first reconstructed model is superimposed with the internal structure of the second reconstructed model to obtain a three-dimensional defect model of the workpiece to be inspected.

[0029] Optionally, generating the defect three-dimensional model of the workpiece to be inspected according to the first reconstruction model and the second reconstruction model includes:

[0030] Superimposing the outer edge contour of the first reconstructed model with the internal structure of the second reconstructed model to obtain a superimposed model;

[0031] Registering the designed three-dimensional model of the workpiece to be inspected and the first reconstructed model;

[0032] The superimposed model is calibrated according to the designed three-dimensional model obtained after the registration to obtain a three-dimensional defect model of the workpiece to be inspected.

[0033] Optionally, calibrating the superposition model according to the designed three-dimensional model obtained after the registration includes:

[0034] For each designed component in the superposition model, the position and posture of the designed component in the superposition model is calibrated according to the position and posture information of the designed component in the designed three-dimensional model obtained after alignment, wherein the position and posture of the defective component in the superposition model is adjusted as the position and posture of the designed component closest to it is adjusted.

[0035] According to a second aspect of an embodiment of the present disclosure, there is provided a defect locating device, comprising:

[0036] A first acquisition module is configured to acquire a plurality of X-ray images of the workpiece to be inspected at different angles and shooting posture information of each of the X-ray images;

[0037] A defect detection module is configured to perform defect detection on each X-ray image to obtain a defect image corresponding to the X-ray image;

[0038] A first model building module is configured to build a first reconstruction model of the workpiece to be inspected according to each of the defect images and the shooting posture information of the X-ray image corresponding to each of the defect images;

[0039] A second model building module is configured to build a second reconstructed model of the workpiece to be inspected according to the designed three-dimensional model of the workpiece to be inspected, the first reconstructed model and the shooting posture information corresponding to each of the defect images;

[0040] A generating module is configured to generate a defect three-dimensional model of the workpiece to be inspected according to the first reconstructed model and the second reconstructed model.

[0041] According to a third aspect of an embodiment of the present disclosure, there is provided a defect locating device, comprising:

[0042] processor;

[0043] a memory for storing processor-executable instructions;

[0044] Wherein, the processor is configured to:

[0045] Acquire multiple X-ray images of the workpiece to be inspected at different angles and shooting posture information of each of the X-ray images;

[0046] For each of the X-ray images, performing defect detection on the X-ray image to obtain a defect image corresponding to the X-ray image;

[0047] Constructing a first reconstruction model of the workpiece to be inspected according to each of the defect images and the shooting posture information of the X-ray image corresponding to each of the defect images;

[0048] Constructing a second reconstructed model of the workpiece to be inspected according to the designed three-dimensional model of the workpiece to be inspected, the first reconstructed model, and the shooting posture information corresponding to each of the defect images;

[0049] A defect three-dimensional model of the workpiece to be inspected is generated according to the first reconstruction model and the second reconstruction model.

[0050] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the defect localization method provided in the first aspect of the present disclosure are implemented.

[0051] According to a fifth aspect of an embodiment of the present disclosure, a chip is provided, comprising a processor and an interface; the processor is used to read instructions to execute the defect location method provided in the first aspect of the present disclosure.

[0052] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: first, defect detection is performed on multiple X-ray images of the workpiece to be detected at different angles to obtain defect images corresponding to each X-ray image; then, a first reconstruction model of the workpiece to be detected is constructed according to each defect image and the shooting posture information of the X-ray image corresponding to each defect image; then, a second reconstruction model of the workpiece to be detected is constructed according to the design three-dimensional model of the workpiece to be detected, the first reconstruction model and the shooting posture information corresponding to each defect image; finally, a defect stereo model of the workpiece to be detected is generated according to the first reconstruction model and the second reconstruction model. In this way, computer vision detection (i.e., defect detection using computer vision) and image mapping matching can be combined to achieve stereo restoration of multiple two-dimensional X-ray images of the workpiece to be detected at different angles. The defect stereo model obtained by stereo restoration can not only intuitively display the location of the defect, but also reflect the position correlation of the defect in three-dimensional space. In addition, the three-dimensional defect morphology of the defect stereo model is helpful for defect cause analysis and improves production efficiency and production yield. In addition, in theory, each defective component is mapped in the defect images at different angles, but the falsely detected defective components usually exist in individual defect images. Therefore, by generating a defect 3D model based on multiple defect images at different angles, some of the falsely detected components can be filtered out based on multi-image voting, thereby improving the accuracy of defect positioning.

[0053] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0055] Figure 1 The figure is a flow chart of a defect location method according to an exemplary embodiment.

[0056] Figure 2 The figure is a flowchart of a method for training a modeling model according to an exemplary embodiment.

[0057] Figure 3 The figure is a block diagram of a defect location device according to an exemplary embodiment.

[0058] Figure 4 It is a block diagram of a device for defect location according to an exemplary embodiment.

[0059] Figure 5 It is a block diagram of a device for defect location according to an exemplary embodiment. DETAILED DESCRIPTION

[0060] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0061] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.

[0062] Figure 1 FIG. 1 is a flow chart of a defect location method according to an exemplary embodiment, wherein the method can be applied to a client or a server. Figure 1 As shown, the defect location method may include the following S101 to S105.

[0063] In S101, a plurality of X-ray images of a workpiece to be inspected at different angles and shooting posture information of each X-ray image are obtained.

[0064] In the present disclosure, a multi-directional X-ray image acquisition can be performed on the workpiece to be inspected from different directions by a X-ray machine to obtain multiple X-ray images at different angles, and the posture information of the X-ray machine is recorded each time the image is acquired. The posture information is the position and posture of the X-ray machine when the X-ray image of the workpiece to be inspected is obtained; the X-ray image and the shooting posture information correspond one to one.

[0065] In S102, for each X-ray image, defect detection is performed on the X-ray image to obtain a defect image corresponding to the X-ray image.

[0066] In the present disclosure, defect detection can be achieved by using X-ray nondestructive testing technology. Among them, X-ray nondestructive testing is a detection method that uses the strong penetrating power of X-rays to detect internal defects in the workpiece to be detected. It mainly detects internal defects in the workpiece to be detected based on the changes in the strength of the X-rays that penetrate the workpiece to be detected. If there are defects in a local area of ​​the workpiece to be detected, it will change the attenuation of the object to the rays, causing changes in the intensity of the transmitted rays. In this way, by using certain detection methods, such as using film photosensitization to detect the intensity of the transmitted rays, it is possible to determine whether there are defects in the workpiece and the location and size of the defects.

[0067] Specifically, for each X-ray image, a defect detection can be performed on the X-ray image to obtain a detection result, and then, based on the detection result, a preset first annotation tool is used to annotate the defective component on the X-ray image to obtain a defect image corresponding to the X-ray image. The first annotation tool refers to a preset tool that can perform component-level segmentation annotation, and the first annotation tool can be, for example, a Labelme image annotation tool.

[0068] In S103, a first reconstruction model of the workpiece to be inspected is constructed according to each defect image and the shooting posture information of the X-ray image corresponding to each defect image.

[0069] In the present disclosure, a three-dimensional model, ie, a first reconstructed model of the workpiece to be inspected, can be constructed based on a multi-view stereo algorithm according to each defect image and the shooting posture information of the X-ray image corresponding to each defect image.

[0070] In S104, a second reconstructed model of the workpiece to be inspected is constructed according to the designed three-dimensional model of the workpiece to be inspected, the first reconstructed model, and the shooting posture information corresponding to each defect image.

[0071] In the present disclosure, each workpiece will be designed with a corresponding three-dimensional structure diagram before production and processing. The designed three-dimensional model of the workpiece to be inspected is a three-dimensional structure diagram designed for the target workpiece before production, for example, a CAD three-dimensional stereogram. The three-dimensional structure diagram includes at least one component, hereinafter referred to as a design component, wherein the workpiece to be inspected belongs to the target workpiece.

[0072] In one implementation, a second reconstructed model of the workpiece to be inspected may be constructed through a mapping matching algorithm based on the designed three-dimensional model of the workpiece to be inspected, the first reconstructed model, and the shooting posture information corresponding to each defect image.

[0073] In S105 , a defect three-dimensional model of the workpiece to be inspected is generated according to the first reconstruction model and the second reconstruction model.

[0074] In the present disclosure, the defect three-dimensional model includes defect components and design components, which can intuitively display the location of the defect and also reflect the position correlation of the defect in three-dimensional space.

[0075] In one embodiment, the defect location method can be applied to defect location of various components on a vehicle, for example, to defect location of a vehicle body, and specifically, to defect location of an underbody.

[0076] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: first, defect detection is performed on multiple X-ray images of the workpiece to be detected at different angles to obtain defect images corresponding to each X-ray image; then, a first reconstruction model of the workpiece to be detected is constructed according to each defect image and the shooting posture information of the X-ray image corresponding to each defect image; then, a second reconstruction model of the workpiece to be detected is constructed according to the design three-dimensional model of the workpiece to be detected, the first reconstruction model and the shooting posture information corresponding to each defect image; finally, a defect stereo model of the workpiece to be detected is generated according to the first reconstruction model and the second reconstruction model. In this way, computer vision detection (i.e., defect detection using computer vision) and image mapping matching can be combined to achieve stereo restoration of multiple two-dimensional X-ray images of the workpiece to be detected at different angles. The defect stereo model obtained by stereo restoration can not only intuitively display the location of the defect, but also reflect the position correlation of the defect in three-dimensional space. In addition, the three-dimensional defect morphology of the defect stereo model is helpful for defect cause analysis and improves production efficiency and production yield. In addition, in theory, each defective component is mapped in the defect images at different angles, but the falsely detected defective components usually exist in individual defect images. Therefore, by generating a defect 3D model based on multiple defect images at different angles, some of the falsely detected components can be filtered out based on multi-image voting, thereby improving the accuracy of defect positioning.

[0077] The following is a detailed description of the specific implementation method of constructing the second reconstructed model of the workpiece to be inspected according to the designed three-dimensional model of the workpiece to be inspected, the first reconstructed model, and the shooting posture information corresponding to each defect image in the above S104. Specifically, it can be achieved by the following steps (1) and (2):

[0078] Step (1): Determine the design components contained in each defect image based on the design three-dimensional model of the workpiece to be inspected, the first reconstructed model and the shooting posture information corresponding to each defect image.

[0079] In the present disclosure, determining the design components contained in the defect image is to determine which design components in the design three-dimensional model are contained in the defect image, that is, to determine the real internal components (ie, design components) corresponding to the local component images in the defect image.

[0080] Step (2): Based on the design components contained in each defect image and the shooting posture information corresponding to each defect image, a second reconstruction model of the workpiece to be inspected is constructed through a pre-constructed modeling model.

[0081] In the present disclosure, the modeling model constructs a second reconstruction model of the tool to be inspected by learning the depth information of the internal components of the workpiece to be inspected.

[0082] The following is a detailed description of the specific implementation method of determining the design component contained in each defect image according to the design 3D model of the workpiece to be inspected, the first reconstructed model and the shooting posture information corresponding to each defect image in the above step (1). Specifically, it can be achieved by the following steps (11) and (12):

[0083] Step (11): aligning the designed three-dimensional model of the workpiece to be inspected and the first reconstructed model.

[0084] Specifically, the Iterative Closest Point (ICP) registration algorithm or the Perspective-N-Point registration algorithm can be used to register the designed 3D model of the workpiece to be inspected and the first reconstructed model. The so-called ICP registration algorithm is to place two point clouds in the same coordinate system to calculate a pose transformation so that one point cloud can be matched (i.e., overlapped) with the other point cloud to the greatest extent through the pose transformation.

[0085] Step (12): For each defect image, the design component included in the defect image is determined based on the design three-dimensional model obtained after registration and the shooting posture information corresponding to the defect image.

[0086] Specifically, the irradiation angle when shooting the X-ray image corresponding to the defect image can be determined according to the shooting posture information corresponding to the defect image; then, the design three-dimensional model obtained after the registration is projected according to the irradiation angle to obtain a projection image; finally, the design component in the projection image is determined as the design component included in the defect image. Among them, the irradiation angle is the angle at which the camera shoots the workpiece to be inspected to obtain the X-ray image corresponding to the defect image.

[0087] The following is a detailed description of the specific implementation method of constructing the second reconstruction model of the workpiece to be inspected according to the design components contained in each defect image and the shooting posture information corresponding to each defect image in the above step (2) through the pre-constructed modeling model. Specifically, it can be achieved by the following steps (21) and (22):

[0088] Step (21): for each defect image, the defect image is annotated with design components according to the design components contained in the defect image to obtain a target image corresponding to the defect image.

[0089] In the present disclosure, a preset second annotation tool can be used to annotate the design component in the defect image to obtain a target image corresponding to the defect image. The second annotation tool refers to a preset tool that can perform component-level segmentation annotation, and the second annotation tool can be, for example, a Labelme image annotation tool.

[0090] Step (22): input each target image and the shooting posture information corresponding to each target image into a pre-constructed model to obtain a second reconstructed model of the workpiece to be inspected.

[0091] The shooting posture information corresponding to the target image is the shooting posture information corresponding to the defect image corresponding to the target image.

[0092] The following is a detailed description of the specific training method for the above modeling model. Figure 2 The above modeling model is obtained by training S201 and S202 shown in .

[0093] In S201 , a plurality of reference images of a reference workpiece, shooting posture information corresponding to each reference image, and a reference stereo model of the reference workpiece are obtained.

[0094] In the present disclosure, the multiple reference images include multiple reference X-ray images at different angles with components annotated. Among them, the reference workpiece can be collected from different orientations by a X-ray machine to obtain multiple reference X-ray images at different angles, and the posture information of the X-ray machine at each image acquisition is recorded, that is, the shooting posture information of each reference X-ray image; then, for each reference X-ray image, the reference X-ray image is annotated with components, that is, each component in the reference X-ray image is annotated to obtain a reference image corresponding to the reference X-ray image, and the shooting posture information of the reference X-ray image is determined as the shooting posture information corresponding to the corresponding reference image.

[0095] The reference solid model of the reference workpiece may be a designed three-dimensional model of the reference workpiece.

[0096] In S202, model training is performed by taking a plurality of reference images and shooting posture information corresponding to each reference image as input of the model building model and taking the reference stereo model as the target output of the model building model to obtain the model building model.

[0097] Specifically, multiple reference images and the shooting posture information corresponding to each reference image can be input into the modeling model to obtain a reference reconstruction model; then, the model parameters of the modeling model are updated according to the difference between the reference reconstruction model and the reference stereo model; thereafter, new training data (i.e., multiple reference images of new reference workpieces, the shooting posture information corresponding to each reference image, and the reference stereo model of the reference workpiece) are reacquired to continue model training, i.e., return to the above S201 until the training cutoff condition is reached.

[0098] In one embodiment, the training cutoff condition may be that the number of trainings reaches a preset number, which can be set according to the actual usage scenario. When the number of trainings reaches the preset number, it can be determined that the number of trainings is sufficient so that the modeling model can learn enough in-depth information about the internal structure of the workpiece.

[0099] In another embodiment, the training cutoff condition may be that the target loss of the modeling model is less than a preset threshold, and the preset threshold may be set according to the actual usage scenario. When the target loss of the modeling model is less than the preset threshold, it can be considered that the three-dimensional model construction accuracy of the modeling model meets the accuracy requirements and can achieve accurate construction of the three-dimensional model.

[0100] The following is a detailed description of a specific implementation method of generating a defect stereo model of the workpiece to be inspected based on the first reconstruction model and the second reconstruction model in the above S105. Specifically, it can be implemented through a variety of implementation methods. In one implementation method, the outer edge contour of the first reconstruction model can be superimposed with the internal structure of the second reconstruction model to obtain the defect stereo model of the workpiece to be inspected.

[0101] In the present disclosure, the outer edge contour of the first reconstructed model constructed by the multi-view stereo algorithm is basically consistent with the edge contour of the designed three-dimensional model of the workpiece to be detected, but because the defect image used to construct the first reconstructed model is still an X-ray image, when the defect image in the form of X-ray is used to construct the first reconstructed model, the image depth is judged by the depth of the shadow. In this way, it is impossible to accurately construct the internal structure of the workpiece to be detected. Therefore, based on the designed three-dimensional model of the workpiece to be detected, the first reconstructed model and the shooting posture information corresponding to each defect image, a second reconstructed model of the workpiece to be detected is constructed, wherein the second reconstructed model can accurately reflect the internal structure of the workpiece to be detected. Therefore, after obtaining the first reconstructed model and the second reconstructed model, the outer edge contour of the first reconstructed model can be superimposed with the internal structure of the second reconstructed model to obtain a defect stereo model of the workpiece to be detected, thereby improving the accuracy of the defect stereo model to improve the accuracy of defect positioning.

[0102] In the model obtained by superimposing the outer edge contour of the first reconstructed model with the internal structure of the second reconstructed model (hereinafter referred to as the superimposed model), each component may have a position or angle offset. In order to further improve the accuracy of the defect three-dimensional model and the accuracy of defect positioning, the position and angle of each component in the superimposed model can be calibrated according to the design three-dimensional model obtained after the workpiece to be inspected is aligned. Specifically, in another embodiment, the outer edge contour of the first reconstructed model can be superimposed with the internal structure of the second reconstructed model to obtain a superimposed model; then, the design three-dimensional model of the workpiece to be inspected and the first reconstructed model are aligned; finally, the superimposed model is calibrated according to the design three-dimensional model obtained after the alignment to obtain the defect three-dimensional model of the workpiece to be inspected.

[0103] Specifically, the superposition model can be calibrated according to the design three-dimensional model obtained after the registration in the following manner: for each design component in the superposition model, the position and posture of the design component in the superposition model is calibrated according to the position and posture information of the design component in the design three-dimensional model obtained after the registration.

[0104] In the present disclosure, the position and posture of the defective component in the superimposed model is adjusted along with the position and posture of the design component that is closest to it. The defective three-dimensional model includes the defective component and the design component, wherein the position and posture of the design component in the defective three-dimensional model can be adjusted according to the position and posture of the design component in the design three-dimensional model obtained after registration, so that the position and posture of the design component in the defective three-dimensional model and the design three-dimensional model obtained after registration are as consistent as possible.

[0105] Among them, when there is one design component adjacent to the defective component in the superposition model, the one design component can be determined as the design component closest to the defective component; when there are multiple design components adjacent to the defective component in the superposition model, the design component whose center of mass is closest to the center of mass of the defective component among the multiple design components adjacent to the defective component can be determined as the design component closest to the defective component.

[0106] Figure 3 FIG. 1 is a block diagram of a defect location device according to an exemplary embodiment. Figure 3 , the device 300 comprises:

[0107] A first acquisition module 301 is configured to acquire a plurality of X-ray images of a workpiece to be inspected at different angles and shooting posture information of each of the X-ray images;

[0108] The defect detection module 302 is configured to perform defect detection on each X-ray image to obtain a defect image corresponding to the X-ray image;

[0109] A first model building module 303 is configured to build a first reconstruction model of the workpiece to be inspected according to each defect image and shooting posture information of the X-ray image corresponding to each defect image;

[0110] A second model building module 304 is configured to build a second reconstructed model of the workpiece to be inspected according to the designed three-dimensional model of the workpiece to be inspected, the first reconstructed model and the shooting posture information corresponding to each of the defect images;

[0111] The generating module 305 is configured to generate a defect three-dimensional model of the workpiece to be inspected according to the first reconstructed model and the second reconstructed model.

[0112] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: first, defect detection is performed on multiple X-ray images of the workpiece to be detected at different angles to obtain defect images corresponding to each X-ray image; then, a first reconstruction model of the workpiece to be detected is constructed according to each defect image and the shooting posture information of the X-ray image corresponding to each defect image; then, a second reconstruction model of the workpiece to be detected is constructed according to the design three-dimensional model of the workpiece to be detected, the first reconstruction model and the shooting posture information corresponding to each defect image; finally, a defect stereo model of the workpiece to be detected is generated according to the first reconstruction model and the second reconstruction model. In this way, computer vision detection (i.e., defect detection using computer vision) and image mapping matching can be combined to achieve stereo restoration of multiple two-dimensional X-ray images of the workpiece to be detected at different angles. The defect stereo model obtained by stereo restoration can not only intuitively display the location of the defect, but also reflect the position correlation of the defect in three-dimensional space. In addition, the three-dimensional defect morphology of the defect stereo model is helpful for defect cause analysis and improves production efficiency and production yield. In addition, in theory, each defective component is mapped in the defect images at different angles, but the falsely detected defective components usually exist in individual defect images. Therefore, by generating a defect 3D model based on multiple defect images at different angles, some of the falsely detected components can be filtered out based on multi-image voting, thereby improving the accuracy of defect positioning.

[0113] Optionally, the second model building module 304 includes:

[0114] A first determination submodule is configured to determine the design component contained in each defect image according to the design three-dimensional model of the workpiece to be inspected, the first reconstructed model and the shooting posture information corresponding to each defect image;

[0115] A first construction submodule is configured to construct a second reconstruction model of the workpiece to be inspected through a pre-constructed modeling model according to the design components contained in each defect image and the shooting posture information corresponding to each defect image;

[0116] Optionally, the first determining submodule includes:

[0117] A first registration submodule is configured to register the designed three-dimensional model of the workpiece to be inspected with the first reconstructed model;

[0118] The second determination submodule is configured to determine, for each defect image, the design component included in the defect image according to the design three-dimensional model obtained after registration and the shooting posture information corresponding to the defect image.

[0119] Optionally, the second determining submodule includes:

[0120] A third determination submodule is configured to determine a ray irradiation angle when shooting an X-ray image corresponding to the defect image according to the shooting posture information corresponding to the defect image;

[0121] A projection submodule is configured to project the designed three-dimensional model obtained after registration according to the ray irradiation angle to obtain a projection image;

[0122] The fourth determining submodule is configured to determine the design component in the projection image as the design component included in the defective image.

[0123] Optionally, the first building submodule includes:

[0124] The annotation submodule is configured to annotate each defect image with design components according to the design components contained in the defect image, so as to obtain a target image corresponding to the defect image;

[0125] The second construction submodule is configured to input each of the target images and the shooting posture information corresponding to each of the target images into a pre-constructed modeling model to obtain a second reconstructed model of the workpiece to be inspected, wherein the shooting posture information corresponding to the target image is the shooting posture information corresponding to the defect image corresponding to the target image.

[0126] Optionally, the modeling model is obtained by training through a model training device, wherein the model training device comprises:

[0127] A second acquisition module is configured to acquire a plurality of reference images of a reference workpiece, shooting posture information corresponding to each of the reference images, and a reference stereo model of the reference workpiece, wherein the plurality of reference images include a plurality of reference X-ray images at different angles and annotated with components;

[0128] The training module is configured to perform model training by taking the multiple reference images and the shooting posture information corresponding to each of the reference images as the input of the modeling model and taking the reference stereo model as the target output of the modeling model to obtain the modeling model.

[0129] Optionally, the generating module 305 includes:

[0130] The superposition submodule is configured to superimpose the outer edge contour of the first reconstructed model with the internal structure of the second reconstructed model to obtain a three-dimensional defect model of the workpiece to be inspected.

[0131] Optionally, the generating module 305 includes:

[0132] a superposition submodule, configured to superimpose the outer edge contour of the first reconstructed model with the internal structure of the second reconstructed model to obtain a superimposed model;

[0133] A second registration submodule is configured to register the designed three-dimensional model of the workpiece to be inspected with the first reconstructed model;

[0134] The calibration submodule is configured to calibrate the superimposed model according to the designed three-dimensional model obtained after the registration, so as to obtain the defect three-dimensional model of the workpiece to be inspected.

[0135] Optionally, the calibration submodule is configured to calibrate the pose of each design component in the superposition model according to the pose information of the design component in the design three-dimensional model obtained after alignment, wherein the pose of the defective component in the superposition model is adjusted as the pose of the design component most adjacent to it is adjusted.

[0136] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0137] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the above-mentioned defect location method provided by the present disclosure are implemented.

[0138] Figure 48 is a block diagram of an apparatus 800 for defect location according to an exemplary embodiment. For example, the apparatus 800 may be a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0139] Reference Figure 4 , the device 800 may include one or more of the following components: a first processing component 802 , a first memory 804 , a first power supply component 806 , a multimedia component 808 , an audio component 810 , a first input / output interface 812 , a sensor component 814 , and a communication component 816 .

[0140] The first processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The first processing component 802 may include one or more first processors 820 to execute instructions to complete all or part of the steps of the defect location method described above. In addition, the first processing component 802 may include one or more modules to facilitate the interaction between the first processing component 802 and other components. For example, the first processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the first processing component 802.

[0141] The first memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The first memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0142] The first power source component 806 provides power to various components of the device 800. The first power source component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 800.

[0143] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0144] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the first memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0145] The first input / output interface 812 provides an interface between the first processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0146] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0147] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0148] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-mentioned defect location method.

[0149] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a first memory 804 including instructions, and the instructions can be executed by a first processor 820 of the apparatus 800 to complete the above-mentioned defect localization method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0150] In addition to being an independent electronic device, the above-mentioned device can also be a part of an independent electronic device. For example, in one embodiment, the device can be an integrated circuit (IC) or a chip, wherein the integrated circuit can be an IC or a collection of multiple ICs; the chip can include but is not limited to the following types: GPU (Graphics Processing Unit), CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), SOC (System on Chip, SoC), etc. The above-mentioned integrated circuit or chip can be used to execute executable instructions (or codes) to implement the above-mentioned defect location method. The executable instructions can be stored in the integrated circuit or chip, or can be obtained from other devices or equipment, such as the integrated circuit or chip includes a processor, a memory, and an interface for communicating with other devices. The executable instruction may be stored in the first memory, and when the executable instruction is executed by the processor, the above-mentioned defect location method may be implemented; alternatively, the integrated circuit or chip may receive the executable instruction through the interface and transmit it to the processor for execution, so as to implement the above-mentioned defect location method.

[0151] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device. The computer program has a code portion for executing the above-mentioned defect localization method when executed by the programmable device.

[0152] Figure 5 1 is a block diagram of a device 1900 for defect location according to an exemplary embodiment. For example, the device 1900 may be provided as a server. Figure 5 The apparatus 1900 includes a second processing component 1922, which further includes one or more second processors, and a memory resource represented by a second memory 1932 for storing instructions executable by the second processing component 1922, such as an application. The application stored in the second memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the second processing component 1922 is configured to execute instructions to perform the above-mentioned defect localization method.

[0153] The device 1900 may further include a second power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and a second input / output interface 1958. The device 1900 may operate based on an operating system stored in the second memory 1932, such as Windows Server 2000. TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or similar.

[0154] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0155] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A defect location method, characterized in that: include: Acquire multiple X-ray images of the workpiece to be inspected at different angles and shooting posture information of each of the X-ray images; For each of the X-ray images, performing defect detection on the X-ray image to obtain a defect image corresponding to the X-ray image; Constructing a first reconstruction model of the workpiece to be inspected according to each of the defect images and the shooting posture information of the X-ray image corresponding to each of the defect images; Constructing a second reconstructed model of the workpiece to be inspected according to the designed three-dimensional model of the workpiece to be inspected, the first reconstructed model, and the shooting posture information corresponding to each of the defect images; A defect three-dimensional model of the workpiece to be inspected is generated according to the first reconstruction model and the second reconstruction model.

2. The method according to claim 1, characterized in that: The step of constructing a second reconstructed model of the workpiece to be inspected according to the designed three-dimensional model of the workpiece to be inspected, the first reconstructed model, and the shooting posture information corresponding to each defect image comprises: Determine the design component contained in each defect image according to the design three-dimensional model of the workpiece to be inspected, the first reconstructed model, and the shooting posture information corresponding to each defect image; According to the design components contained in each defect image and the shooting posture information corresponding to each defect image, a second reconstruction model of the workpiece to be inspected is constructed through a pre-constructed modeling model.

3. The method according to claim 2, characterized in that The step of determining the design component contained in each defect image according to the design three-dimensional model of the workpiece to be inspected, the first reconstructed model, and the shooting posture information corresponding to each defect image comprises: Registering the designed three-dimensional model of the workpiece to be inspected and the first reconstructed model; For each defect image, the design component included in the defect image is determined according to the design three-dimensional model obtained after registration and the shooting posture information corresponding to the defect image.

4. The method according to claim 3, characterized in that The step of determining the design component included in the defect image according to the design three-dimensional model obtained after registration and the shooting posture information corresponding to the defect image comprises: Determining, according to the shooting posture information corresponding to the defect image, a ray irradiation angle when shooting an X-ray image corresponding to the defect image; Projecting the designed three-dimensional model obtained after registration according to the ray irradiation angle to obtain a projection image; The design component in the projection image is determined as the design component included in the defective image.

5. The method according to claim 2, characterized in that: The step of constructing a second reconstruction model of the workpiece to be inspected based on the design components contained in each defect image and the shooting posture information corresponding to each defect image by using a pre-constructed modeling model includes: For each of the defect images, according to the design components contained in the defect image, the defect image is annotated with design components to obtain a target image corresponding to the defect image; Input each of the target images and the shooting posture information corresponding to each of the target images into a pre-built modeling model to obtain a second reconstructed model of the workpiece to be inspected, wherein the shooting posture information corresponding to the target image is the shooting posture information corresponding to the defect image corresponding to the target image.

6. The method according to claim 2, characterized in that The modeling model is trained in the following way: Acquire multiple reference images of a reference workpiece, shooting posture information corresponding to each of the reference images, and a reference stereo model of the reference workpiece, wherein the multiple reference images include multiple reference X-ray images at different angles with components annotated thereon; The model is trained by taking the multiple reference images and the shooting posture information corresponding to each of the reference images as the input of the model and taking the reference stereo model as the target output of the model to obtain the model.

7. The method according to any one of claims 1 to 6, characterized in that The step of generating a defect three-dimensional model of the workpiece to be inspected according to the first reconstruction model and the second reconstruction model comprises: The outer edge contour of the first reconstructed model is superimposed with the internal structure of the second reconstructed model to obtain a three-dimensional defect model of the workpiece to be inspected.

8. The method according to any one of claims 1 to 6, characterized in that The step of generating a defect three-dimensional model of the workpiece to be inspected according to the first reconstruction model and the second reconstruction model comprises: Superimposing the outer edge contour of the first reconstructed model with the internal structure of the second reconstructed model to obtain a superimposed model; Registering the designed three-dimensional model of the workpiece to be inspected and the first reconstructed model; The superimposed model is calibrated according to the designed three-dimensional model obtained after the registration to obtain a three-dimensional defect model of the workpiece to be inspected.

9. The method according to claim 8, characterized in that The step of calibrating the superimposed model according to the designed three-dimensional model obtained after the registration includes: For each designed component in the superposition model, the position and posture of the designed component in the superposition model is calibrated according to the position and posture information of the designed component in the designed three-dimensional model obtained after alignment, wherein the position and posture of the defective component in the superposition model is adjusted as the position and posture of the designed component closest to it is adjusted.

10. A defect locating device, characterized in that: include: A first acquisition module is configured to acquire a plurality of X-ray images of the workpiece to be inspected at different angles and shooting posture information of each of the X-ray images; A defect detection module is configured to perform defect detection on each X-ray image to obtain a defect image corresponding to the X-ray image; A first model building module is configured to build a first reconstruction model of the workpiece to be inspected according to each of the defect images and the shooting posture information of the X-ray image corresponding to each of the defect images; A second model building module is configured to build a second reconstructed model of the workpiece to be inspected according to the designed three-dimensional model of the workpiece to be inspected, the first reconstructed model and the shooting posture information corresponding to each of the defect images; A generating module is configured to generate a defect three-dimensional model of the workpiece to be inspected according to the first reconstructed model and the second reconstructed model.

11. A defect locating device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Acquire multiple X-ray images of the workpiece to be inspected at different angles and shooting posture information of each of the X-ray images; For each of the X-ray images, performing defect detection on the X-ray image to obtain a defect image corresponding to the X-ray image; Constructing a first reconstruction model of the workpiece to be inspected according to each of the defect images and the shooting posture information of the X-ray image corresponding to each of the defect images; Constructing a second reconstructed model of the workpiece to be inspected according to the designed three-dimensional model of the workpiece to be inspected, the first reconstructed model, and the shooting posture information corresponding to each of the defect images; A defect three-dimensional model of the workpiece to be inspected is generated according to the first reconstruction model and the second reconstruction model.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.

13. A chip, characterized in that: The method comprises a processor and an interface; the processor is used to read instructions to execute the method according to any one of claims 1 to 9.

Images