Industrial defect recognition method, system, computing device, and storage medium

Abstract

The embodiment of the application discloses an industrial defect recognition method, system, computing device and storage medium, and belongs to the AI field. In the embodiment of the application, a target region is first extracted from a to-be-recognized image, and then a defect rough selection region containing an industrial defect is obtained from the target region. Through twice region extraction, the size proportion of the industrial defect relative to the background is improved, and the detection probability of small-size industrial defects can be improved. After the defect rough selection region containing the industrial defect is detected, the industrial defect in the defect rough selection region is further positioned and sized, which can effectively improve the positioning and sizing accuracy of the defect. In addition, in the embodiment of the application, not only the defect type is recognized, but also the industrial defect can be positioned and sized, which enriches the recognition dimension of the industrial defect.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to an industrial defect identification method, system, and storage medium. Background Technology

[0002] In the industrial sector, industrial products may develop defects during production, manufacturing, and maintenance, such as weld defects. To ensure the proper functioning of industrial products, defect detection is essential. Currently, image-based detection of industrial products is one of the important methods for industrial defect detection. Furthermore, with the widespread application of neural network models in image recognition in recent years, some methods for identifying industrial defects in images using neural network models have emerged. However, current methods for industrial defect identification using neural network models are often only suitable for identifying large-sized industrial defects, with limited ability to identify small-sized defects. Moreover, the identification results often only include the type of industrial defect, resulting in a single identification dimension. Summary of the Invention

[0003] This application provides an industrial defect identification method, system, computing device, and storage medium, which can improve the detection probability of small-sized industrial defects, improve the accuracy of defect location and size estimation, and enrich the dimensions of industrial defect identification. The technical solution is as follows:

[0004] In a first aspect, an industrial defect identification method is provided, the method comprising: acquiring an image to be identified, the image to be identified being an image reflecting an industrial product; extracting at least one target region from the image to be identified according to a target region detection model, the at least one target region being a region where industrial defects may exist; obtaining a preliminary defect region containing industrial defects in the target region, and the type of industrial defect contained in the preliminary defect region, based on the at least one target region and the defect detection model; and determining the size and location of the industrial defects contained in the preliminary defect region.

[0005] In this embodiment, the target region is first extracted from the image to be identified, and then a coarse defect selection region containing industrial defects is obtained from the target region. By extracting regions twice, the size ratio of industrial defects relative to the background is increased, thereby improving the detection probability of small-sized industrial defects. After detecting the coarse defect selection region containing industrial defects, the industrial defects within the coarse defect selection region are further located and their size estimated, effectively improving the accuracy of defect location and size estimation. Furthermore, this embodiment not only identifies the defect type but also locates and estimates the size of industrial defects, enriching the dimensions of industrial defect identification.

[0006] Optionally, the process of obtaining a preliminary defect region containing industrial defects in the target region and the type of industrial defects contained in the preliminary defect region based on the at least one target region and the defect detection model is as follows: taking the at least one target region as input to the defect detection model, performing inference through the defect detection model to obtain the position of the preliminary defect region containing industrial defects in the target region and the defect type of the industrial defects contained in the preliminary defect region, wherein the defect detection model is an artificial intelligence (AI) model trained using defect training samples; and extracting the preliminary defect region from the target region based on the position of the preliminary defect region in the target region.

[0007] At least one target region is a region where industrial defects may occur. Thus, any given target region may or may not contain industrial defects. Based on this, after extracting the target regions, a defect detection model can be used to further detect whether the target regions contain industrial defects. If they do contain industrial defects, the defect detection model can output a preliminary selection of defect regions containing the industrial defect and the type of the industrial defect.

[0008] In this embodiment, the target region extracted by the target region detection model is a relatively large area where a certain industrial defect may occur. The defect coarse selection region obtained by the defect detection model is a smaller region containing industrial defects, extracted from the target region after further narrowing down the range. Thus, the proportion of industrial defects in the target region will be greater than their proportion in the image to be identified, and the proportion in the defect coarse selection region will be greater than their proportion in the target region. In other words, by extracting the target region from the image to be identified and then extracting the defect coarse selection region from the target region, the proportion of industrial defects relative to the background is gradually increased, thereby improving the detection probability of small-sized industrial defects.

[0009] Optionally, the process of determining the size and location of the industrial defects contained in the coarse defect selection area can be as follows: based on the grayscale distribution within the coarse defect selection area, determine the location and size of the industrial defects contained in the coarse defect selection area within the coarse defect selection area; based on the location of the coarse defect selection area in the target area, the location of the target area in the image to be identified, and the location of the industrial defects contained in the coarse defect selection area within the coarse defect selection area, determine the location of the industrial defects contained in the coarse defect selection area in the image to be identified.

[0010] In this embodiment of the application, after detecting a rough selection area containing industrial defects, the industrial defects in the rough selection area are further located and their size estimated based on the grayscale distribution within the rough selection area, which can effectively improve the accuracy of defect location and size estimation.

[0011] Optionally, after locating and estimating the size of industrial defects within the coarsely selected defect area, the method further includes: generating an inspection report, the inspection report including the defect type, size, and location of the industrial defects detected in the image to be identified; and providing the inspection report to a user.

[0012] Optionally, the industrial defect is a weld defect, and the target area is the area where the weld is located.

[0013] Optionally, the method further includes: acquiring multiple industrial training samples, each industrial training sample corresponding to target region annotation information, the target region annotation information including the position information of the multi-degree-of-freedom quadrilaterals annotated in the corresponding industrial training sample and the annotation category information used to indicate that the corresponding multi-degree-of-freedom quadrilateral is a target region; training the target region detection network using the multiple industrial training samples and the target region annotation information of each industrial training sample to obtain the target region detection model.

[0014] In this embodiment of the application, a multi-degree-of-freedom quadrilateral is used to label the target regions in the industrial training samples that may contain industrial defects. Then, the target region detection network is trained using the industrial training samples to obtain a target region detection model. In this way, the target region detection model can more accurately detect various target regions in the image that may contain defects and are distributed in different directions and shapes.

[0015] Optionally, the process of training the target region detection network using the multiple industrial training samples and the target region annotation information of each industrial training sample can be as follows: Based on the target region annotation information of the first industrial training sample, determine a first bounding rectangle and a second bounding rectangle, where the first bounding rectangle is the minimum rotational bounding rectangle of the multi-degree-of-freedom quadrilateral annotated in the first industrial training sample, and the second bounding rectangle is the horizontal bounding rectangle of the first bounding rectangle; determine the predicted values ​​of the multi-degree-of-freedom quadrilaterals in the first industrial training sample, the predicted values ​​of the first bounding rectangle, the predicted values ​​of the second bounding rectangle, and the predicted category information using the target region detection network; calculate a loss function value based on the target region annotation information of the first industrial training sample, the first bounding rectangle, the second bounding rectangle, the predicted values ​​of the multi-degree-of-freedom quadrilaterals in the first industrial training sample, the predicted values ​​of the first bounding rectangle, the predicted values ​​of the second bounding rectangle, and the predicted category information; and update the parameters of the target region detection network based on the loss function value.

[0016] Optionally, the method further includes: acquiring multiple defect training samples; extracting multiple target regions from the multiple defect training samples using the target region detection model; acquiring defect annotation information of the target regions containing industrial defects, wherein the defect annotation information is used to indicate the defect type and location of the industrial defects in the corresponding target regions; and training the defect detection network using the target regions containing industrial defects and the defect annotation information of the corresponding target regions to obtain the defect detection model.

[0017] In this embodiment, a target region detection model is first trained. Then, this model is used to coarsely locate industrial defects in the defect training samples to extract the target region. Subsequently, a defect detection network is trained using the fine-grained locations and types of defects marked within this target region to obtain the defect detection model. This allows for more accurate localization of industrial defects within the target region when the defect detection model is used to detect the extracted target region from the image to be identified. It also enables the output of the specific defect type, improving both the accuracy of localization and the dimensions of defect identification.

[0018] Optionally, the target region detection model uses a multi-degree-of-freedom quadrilateral as the detection box, and the angle between each side of the multi-degree-of-freedom quadrilateral and each side of the image to be identified is any value between 0 degrees and 180 degrees.

[0019] In this embodiment of the application, a neural network detection model with a multi-degree-of-freedom quadrilateral as the detection box is used in the target region extraction stage. This model is suitable for different types of target regions and has good versatility.

[0020] Secondly, an industrial defect identification system is provided, which has the function of implementing the industrial defect identification method of the first aspect described above. The industrial defect identification system includes at least one module for implementing the industrial defect identification system method provided in the first aspect.

[0021] Thirdly, a computing device is provided, comprising a processor and a memory. The memory stores programs that support the computing device in executing the industrial defect identification method provided in the first aspect, and stores data related to implementing the industrial defect identification method provided in the first aspect. The processor is configured to execute the programs stored in the memory. The computing device may further include a communication bus for establishing a connection between the processor and the memory.

[0022] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the industrial defect identification method described in the first aspect.

[0023] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute the industrial defect identification method described in the first aspect above.

[0024] The technical effects achieved by the second, third, fourth, and fifth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here.

[0025] The beneficial effects of the technical solutions provided in this application include at least the following three aspects:

[0026] First, in this embodiment, the target region is extracted from the image to be identified, and then a coarse selection region containing industrial defects is obtained from the target region. By extracting regions twice, the size ratio of industrial defects relative to the background is increased, which can improve the detection probability of small-sized industrial defects.

[0027] Secondly, after detecting the defect coarse selection area containing industrial defects, further locating and estimating the size of the industrial defects in the defect coarse selection area can effectively improve the accuracy of defect location and size estimation.

[0028] Third, in this application embodiment, not only are defect types identified, but industrial defects can also be located and their size estimated, enriching the identification dimensions of industrial defects. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the structure of an industrial defect identification system provided in an embodiment of this application;

[0030] Figure 2 This is a deployment diagram of an industrial defect identification system provided in an embodiment of this application;

[0031] Figure 3 This is a schematic diagram illustrating the application of an industrial defect identification system provided in an embodiment of this application;

[0032] Figure 4 This is a deployment diagram of another industrial defect identification system provided in an embodiment of this application;

[0033] Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0034] Figure 6 This is a flowchart of a training target region detection model and a defect detection model provided in an embodiment of this application;

[0035] Figure 7 This is a schematic diagram of a multi-degree-of-freedom quadrilateral obtained by annotation in an industrial training sample, provided in an embodiment of this application;

[0036] Figure 8 This is a schematic diagram of a multi-degree-of-freedom quadrilateral labeled in an industrial training sample, and the generated first and second bounding rectangles provided in an embodiment of this application;

[0037] Figure 9 This is a flowchart of an industrial defect identification method provided in an embodiment of this application;

[0038] Figure 10 This is a schematic diagram of the structure of a computer system provided in an embodiment of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0040] Before providing a detailed explanation of the embodiments of this application, the application scenarios of the embodiments of this application will be introduced first.

[0041] In the industrial sector, industrial products may develop defects during production, manufacturing, and maintenance. For example, welding is frequently used in the manufacturing or maintenance of industrial products such as oil and gas pipelines, spacecraft, and automobiles, and weld defects may exist at the weld seams. Similarly, a component in an automobile may have cracks, scratches, or other defects due to manufacturing or maintenance issues. Such defects in industrial products often affect their normal use; therefore, the detection of industrial defects is essential. Currently, image-based detection of industrial products is one of the important methods for industrial defect detection. One major method relies on manual identification of industrial defects from images. However, this method is not only detrimental to the health of workers but also makes it difficult to guarantee the quality of identification. In recent years, with the widespread application of neural network models in image recognition, several neural network-based industrial defect identification methods have emerged to replace manual identification. However, current industrial defect identification methods based on neural network models still have some problems, mainly in two aspects. First, these methods are often only suitable for identifying large-sized defects, and their effect on identifying small-sized defects is poor. Second, these methods can often only identify whether industrial defects exist in an image, but cannot locate or estimate the size of the defects, resulting in a relatively limited identification dimension. Therefore, this application provides a method for identifying industrial defects in industrial product images based on an AI model, to enrich the identification dimensions of industrial defects and improve the identification effect of small-sized industrial defects.

[0042] It should be noted that the industrial defect identification method provided in this application embodiment is executed by the industrial defect identification system 01. This industrial defect identification system 01 can be implemented by a software system, a hardware device, or a combination of both.

[0043] When the industrial defect identification system 01 is a software system, see [link to relevant documentation]. Figure 1 The industrial defect identification system 01 can be logically divided into an identification device 10 and a training device 11. Both the identification device 10 and the training device 11 can be logically divided into multiple modules, each with different functions. For example, see... Figure 1 The identification device 10 includes a target area detection module 101, a coarse defect detection module 102, and a fine defect identification module 103. It should be noted that the embodiments in this application only provide an exemplary division of the structure and functional modules of the identification device 10, but do not limit its specific division in any way.

[0044] The target region detection module 101 is used to acquire an image to be identified and extract at least one target region from the image to be identified according to a target region detection model. The image to be identified is an image containing industrial products. At least one target region refers to an area where industrial defects may exist. For example, in the detection of weld defects, the target region can be an area in the image to be identified containing a complete weld. The target region can be a polygonal region. For example, the target region is a multi-degree-of-freedom quadrilateral. A multi-degree-of-freedom quadrilateral means that the angles or parallel relationships between the sides of the quadrilateral and the sides of the image to be identified are free, not limited to perpendicular or parallel relationships, and the angles or parallel relationships between the four sides of the quadrilateral are free, not limited to perpendicular or parallel relationships. That is, the angles between the sides of the multi-degree-of-freedom quadrilateral and the sides of the image to be identified can be any value from 0 degrees to 180 degrees. Furthermore, adjacent sides of the multi-degree-of-freedom quadrilateral region can be perpendicular or not perpendicular, and opposite sides can be parallel or not parallel.

[0045] It should be noted that the target region detection model is an AI model trained using industrial training samples. The target region detection model is used to identify regions of interest (i.e., target regions) in an image to be identified that may contain certain industrial defects, and outputs the location of the target region in the image.

[0046] The coarse defect detection module 102 communicates with the target area detection module 101 and the fine defect identification module 103. It receives at least one target area extracted by the target area detection module 101 and, based on the at least one target area and the defect detection model, obtains a coarsely selected defect region containing industrial defects within the target area and the type of industrial defect contained in the coarsely selected defect region. The defect detection model is an AI model trained using defect training samples. Furthermore, this defect detection model is used to further detect the coarsely selected defect region containing industrial defects from the target area and outputs the location information of the coarsely selected defect region within the target area. For example, in the detection of weld defects, the coarsely selected defect region may refer to a smaller region containing weld defects extracted from a region containing a complete weld. In addition, the defect detection model is also used to output the type of industrial defect contained within the coarsely selected defect region. Of course, it should be noted that since at least one target area extracted by the aforementioned target area detection module 101 is an area where industrial defects may occur, that is, some target areas may not contain industrial defects. In this case, after the defect detection model detects these target areas, it will output an identification result indicating that there are no defects in the target area.

[0047] The fine defect identification module 103 is used to communicate with the coarse defect detection module 102, receive the coarse defect selection area extracted by the coarse defect detection module 102, and determine the size and location of the industrial defects contained in the coarse defect selection area.

[0048] In some possible implementations, the fine defect identification module 103 is also used to receive the location information of the coarse defect detection region in the target region sent by the coarse defect detection module 102, and to communicate with the target region detection module 101 to receive the location information of the target region in the image to be identified sent by the target region detection module 101. Then, the fine defect identification module 103 determines the size of the industrial defect contained in the coarse defect detection region and the location of the industrial defect in the coarse defect detection region based on the grayscale distribution in the coarse defect detection region. Based on the location information of the coarse defect detection region in the target region, the location information of the target region containing the coarse defect detection region in the image to be identified, and the location of the industrial defect in the coarse defect detection region, the location of the industrial defect in the image to be identified is obtained.

[0049] Alternatively, in some possible cases, see Figure 1 The identification device 10 may further include a report generation module 104, which communicates with the defect fine identification module 103, receives the identification results sent by the defect fine identification module 103, and then generates an inspection report based on the identification results, providing the inspection report to the user. The identification results include the type, size, and location of industrial defects detected in the image to be identified. Accordingly, the inspection report includes the aforementioned identification results. In addition, the inspection report may also include other content, such as information on whether the industrial product in the image to be identified is qualified, as determined by the identification results. This embodiment of the application does not limit this aspect.

[0050] Optionally, the report generation module 104 is also used to communicate with the defect coarse detection module 102, receive the identification results output by the defect coarse detection module 102 indicating that there are no defects in certain target areas, and then generate an inspection report based on the identification results.

[0051] As described above, both the target area detection model and the defect detection model are pre-trained AI models. Before these two models are used for industrial defect detection, the training device 11 can train the target area detection network to obtain the target area detection model, and train the defect detection network to obtain the defect detection model. See [link to documentation]. Figure 1 The training device 11 includes a target region detection model generation module 111, a target region extraction module 112, a defect detection model generation module 113, and a storage module 114.

[0052] The target region detection model generation module 111 is used to acquire multiple industrial training samples from the storage module 114 and train the target region detection network using these samples to obtain a target region detection model. These multiple industrial training samples are images that all contain target regions, which are areas where industrial defects may occur. Furthermore, each industrial training sample corresponds to target region annotation information, which includes the positional information of the multi-degree-of-freedom quadrilaterals annotated in the corresponding industrial training sample and annotation category information indicating that the corresponding multi-degree-of-freedom quadrilateral is a target region.

[0053] The target region extraction and defect annotation module 112 is used to obtain multiple defect training samples from the storage module 114 and extract the target region from each defect training sample using a trained target region detection model. Each of the multiple defect training samples contains a target region. After extracting the target region from each defect training sample, the target region extraction and defect annotation module 112 can also obtain defect annotation information obtained by the user annotating industrial defects in the target region. This defect annotation information indicates the defect type and location of the industrial defects within the corresponding target region.

[0054] The defect detection model generation module 113 is used to communicate with the target region extraction and defect labeling module 112, receive multiple target regions and defect labeling information of each target region sent by the target region extraction and defect labeling module 112, and then train the defect detection network through the multiple target regions and defect labeling information of each target region to obtain the defect detection model.

[0055] In this embodiment, the training device 11 and the recognition device 10 are two independent devices. In this case, the target area detection model and the defect detection model trained by the training device 11 will be deployed to the recognition device 10. Optionally, the training device 11 can also be part of the recognition device 10.

[0056] The industrial defect identification system 01 described above can be deployed flexibly. For example, this industrial defect identification system 01 can be deployed in a cloud environment. A cloud environment is an entity that provides cloud services to users using basic resources under the cloud computing model. A cloud environment includes cloud data centers and cloud service platforms.

[0057] A cloud data center encompasses a vast amount of basic resources (including computing, storage, and network resources) owned by a cloud service provider. The computing resources within a cloud data center can be a large number of computing devices (e.g., servers). The industrial defect identification system 01 can be a software system deployed on servers or virtual machines within a cloud data center. This software system can be used to identify industrial defects and can be deployed in a distributed manner across multiple servers, multiple virtual machines, or a combination of virtual machines and servers. For example, as... Figure 2 As shown, the industrial defect identification system 01 is deployed in a cloud environment. Client 02 can send an image to be identified to the industrial defect identification system 01. After receiving the image, the industrial defect identification system 01 can extract at least one target region from the image based on a target region detection model. Based on the at least one target region and the defect detection model, it obtains a preliminary defect region containing industrial defects within the at least one target region, and the type of industrial defect contained in the preliminary defect region. Then, it determines the size and location of the industrial defects contained in the preliminary defect region. Finally, the industrial defect identification system 01 feeds back the detected industrial defect type, location, and size to client 02.

[0058] For example, Figure 3 This is a schematic diagram illustrating one application of the industrial defect identification system 01 in this application, such as... Figure 3 As shown, the industrial defect identification system 01 can be deployed by a cloud service provider in a cloud data center. The cloud service provider abstracts the functions provided by the industrial defect identification system 01 into a cloud service, and the cloud service platform allows users to consult and purchase this cloud service. After purchasing this cloud service, users can use the industrial defect identification service provided by the industrial defect identification system 01 in the cloud data center. The industrial defect identification system 01 can also be deployed by tenants on the computing resources of the cloud data center rented by the tenants. Tenants purchase computing resource cloud services provided by the cloud service provider through the cloud service platform, and run the industrial defect identification system 01 on the purchased computing resources, enabling the industrial defect identification system 01 to identify industrial defects.

[0059] Optionally, the industrial defect identification system 01 can also be a software system running on edge computing devices in an edge environment, or it can be composed of one or more edge computing devices in an edge environment. An edge environment refers to a collection of devices in an application scenario, including one or more edge computing devices. These one or more edge computing devices can be computing devices within a data center or computing devices in multiple data centers. When the industrial defect identification system 01 is a software system, it can be deployed distributed across multiple edge computing devices or centrally deployed on a single edge computing device. For example, as... Figure 4 As shown, the industrial defect identification system 01 is deployed in a distributed manner in the edge computing device 03 of a company's data center. The client 04 in the company can send the image to be identified to the industrial defect identification system 01. After receiving the image to be identified, the industrial defect identification system 01 can identify the industrial defects contained in the image to be identified through the method provided in the embodiments of this application, and feed back the identification result to the client 04.

[0060] When the industrial defect identification system 01 is a hardware device, it can be a computing device in any environment, such as the edge computing device described above or the computing device in the cloud environment described above. Figure 5 This is a schematic diagram of the structure of a computing device 500 provided in an embodiment of this application. The computing device 500 includes a processor 501, a communication bus 502, a memory 503, and at least one communication interface 504.

[0061] Processor 501 can be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), or any combination thereof. Processor 501 may include one or more chips, and processor 501 may include an AI accelerator, such as a neural processing unit (NPU).

[0062] The communication bus 502 may include a path for transmitting information between various components of the computing device 500 (e.g., processor 501, memory 503, communication interface 504).

[0063] Memory 503 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 503 may exist independently and be connected to processor 501 via communication bus 502. Memory 503 may also be integrated with processor 501. The memory 503 can store computer instructions. When the computer instructions stored in the memory 503 are executed by the processor 501, the industrial defect identification method of this application can be implemented. In addition, the memory 503 can also store data required by the processor during the execution of the above method, as well as intermediate data and / or result data generated.

[0064] Communication interface 504 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Networks (WLAN), etc.

[0065] In a specific implementation, as one example, processor 501 may include one or more CPUs.

[0066] In a specific implementation, as one example, a computer device may include multiple processors. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0067] In this embodiment, the industrial defect recognition system employs a target region detection model and a defect detection model to identify industrial defects in the image to be recognized. Both the target region detection model and the defect detection model are pre-trained AI models. Therefore, the training process of the target region detection model and the defect detection model will be described below.

[0068] Figure 6 This is a flowchart illustrating a training target region detection model and a defect detection model provided in an embodiment of this application. This method can be applied to the training device in the aforementioned industrial defect identification system. See [link to relevant documentation]. Figure 6 The method includes the following steps:

[0069] Step 601: Obtain multiple industrial training samples, each of which has corresponding target region annotation information.

[0070] In this embodiment, the training device collects multiple industrial training samples. These industrial training samples can be images captured and transmitted to the training device by other acquisition devices, such as cameras in an industrial environment. Alternatively, the industrial training samples can be collected by an acquisition device, stored by another device, and then transmitted to the training device. Or, the industrial training samples can be images sent by a client to an industrial defect identification system for identification. That is, the training device can collect images identified by the identification device in the industrial defect identification system and use these images as industrial training samples.

[0071] It should be noted that all of these industrial training samples are images containing target regions. These target regions refer to areas where industrial defects may exist. These industrial defects are defects that arise during the production, manufacturing, or maintenance of industrial products. For example, if the industrial defect is a weld defect, since weld defects only occur in the area where the weld is located, then the target region is the weld area. As another example, if a certain industrial defect only occurs on a specific part of a car, then the target region is the area where that part is located. This will not be elaborated further in the embodiments of this application.

[0072] After acquiring multiple industrial training samples, the training device displays each sample to the user sequentially, allowing the user to annotate the target region for each sample. The training device then acquires the target region annotation information generated by the user during the annotation process for each industrial training sample.

[0073] In this process, the user uses a quadrilateral bounding box to annotate the target region in the industrial training sample, thereby obtaining a quadrilateral containing the target region.

[0074] It should be noted that this quadrilateral annotation box is a multi-degree-of-freedom quadrilateral annotation box. Correspondingly, the annotated quadrilateral is also a multi-degree-of-freedom quadrilateral. A multi-degree-of-freedom quadrilateral refers to a quadrilateral whose four sides have angles between 0 and 180 degrees with the four sides of the industrial training sample. In other words, the sides of this multi-degree-of-freedom quadrilateral can be perpendicular or not perpendicular, parallel or not parallel to the sides of the industrial training sample. Furthermore, adjacent sides of this multi-degree-of-freedom quadrilateral can be perpendicular or not perpendicular, and opposite sides can be parallel or not parallel.

[0075] For example, Figure 7 This is a multi-degree-of-freedom quadrilateral obtained by labeling industrial training samples, as shown in the embodiments of this application. For example... Figure 7 As shown, the sides Q1 and Q2 of this multi-degree-of-freedom quadrilateral are neither perpendicular nor parallel to the sides of the industrial training sample, and the other three sides are also neither perpendicular nor parallel to the sides of the industrial training sample. Furthermore, as... Figure 7 As shown, the sides Q1Q2 and Q3Q4 of the multi-degree-of-freedom quadrilateral are parallel, while the sides Q1Q4 and Q2Q3 are not parallel. Furthermore, no two adjacent sides of this multi-degree-of-freedom quadrilateral are perpendicular to each other.

[0076] Taking the industrial defect as a weld defect and the target area as the weld area as an example, when a user uses a quadrilateral annotation frame to annotate in the industrial training sample, the quadrilateral annotation frame can be used to surround the weld, resulting in a quadrilateral surrounding the weld. It should be noted that since welds are usually long, irregular quadrilaterals, and welds are not horizontal or vertical in the industrial training sample, in this embodiment, the user can use a multi-degree-of-freedom quadrilateral annotation frame to select the weld. In this way, the multi-degree-of-freedom quadrilateral annotation frame can better adapt to the shape of the weld and more tightly surround the weld, thereby minimizing the area within the multi-degree-of-freedom quadrilateral that does not contain the weld defect.

[0077] After the user annotates the quadrilateral in the industrial training sample using a quadrilateral annotation box, the training device automatically obtains the position information of the quadrilateral in the industrial training sample. This position information may include the coordinates of the four vertices of the quadrilateral in the industrial training sample.

[0078] In addition, for each industrial training sample, after the user annotates the quadrilateral, the training device can generate annotation category information for the quadrilateral, which is used to indicate that the quadrilateral is a target region. The annotation category information can be a first identifier, exemplarily "1". Accordingly, the training device can use the annotation category information and the position information of the quadrilateral as the target region annotation information corresponding to the industrial training sample.

[0079] Step 602: Train the target region detection network using multiple industrial training samples and the target region annotation information of each industrial training sample to obtain the target region detection model.

[0080] After acquiring multiple industrial training samples and the target region annotation information for each industrial training sample, the training device trains the target region detection network using these multiple industrial training samples and the target region annotation information for each industrial training sample, thereby obtaining the target region detection model. The following explanation uses one of the multiple industrial training samples, where the annotated quadrilateral region is a multi-degree-of-freedom quadrilateral, as an example to illustrate this process. For ease of explanation, this industrial training sample will be referred to as the first industrial training sample below.

[0081] For example, the training device determines a first circumscribed rectangle and a second circumscribed rectangle based on the target region annotation information of the first industrial training sample. The first circumscribed rectangle refers to the minimum rotational circumscribed rectangle of the multi-degree-of-freedom quadrilateral annotated in the first industrial training sample, and the second circumscribed rectangle refers to the horizontal circumscribed rectangle of the first circumscribed rectangle. The target region detection network determines the predicted values ​​of the multi-degree-of-freedom quadrilaterals, the first circumscribed rectangle, the second circumscribed rectangle, and the predicted category information in the first industrial training sample. Based on the target region annotation information, the first circumscribed rectangle, the second circumscribed rectangle, the predicted values ​​of the multi-degree-of-freedom quadrilaterals, the first circumscribed rectangle, the second circumscribed rectangle, and the predicted category information in the first industrial training sample, a loss function value is calculated. The parameters of the target region detection network are updated based on the loss function value.

[0082] In this embodiment, considering the high degree of freedom of multi-degree-of-freedom quadrilaterals and the difficulty of regression, the training device first determines the smallest rotational circumscribed rectangle of the labeled multi-degree-of-freedom quadrilateral in the first industrial training sample, i.e., the first circumscribed rectangle. Then, it determines the horizontal circumscribed rectangle of the first circumscribed rectangle, i.e., the second circumscribed rectangle. The first circumscribed rectangle is the circumscribed rectangle with the smallest area, where none of its four sides are parallel to the four sides of the first industrial training sample. The second circumscribed rectangle is the circumscribed rectangle with the smallest area among the circumscribed rectangles of the first circumscribed rectangle, and where two adjacent sides are parallel to two adjacent sides of the first industrial training sample. For example, as... Figure 8 As shown, the multi-degree-of-freedom quadrilateral labeled in the first industrial training sample is Q1Q2Q3Q4, the first circumscribed rectangle is R1R2R3R4, and the second circumscribed rectangle is K1K2K3K4.

[0083] In addition, the training device uses a target region detection network to detect in the first industrial training samples, thereby obtaining the predicted values ​​of the labeled multi-degree-of-freedom quadrilateral, the predicted values ​​of the first and second bounding rectangles, and prediction category information. This prediction category information indicates whether the predicted value of the labeled multi-degree-of-freedom quadrilateral is a foreground or background region. For example, if the predicted value of the labeled multi-degree-of-freedom quadrilateral is a foreground region, the prediction category information is a first identifier; if the predicted value of the labeled multi-degree-of-freedom quadrilateral is a background region, the prediction category information is a second identifier.

[0084] After obtaining the predicted values ​​of the labeled multi-degree-of-freedom quadrilaterals, the predicted values ​​of the first circumscribed rectangle, the predicted values ​​of the second circumscribed rectangle, and the predicted category information through the target region detection network, the training device can use the above predicted values, the parameter information of each quadrilateral obtained from the labeling, the predicted category information, and the labeled category information to calculate the loss function value, and then update the parameters of the target region detection network based on the calculated loss function value.

[0085] In calculating the loss function value, the training device can parameterize the labeled data and the predicted data respectively, and then use the parameterized parameter values ​​to calculate the loss function value.

[0086] For example, taking a multi-degree-of-freedom quadrilateral labeled in the first industrial training sample as an example where one set of opposite sides are parallel and the other set of opposite sides are not parallel, such as... Figure 8 As shown, at this point, the multi-degree-of-freedom quadrilateral can be characterized by the vertex coordinates of its three vertices Q1, Q2, and Q3 and the length of its side Q3Q4. That is, the parameter information of the multi-degree-of-freedom quadrilateral is... in, Let Q1 be the coordinate of the vertex. Let Q2 be the coordinates. Let w be the coordinates of vertex Q3. Q Let be the length of side Q3Q4. Similarly, assuming the predicted value of this multi-degree-of-freedom quadrilateral is q1q2q3q4, then the parameter information of the predicted value of this multi-degree-of-freedom quadrilateral is:

[0087] The minimum rotational bounding rectangle of the multi-degree-of-freedom quadrilateral in the first industrial training sample, i.e., the first bounding rectangle, can be characterized by the coordinates of two adjacent vertices and the lengths of the adjacent sides of the edge formed by these two vertices. For example, as... Figure 8 As shown, the first bounding rectangle R1R2R3R4 can be characterized by the coordinates of vertices R1 and R2 and the lengths of edges R2 and R3. In this case, the parameter information of the first bounding rectangle is... in, Let R1 be the coordinate of the vertex. Let h be the coordinates of vertex R2. R The length of R2R3 is also the height of the first circumscribed rectangle. Alternatively, the first circumscribed rectangle can also be characterized by the coordinates of vertices R2 and R3 and the length of edge R1R2, which will not be elaborated further in this embodiment. Similarly, assuming the predicted value of the first circumscribed rectangle is r1r2r3r4, the parameter information of the predicted value of the first circumscribed rectangle can be...

[0088] For the second bounding rectangle, since it is a horizontal bounding rectangle, it can be characterized by the coordinates of one vertex and the width and height of the bounding rectangle. For example, as shown... Figure 8 As shown, the second circumscribed rectangle K1K2K3K4 can be characterized by the coordinates of vertex K1, and the lengths of edges K1K2 and K2K3. That is, the parameter information of the second circumscribed rectangle can be... in, Let w be the coordinates of vertex K1. K h is the width of the second circumscribed rectangle, which is also the length of side K1K2. K Let K1 be the height of the second circumscribed rectangle, which is also the length of side K2K3. Similarly, assuming the predicted value of the second circumscribed rectangle is k1k2k3k4, then the parameter information of the predicted value of the second circumscribed rectangle can be...

[0089] For the parameter information of each quadrilateral obtained from the above annotation and the parameter information of each quadrilateral obtained from the prediction, the training device can use the horizontal rectangular target region generated by the target region detection network in the process of predicting the above quadrilaterals to parameterize them. Assuming that the parameter information of the horizontal rectangular target region generated in the process of predicting the above quadrilaterals using the target region detection network is (x,y,w,h), the training device can parameterize the parameter information of the multi-degree-of-freedom quadrilateral annotated in the first industrial training sample and the predicted value of the multi-degree-of-freedom quadrilateral using the following formulas (1)-(3), parameterize the parameter information of the first circumscribed rectangle and the predicted value of the first circumscribed rectangle using formulas (1), (2) and (4), and parameterize the parameter information of the predicted value of the second circumscribed rectangle and the second circumscribed rectangle using formulas (1)-(4), thereby obtaining the parameter set of each quadrilateral.

[0090]

[0091]

[0092]

[0093]

[0094] In this context, when using formulas (1)-(3) to parameterize the parameters of the multi-degree-of-freedom quadrilateral labeled in the first industrial training sample and the predicted value of that multi-degree-of-freedom quadrilateral, the value of i is 1, 2, 3. The value of N is Q, q. Thus, the parameter set of the multi-degree-of-freedom quadrilateral can be expressed as follows: The parameter set of the predicted value of this multi-degree-of-freedom quadrilateral can be expressed as: When parameterizing the parameters of the first circumscribed rectangle and its predicted value using formulas (1), (2), and (4), the value of i is 1, 2, and the value of N is R, r. Thus, the parameter set of the first circumscribed rectangle can be expressed as: The parameter set of the predicted value of the first circumscribed rectangle can be expressed as follows: When parameterizing the predicted values ​​of the second circumscribed rectangle using formulas (1)-(4), the value of i is 1. The values ​​of N are K and k. Thus, the parameter set of the second circumscribed rectangle can be expressed as: The parameter set of the predicted values ​​of the second circumscribed rectangle can be expressed as:

[0095] After parameterizing the labeled data and the predicted data respectively, the training device calculates the loss function value based on the parameterized parameter groups as described above using the following formula (5).

[0096]

[0097] Among them, L cls () represents the classification loss function, l p To predict category information, l g To label category information, λ1, λ2, and λ3 are preset balancing factors used to adjust the relative contribution of the corresponding items in the process of calculating the loss function value; The regression function for the parameters of a labeled multi-degree-of-freedom quadrilateral. Let be the regression function of the parameters of the first bounding rectangle. It is a regression function of the parameters of the second circumscribed rectangle.

[0098] After obtaining the loss function value through the above formula, the training device can iteratively train the target region detection network under the guidance of the calculated loss function, thereby obtaining the target region detection model.

[0099] It should be noted that the parameters used above to characterize the quadrilaterals obtained from annotation and prediction are only one example provided by the embodiments of this application. In some possible scenarios, other parameter information can also be used to characterize the quadrilaterals, for example, the coordinates of the four vertices of each quadrilateral can be used to characterize the corresponding quadrilateral. The embodiments of this application do not limit this.

[0100] Step 603: Obtain multiple defect training samples.

[0101] In this embodiment, the training device collects multiple defect training samples. These multiple defect training samples are all images containing a target region. It should be noted that in some cases, these multiple defect training samples do not overlap with the multiple industrial training samples used to train the target region detection model; in other cases, the multiple defect training samples may include industrial training samples containing industrial defects from the aforementioned multiple industrial training samples.

[0102] Step 604: Extract multiple target regions from multiple defect training samples using the target region detection model.

[0103] After training the target region detection model and obtaining defect training samples, the training device extracts the target region from each defect training sample using the target region detection model.

[0104] It should be noted that since the target region detection model is trained by annotating quadrilaterals that may contain industrial defects in industrial training samples, this target region detection model uses multi-degree-of-freedom quadrilaterals as detection boxes. Thus, this target region detection model can detect quadrilateral target regions that may contain industrial defects from each defect training sample. Specifically, when multi-degree-of-freedom quadrilaterals are used during training, the quadrilaterals detected from each defect training sample will also be multi-degree-of-freedom quadrilaterals. However, since the training samples input to the defect detection network are usually required to be rectangular when training the network, in this embodiment, after detecting the target region, the training device can further determine the minimum bounding rectangle of the target region, fill the space between the minimum bounding rectangle and the target region with background color, and then extract the minimum bounding rectangle from the defect training samples, using the extracted image as input to the subsequent defect detection network.

[0105] For example, for a target region detection model for detecting weld defects, welds can be labeled with multi-degree-of-freedom quadrilaterals in various industrial training samples used to train the model. This allows the target region detection model to detect target regions containing welds and shaped as multi-degree-of-freedom quadrilaterals from the defect training samples. The training device can determine the minimum bounding rectangle of the target region and fill the space between the minimum bounding rectangle and the target region with a background color. Then, the minimum bounding rectangle containing the target region is extracted from the defect training samples.

[0106] Step 605: Obtain defect annotation information for target areas containing industrial defects from multiple target areas.

[0107] After extracting the target region from each defect training sample, since the target region may or may not contain industrial defects, the training device can display multiple extracted target regions to the user. The user can then select the target region containing industrial defects and annotate the industrial defects within that region. The training device then acquires the defect annotation information generated by the user during annotation in each target region.

[0108] Users can use horizontal rectangular bounding boxes to mark industrial defects within a target area, and input the defect type of the industrial defect contained within the horizontal rectangular bounding box in the defect type option. Correspondingly, the training device obtains the position of the horizontal rectangular bounding box marked by the user within the target area, and uses this position as the location of the industrial defect contained within the target area. Simultaneously, the training device obtains the defect type of the industrial defect input by the user. The obtained location and defect type of the industrial defect are used as the defect labeling information for the target area.

[0109] It should be noted that when marking industrial defects contained within a target area using a horizontal rectangular annotation box, the area of ​​the horizontal rectangular annotation box should be as small as possible while enclosing the industrial defect; that is, the horizontal rectangular annotation box should tightly enclose the industrial defect. When acquiring the position of the horizontal rectangular annotation box, the training device can acquire the position of its center point.

[0110] Step 606: Train the defect detection network using the target area containing industrial defects and the corresponding defect annotation information of the target area to obtain the defect detection model.

[0111] After obtaining the target area containing industrial defects and the corresponding defect annotation information of the target area, the training device trains the defect detection network through each target area containing industrial defects and the corresponding defect annotation information, thereby obtaining the defect detection model.

[0112] As described in step 604 above, when extracting the target region, the extracted region can be the smallest bounding rectangle containing the target region. Based on this, for any target region containing an industrial defect, the training device inputs the smallest bounding rectangle containing the target region into the defect detection network. The defect detection network identifies the industrial defects contained in the target region, thereby obtaining the predicted location and predicted defect type of the industrial defects contained in the target region. A position loss function value is calculated based on the predicted location of the industrial defect and the labeled defect location contained in the defect labeling information of the target region. A type loss function value is calculated based on the predicted defect type of the industrial defect and the labeled defect type contained in the defect labeling information. Then, the parameters of the defect detection network are updated based on the position loss function value and the type loss function value, thereby obtaining the defect detection model.

[0113] In this embodiment, target regions potentially containing industrial defects in industrial training samples are labeled using multi-degree-of-freedom quadrilaterals. The target region detection network is then trained using these industrial training samples to obtain a target region detection model. This model can more accurately detect various target regions in an image that may contain defects, distributed in different directions and shapes. Furthermore, in this embodiment, the target region detection model is first trained. Then, it is used to coarsely locate industrial defects in the defect training samples to extract target regions. The fine-grained locations and types of defects labeled within these target regions are then used to train a defect detection network to obtain a defect detection model. This allows for more accurate localization of industrial defects within the target regions when the defect detection model is used to detect them in the image to be identified. It also enables the output of specific defect types, improving both the accuracy of localization and the dimensions of defect identification.

[0114] The above embodiments mainly introduce the training process of the target region detection model and the defect detection model. Next, the process of using the target region detection model and the defect detection model to identify the image to be identified will be introduced.

[0115] Figure 9 This is a flowchart illustrating an industrial defect identification method provided in an embodiment of this application. This method can be applied to the identification device in the aforementioned industrial defect identification system. (See also...) Figure 9 The method includes the following steps:

[0116] Step 901: Obtain the image to be identified, which is an image reflecting an industrial product.

[0117] In this embodiment, the identification device acquires an image to be identified, which may be stored in the identification device or sent to the identification device by another device. The other device may be a device that acquires the image to be identified or a device that stores the image; this embodiment does not limit the specific device.

[0118] Step 902: Extract at least one target region from the image to be identified based on the target region detection model. The at least one target region is a region where industrial defects may exist.

[0119] After acquiring the image to be identified, the identification device uses the image to be identified as input to the target area detection model, and extracts at least one target area from the image to be identified through the target area detection model. The at least one target area is an area where industrial defects may exist, that is, the at least one target area refers to an area where a certain industrial defect may occur.

[0120] For example, taking a target region as an example, the recognition device determines the target region in the image to be recognized using a target region detection model and outputs the position of the target region in the image to be recognized. Then, based on the position of the target region in the image to be recognized, the recognition device obtains the smallest bounding rectangle containing the target region. The position of the target region in the image to be recognized can be characterized by the coordinates of each vertex of the target region, or it can be characterized by other parameters based on the shape of the target region; this embodiment does not limit this approach.

[0121] It should be noted that since the target region detection model is trained using quadrilaterals labeled in industrial training samples, it can identify quadrilateral target regions from the image to be recognized. Specifically, if the quadrilateral used during training is a multi-degree-of-freedom quadrilateral, the target region identified from the image to be recognized will also be a multi-degree-of-freedom quadrilateral. Since subsequent detection of industrial defects in the first region requires a defect detection model, and the image input to the defect detection model is typically required to be rectangular, in this embodiment, after detecting the target region, the recognition device can further determine the minimum bounding rectangle of the target region, fill the space between the minimum bounding rectangle and the target region with background color, and then crop the image to be recognized based on the minimum bounding rectangle of the target region to obtain an image containing the target region.

[0122] For example, in the scenario of weld defect identification, when training the target region detection model, welds are labeled with multi-degree-of-freedom quadrilateral bounding boxes in industrial training samples. Thus, when the target region detection model is used to detect an image containing welds, it can identify the multi-degree-of-freedom target region containing the welds from the image. Then, the identification device can determine the minimum bounding rectangle of the target region, fill the gap between the target region and the minimum bounding rectangle with background color, and extract the minimum bounding rectangle from the image to obtain an image containing the target region.

[0123] Step 903: Based on at least one target area and a defect detection model, obtain a rough selection area of ​​defects in the target area containing industrial defects, and the type of industrial defects contained in the rough selection area.

[0124] After extracting at least one target region through the target region detection model, the identification device uses the at least one target region as input to the defect detection model, and performs reasoning through the defect detection model to obtain the position of the defect coarse selection region of the target region containing industrial defects in the target region, and the defect type of the industrial defects contained in the defect coarse selection region; then, based on the position of the defect coarse selection region in the target region, the defect coarse selection region is extracted from the target region.

[0125] Taking one target region from at least one target region as an example, the recognition device inputs an image containing that target region into the defect detection model. The defect detection model processes the target region. It should be noted that since the target region is an area where industrial defects may occur, it may or may not contain industrial defects. Therefore, if the target region does not contain industrial defects, the defect detection model will output a recognition result indicating that there are no defects in the target region. If the target region contains industrial defects, the defect detection model can determine a preliminary defect region within the target region and identify the industrial defects within that region. Then, the defect detection model outputs the location of the preliminary defect region within the target region and the defect type of the identified industrial defects within the preliminary defect region. After obtaining the location of the preliminary defect region within the target region, the recognition device crops the target region based on this location to obtain the preliminary defect region. The location of the preliminary defect region within the target region can also be represented by the vertex coordinates of the preliminary defect region, or by other parameters based on the shape of the preliminary defect region.

[0126] It should be noted that if, during the training of the defect detection model, the industrial defects in the target area extracted from the defect training samples were annotated with horizontal rectangular boxes, then in this step, the rough selection area of ​​defects determined in the target area is also a horizontal rectangular area.

[0127] For example, in a weld defect identification scenario, a target region containing the weld is extracted from the image to be identified using a target region detection model. This target region is then input into the defect detection model, which processes it to identify a horizontal rectangular region containing the weld defect. Based on the location of this identified horizontal rectangular region, it is extracted from the target region, thus obtaining a coarsely selected defect region. In this case, the coarsely selected defect region is a smaller area than the target region, containing the weld defect. Therefore, the proportion of the weld defect in the coarsely selected defect region will be greater than its proportion in the target region.

[0128] As described above, the target region extracted by the target region detection model is a relatively large area where a certain industrial defect may occur. The defect coarse selection region obtained by the defect detection model is a smaller region containing industrial defects, extracted after further narrowing down the target region. Thus, the proportion of the industrial defect in the target region will be greater than its proportion in the image to be identified, and the proportion in the defect coarse selection region will be greater than its proportion in the target region. In other words, by extracting the target region from the image to be identified, and then extracting the defect coarse selection region from the target region, the proportion of the industrial defect relative to the background is gradually increased. Based on this, further localization and size estimation of the industrial defects in the defect coarse selection region can improve the accuracy of defect localization and size estimation. Especially for small industrial defects, continuously increasing the proportion of the industrial defect in the image using the above method is more beneficial for subsequent localization and size estimation of the industrial defect.

[0129] For each target region extracted from the image to be identified, the identification device can detect the at least one target region by referring to the above method, thereby obtaining a rough selection area of ​​the target region containing industrial defects and the type of industrial defects contained in the rough selection area from the at least one target region.

[0130] Step 904: Determine the size and location of industrial defects contained in the rough defect selection area.

[0131] After obtaining the defect coarse selection area and the defect type of the industrial defects contained in the defect coarse selection area through step 903, the identification device can further determine the size and location of the industrial defects contained in the defect coarse selection area by segmenting the defect coarse selection area.

[0132] In one possible implementation, if the coarsely selected defect region is not a grayscale image, the recognition device first converts the coarsely selected defect region to grayscale. Of course, if the coarsely selected defect region itself is a grayscale image, then the grayscale conversion operation is unnecessary. Based on this, the recognition device determines the size of the industrial defect contained in the coarsely selected defect region and the position of the industrial defect within the coarsely selected defect region according to the grayscale distribution within the grayscale region. Based on the position of the coarsely selected defect region in the target region, the position of the target region in the image to be recognized, and the position of the industrial defect contained in the coarsely selected defect region within the coarsely selected defect region, the recognition device determines the position of the industrial defect contained in the coarsely selected defect region in the image to be recognized.

[0133] It should be noted that, since the grayscale of the industrial defect area in the image is usually different from that of the non-defect area, in this embodiment of the application, the identification device can use an adaptive threshold binarization method within the region to determine the location and size of the industrial defect in the region based on the grayscale distribution within the rough selection region of the defect.

[0134] The identification device determines a grayscale threshold based on the grayscale value distribution within a coarsely selected grayscale defect area. Then, the device compares the grayscale value of each pixel within the coarsely selected grayscale defect area with this threshold. If the grayscale value of a corresponding pixel is less than the threshold, it is set to a first value; if the grayscale value is not less than the threshold, it is set to a second value. The first and second values ​​are not equal. For example, the first value is 0, and the second value is 255. Through this method, the identification device binarizes the grayscale values ​​of each pixel within the coarsely selected grayscale defect area. Then, the device determines the location and size of the industrial defect based on the pixels within the coarsely selected grayscale defect area with the first grayscale value.

[0135] After determining the location of the industrial defect within the preliminary defect selection area, the identification device, based on the location of the industrial defect within the preliminary defect selection area and the location of the preliminary defect selection area within the corresponding target area, obtains the location of the industrial defect within the target area through coordinate transformation between the target area and the preliminary defect selection area. Then, the identification device, based on the location of the industrial defect within the target area and the location of the target area in the image to be identified, obtains the location of the industrial defect in the image to be identified through coordinate transformation between the target area and the image to be identified.

[0136] It should be noted that in some possible cases, the identification device may also directly use the determined location of the industrial defect within the defect coarse selection area as the final location of the industrial defect.

[0137] After obtaining the location and size of the industrial defect, the identification device can generate an inspection report and send it back to the client. The inspection report includes the type, size, and location of the industrial defect detected in the image to be identified. Optionally, the inspection report may also include other content, such as information determining whether the industrial product in the image to be identified is qualified based on the above identification results; this embodiment of the application does not limit this.

[0138] Optionally, if the defect detection model outputs an identification result indicating that there are no defects in the target area when detecting the target area, the identification device generates a detection report containing the identification result indicating that there are no defects in the target area and sends the detection report back to the client.

[0139] In summary, the beneficial effects of the industrial defect identification method provided in this application include at least the following four aspects:

[0140] First, in the embodiments of this application, in the target region extraction stage, a neural network detection model with a multi-degree-of-freedom quadrilateral as the detection box is used, which can be suitable for different types of target regions, and the target region detection model has good versatility.

[0141] Second, in this embodiment, the target region is first extracted from the image to be identified, and then a coarse selection region containing industrial defects is obtained from the target region. By extracting regions twice, the size ratio of industrial defects relative to the background is increased, which can improve the detection probability of small-sized industrial defects.

[0142] Third, after detecting the rough defect selection area containing industrial defects, further locating and estimating the size of the industrial defects in the rough defect selection area can effectively improve the accuracy of defect location and size estimation.

[0143] Fourth, in this application embodiment, not only are defect types identified, but industrial defects can also be located and their size estimated, enriching the identification dimensions of industrial defects.

[0144] This application also provides, as in the embodiments, Figure 1 The identification device 10 shown herein includes modules and functions as described above, and will not be repeated here.

[0145] In some embodiments, the target area detection module 101 in the identification device 10 is used to perform steps 901 and 902 in the foregoing embodiments. The coarse defect detection module 102 is used to perform step 903 in the foregoing embodiments. The fine defect identification module 103 is used to perform step 904 in the foregoing embodiments.

[0146] Optionally, the model optimization device 10 may further include a report generation module 104, which can be used to generate a detection report including the recognition results and provide the detection report to the user.

[0147] This application also provides, as follows: Figure 1 The training device 11 shown herein includes modules and functions as described above, and will not be repeated here.

[0148] In some embodiments, the target region detection model generation module 111 in the training device 11 can be used to perform steps 601 and 602 in the foregoing embodiments. The target region extraction and defect annotation module 112 can be used to perform steps 603 to 605 in the foregoing embodiments. The defect detection model generation module 113 can be used to perform step 606 in the foregoing embodiments.

[0149] This application embodiment also provides a method such as Figure 5 The computing device 500 is shown. The processor 501 in the computing device 500 reads a set of computer instructions stored in the memory 503 to execute the aforementioned industrial defect identification method.

[0150] Since the modules in the identification device 10 provided in this application embodiment can be distributed and deployed on multiple computers in the same environment or different environments, this application also provides a method such as... Figure 10 The computing device shown (also referred to as a computer system) includes multiple computers 1000, each computer 1000 having a structure similar to that described above. Figure 5 The computing devices 500 in this context have the same or similar structures, and will not be described in detail here.

[0151] Each of the aforementioned computers 1000 establishes a communication channel through a communication network. Each computer 1000 runs any one or more of the aforementioned target area detection module 101, coarse defect detection module 102, fine defect identification module 103, and report generation module 104. Any computer 1000 can be a computer in a cloud data center (e.g., a server), an edge computer, or a terminal computing device.

[0152] The descriptions of the processes corresponding to the above-mentioned figures each have their own emphasis. For parts of a process that are not described in detail, please refer to the relevant descriptions of other processes.

[0153] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product for industrial defect identification includes one or more computer instructions for performing industrial defect identification. When these computer program instructions are loaded and executed on a computer, all or part of the defects according to the embodiments of this application are generated. Figure 6 and Figure 9 The process or function described.

[0154] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic cable, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium may be magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid-state disk (SSD)).

[0155] It should be understood that "at least one" as mentioned herein refers to one or more, and "multiple" refers to two or more. In the description herein, unless otherwise stated, " / " signifies "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0156] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0157] The above description is not intended to limit the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. An industrial defect identification method, characterized in that, The method includes: Acquire an image to be identified, wherein the image to be identified is an image reflecting an industrial product; At least one target region is extracted from the image to be identified according to the target region detection model. The at least one target region is an area where there may be industrial defects. The target region detection model uses a multi-degree-of-freedom quadrilateral as the detection box. The angle between each side of the multi-degree-of-freedom quadrilateral and each side of the image to be identified is any value between 0 degrees and 180 degrees. Based on the at least one target area and the defect detection model, a rough selection area of ​​defects containing industrial defects is obtained in the at least one target area, and the type of industrial defects contained in the rough selection area. Based on the grayscale distribution within the defect coarse selection area, determine the location and size of the industrial defects contained within the defect coarse selection area. Based on the location of the defect coarse selection area in the target area, the location of the target area in the image to be identified, and the location of the industrial defects contained in the defect coarse selection area in the defect coarse selection area, the location of the industrial defects contained in the defect coarse selection area in the image to be identified is determined.

2. The method according to claim 1, characterized in that, The step of obtaining a preliminary defect region containing industrial defects within the at least one target region, and the types of industrial defects contained in the preliminary defect region, based on the at least one target region and the defect detection model, includes: The at least one target region is used as the input of the defect detection model. The defect detection model is used to infer the location of the target region containing the industrial defect in the at least one target region, and the defect type of the industrial defect contained in the target region. The defect detection model is an artificial intelligence (AI) model trained using defect training samples. Based on the location of the defect coarse selection region in the target region, the defect coarse selection region is extracted from the target region.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Generate an inspection report, which includes the defect type, size, and location of the industrial defects detected in the image to be identified; Provide the user with the aforementioned test report.

4. The method according to any one of claims 1-3, characterized in that, The industrial defect is a weld defect, and the target area is the area where the weld is located.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Multiple industrial training samples are obtained, and each industrial training sample corresponds to target region annotation information. The target region annotation information includes the position information of the multi-degree-of-freedom quadrilaterals annotated in the corresponding industrial training sample and the annotation category information used to indicate that the corresponding multi-degree-of-freedom quadrilateral is the target region. The target region detection network is trained using the multiple industrial training samples and the target region annotation information of each industrial training sample to obtain the target region detection model.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Obtain multiple defect training samples; The target region detection model extracts multiple target regions from the multiple defect training samples. Obtain defect labeling information for target areas containing industrial defects from the multiple target areas. The defect labeling information is used to indicate the defect type and location of the industrial defects within the corresponding target areas. The defect detection model is obtained by training the defect detection network with target regions containing industrial defects and corresponding defect annotation information of the target regions.

7. An industrial defect identification system, characterized in that, The industrial defect identification system includes: The target region detection module is used to acquire an image to be identified, which is an image reflecting an industrial product; and to extract at least one target region from the image to be identified according to a target region detection model, wherein the at least one target region is an area where industrial defects may exist. The target region detection model uses a multi-degree-of-freedom quadrilateral as a detection box, and the angle between each side of the multi-degree-of-freedom quadrilateral and each side of the image to be identified is any value between 0 degrees and 180 degrees. A coarse defect detection module is used to obtain, based on the at least one target region and the defect detection model, a coarse defect selection region containing industrial defects in the at least one target region, and the type of industrial defects contained in the coarse defect selection region; The fine defect recognition module is used to determine the position and size of the industrial defects contained in the coarse defect selection area based on the grayscale distribution in the coarse defect selection area; and to determine the position of the industrial defects contained in the coarse defect selection area in the image to be recognized based on the position of the coarse defect selection area in the target area, the position of the target area in the image to be recognized, and the position of the industrial defects contained in the coarse defect selection area in the coarse defect selection area.

8. The system according to claim 7, characterized in that, The coarse defect detection module is used for: The at least one target region is used as the input of the defect detection model. The defect detection model is used to infer the location of the target region containing the industrial defect in the at least one target region, and the defect type of the industrial defect contained in the target region. The defect detection model is an artificial intelligence (AI) model trained using defect training samples. Based on the location of the defect coarse selection region in the target region, the defect coarse selection region is extracted from the target region.

9. The system according to any one of claims 7 or 8, characterized in that, The system also includes a report generation module, which is used for: Generate an inspection report, which includes the defect type, size, and location of the industrial defects detected in the image to be identified; Provide the user with the aforementioned test report.

10. The system according to any one of claims 7-9, characterized in that, The industrial defect is a weld defect, and the target area is the area where the weld is located.

11. The system according to any one of claims 7-10, characterized in that, The system is also used for: Multiple industrial training samples are obtained, and each industrial training sample corresponds to target region annotation information. The target region annotation information includes the position information of the multi-degree-of-freedom quadrilaterals annotated in the corresponding industrial training sample and the annotation category information used to indicate that the corresponding multi-degree-of-freedom quadrilateral is the target region. The target region detection network is trained using the multiple industrial training samples and the target region annotation information of each industrial training sample to obtain the target region detection model.

12. The system according to any one of claims 7-11, characterized in that, The system is also used for: Obtain multiple defect training samples; The target region detection model extracts multiple target regions from the multiple defect training samples. Obtain defect labeling information for target areas containing industrial defects from the multiple target areas. The defect labeling information is used to indicate the defect type and location of the industrial defects within the corresponding target areas. The defect detection model is obtained by training the defect detection network with target regions containing industrial defects and corresponding defect annotation information of the target regions.

13. A computing device, characterized in that, The computing device includes a processor and a memory, the memory being used to store a set of computer instructions, wherein when the processor executes the set of computer instructions, the computing device performs the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program code, which, when executed by a computing device, performs the method according to any one of claims 1 to 6.

15. A computer program product, characterized in that, When the computer program product is run on a computing device, the computing device performs the method of any one of claims 1 to 6.

Images