Method, device, device and medium for detecting workpiece

Abstract

The invention discloses a method, equipment, a device and a medium for detecting a workpiece. The method comprises the following steps: executing data expansion on an original image of an original workpiece; training a neural network model comprising a plurality of feature extraction branches based on the workpiece image group obtained through the data expansion to obtain a workpiece detection model; converting the workpiece detection model into a lightweight workpiece detection model; and detecting a workpiece with the same shape and at least one different size as the original workpiece based on the lightweight workpiece detection model. And new workpieces with different sizes can be rapidly detected. In addition, the lightweight workpiece detection model facilitates deployment work and reduces resource stress on the deployment side, which is particularly suitable for edge devices.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence (AI) technology, and in particular to a method, equipment, device and medium for detecting a workpiece. Background Art

[0002] With the advancement of industrial automation, intelligent industrial robots are gradually replacing manual sorting tasks in various industrial scenarios, which not only improves work efficiency but also effectively saves production costs. To achieve intelligent sorting, robots need to use machine vision technology to automatically identify and locate the sorted mixture. With the major breakthroughs in feature extraction by deep learning technology, object detection technology based on deep learning can automatically extract comprehensive image features. However, in order to achieve high detection accuracy, deep learning-based object detection networks usually design deep network structures, which leads to a significant increase in their space and time costs.

[0003] Generally speaking, performing AI-based workpiece inspection processing on a graphics processing unit (GPU) takes a long time and slows down the entire process, making it difficult to implement on industrial edge (IE) devices. Summary of the invention

[0004] The embodiments of the present invention provide a method, a device, a apparatus and a medium for detecting a workpiece.

[0005] In a first aspect, a method for detecting a workpiece is provided. The method comprises:

[0006] performing data augmentation on the original image of the original artifact;

[0007] Based on the workpiece image group obtained by the data augmentation, a neural network model including a plurality of feature extraction branches is trained to obtain a workpiece detection model;

[0008] converting the artifact detection model into a lightweight artifact detection model; and

[0009] Detection is performed on a workpiece having the same shape and at least one different size as the original workpiece based on the lightweight workpiece detection model.

[0010] Therefore, based on the model trained for data augmentation, new artifacts of different sizes can be quickly detected. Multiple feature extraction branches can improve detection performance. In addition, the lightweight artifact detection model facilitates deployment and reduces resource pressure on the deployment side, which is especially suitable for edge devices.

[0011] Preferably, performing data augmentation on the original image of the original workpiece comprises:

[0012] generating a new workpiece image in the workpiece image group based on the original image, wherein the workpiece in the new workpiece image has the same shape as the original workpiece;

[0013] wherein the length of the workpiece in the new workpiece image is the same as the length of the original workpiece, and the width of the workpiece in the new workpiece image is different from the width of the original workpiece; or

[0014] The width of the workpiece in the new workpiece image is the same as the width of the original workpiece, and the length of the workpiece in the new workpiece image is different from the length of the original workpiece.

[0015] Therefore, embodiments of the present invention can identify new workpieces that have the same shape but different lengths or widths, which is particularly suitable for mixed sorting scenarios of workpieces.

[0016] Preferably, generating a new workpiece image in the workpiece image group based on the original image comprises:

[0017] Dividing the original workpiece in the original image into a first region having an invariant feature and a second region having a variable feature;

[0018] changing the length of the second region to generate a second region of changed length;

[0019] varying a width of the second region to produce a second region of varying width;

[0020] The first region and the second region with a changed length are combined to form the new workpiece image; or the first region and the second region with a changed width are combined to form the new workpiece image.

[0021] Therefore, by changing the length or width of the second region having the variable feature, various types of new workpiece images can be easily generated.

[0022] Preferably, it includes:

[0023] Get a meta transformer;

[0024] Replacing the feature extraction layer of the meta-transformer with a plurality of feature extraction branches, wherein the plurality of feature extraction branches include at least one of: a shortcut to an adder; a 1x1 convolution kernel in parallel with a pooling unit; a 1x1 convolution kernel in parallel with a convolution unit;

[0025] The replaced meta-transformer is determined as the neural network model.

[0026] Therefore, replacing the feature extraction layer with multiple feature extraction branches can improve the detection performance.

[0027] Preferably, converting the artifact detection model into a lightweight artifact detection model comprises:

[0028] Based on the same transformation method, converting a first batch normalization unit in the workpiece detection model, a first adder connected to the first batch normalization unit, and a shortcut from the model input to the first adder into a new first batch normalization unit;

[0029] Based on the same transformation method, the pooling unit in the workpiece detection model, the second adder connected to the pooling unit, the 1x1 convolution kernel connected in parallel to the pooling unit, and the shortcut from the first adder to the second adder are converted into a new pooling unit;

[0030] Based on the same transformation method, converting the discarding unit in the workpiece detection model, the third adder connected to the discarding unit, and the shortcut from the input of the discarding unit to the third adder into a new discarding unit;

[0031] Based on the same transformation method, converting a second batch normalization unit in the workpiece detection model, a fourth adder connected to the second batch normalization unit, and a shortcut from an input of the second batch normalization unit to the fourth adder into a new second batch normalization unit;

[0032] Based on the same transformation method, the convolution unit, the fifth adder, the 1x1 convolution kernel connected in parallel with the convolution unit, and the shortcut from the input of the convolution unit to the fifth adder in the workpiece detection model are converted into a new convolution unit;

[0033] The new first batch normalization unit, the new pooling unit, the new discarding unit, the new second batch normalization unit, and the new convolution unit are sequentially connected to form a feature detection layer in the lightweight artifact detection model.

[0034] Therefore, replacing components in the feature extraction layer with lightweight components through identity transformation can improve detection speed and reduce storage requirements, which is especially suitable for edge devices.

[0035] Preferably, performing training on the neural network model comprises:

[0036] The neural network model is trained based on the workpiece image including the local label of the first region and the local label of the second region in the workpiece image group, wherein the method comprises:

[0037] Deploying the lightweight artifact detection model in an edge device;

[0038] Inputting a reference workpiece image including preset grab point coordinates into the lightweight workpiece detection model to output the center coordinates of the first area and the center coordinates of the second area in the reference workpiece image;

[0039] storing the center coordinates of the first area in the reference workpiece image, the center coordinates of the second area in the reference workpiece image, and the preset grasping point coordinates in association;

[0040] Inputting the image of the workpiece to be detected into the lightweight workpiece detection model to output the center coordinates of the first area in the image of the workpiece to be detected and the center coordinates of the second area in the image of the workpiece to be detected;

[0041] The grab point coordinates in the image of the workpiece to be detected are determined based on the center coordinates of the first area in the reference workpiece image, the center coordinates of the second area in the reference workpiece image, the preset grab point coordinates, the center coordinates of the first area in the image of the workpiece to be detected, and the center coordinates of the second area in the image of the workpiece to be detected.

[0042] Therefore, it is convenient and flexible to quickly obtain the coordinates of the gripping point of a new workpiece, which is especially suitable for sorting scenarios of mixed workpieces with the same shape but different lengths or widths.

[0043] Preferably, performing training on the neural network model comprises:

[0044] The neural network model is trained based on an artifact image in the artifact image group that includes a local label of the first region, a local label of the second region, and a global label of the artifact image, wherein the method comprises:

[0045] inputting an image including a plurality of artifacts having overlapping relationships into the lightweight artifact detection model;

[0046] An inspection result of each of the plurality of workpieces is obtained from the lightweight workpiece inspection model, wherein each inspection result of each workpiece includes a first region and a second region of the corresponding workpiece.

[0047] The artifact detection model trained on global and local labels can accurately perform detection on multiple artifacts with overlapping relationships.

[0048] In a second aspect, a device for detecting a workpiece is provided. The device comprises:

[0049] a data augmentation module configured to perform data augmentation on an original image of an original workpiece;

[0050] A training module, the training module being configured to perform training on a neural network model including a plurality of feature extraction branches based on the workpiece image group obtained by the data augmentation to obtain a workpiece detection model;

[0051] a conversion module configured to convert the artifact detection model into a lightweight artifact detection model; and

[0052] The detection module is configured to perform detection on a workpiece having the same shape and at least one different size as the original workpiece based on the lightweight workpiece detection model.

[0053] In a third aspect, an electronic device is provided, comprising a processor and a memory, wherein an application program executable by the processor is stored in the memory, so as to enable the processor to execute the method for detecting a workpiece as described in any one of the above items.

[0054] In a fourth aspect, a computer-readable medium is provided, the computer-readable medium comprising computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement a method for detecting a workpiece as described in any one of the above items.

[0055] In a fifth aspect, a computer program product is provided, the computer program product comprising a computer program, and when the computer program is executed by a processor, the computer program is used to execute the method for detecting a workpiece as described in any one of the above items. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to make the technical solutions of the examples of the present disclosure clearer, only the drawings used to describe the examples will be introduced below. Obviously, the drawings to be described below are only some examples of the present disclosure. Those skilled in the art can obtain other drawings based on these drawings without creative work.

[0057] Figure 1 is an exemplary flow chart of a workpiece detection method according to an embodiment of the present invention.

[0058] Figure 2A is a first schematic diagram of exemplary workpieces having the same shape and different lengths according to an embodiment of the present invention.

[0059] Figure 2B is a second schematic diagram of exemplary workpieces having the same shape and different lengths according to an embodiment of the present invention.

[0060] Figure 3 FIG. 4 is an exemplary diagram of performing data expansion processing on a workpiece image according to an embodiment of the present invention.

[0061] Figure 4A Schematic diagram of the feature extraction layer in the meta-transformer.

[0062] Figure 4B is a schematic diagram of a feature extraction layer participating in training according to an embodiment of the present invention.

[0063] Figure 4C is a schematic diagram of a lightweight feature extraction layer according to an embodiment of the present invention.

[0064] Figure 5 is a schematic diagram of a lightweight workpiece detection model obtained through training according to an embodiment of the present invention.

[0065] Figure 6 is an exemplary schematic diagram of the same transformation according to an embodiment of the present invention.

[0066] Figure 7 It is a comparison of the effects between the embodiments of the present invention and the prior art.

[0067] Figure 8 is an exemplary schematic diagram of an apparatus for inspecting a workpiece according to an embodiment of the present invention.

[0068] Fig. 9 is an exemplary structural diagram of an electronic device according to an embodiment of the present invention.

[0069] List of reference numerals:

[0070]

[0071]

[0072] DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the present invention clearer, the following examples are given to further explain the present invention in detail.

[0074] For the sake of brevity and intuitiveness of description, the scheme of the present invention is described below by describing several representative embodiments. Many details in the embodiments are only used to help understand the scheme of the present invention. However, it is obvious that the technical scheme of the present invention can be implemented without being limited to these details. In order to avoid unnecessarily obscuring the scheme of the present invention, some embodiments are not described in detail, but only a framework is given. In the following, "including" means "including but not limited to", and "according to..." means "at least according to..., but not limited to...". Due to the language habits of Chinese, when the number of elements is not specifically specified below, it means that the elements may be one or more, or can be understood as at least one.

[0075] In the industrial field, pick and place workpieces for self-assembly, printed circuit boards (PCBs) for automatic testing, and electronic waste sorting often rely on AI-based workpiece detection. AI-based workpiece detection takes a long time to execute on the GPU and affects the speed of the entire process, making it difficult to perform workpiece detection on industrial edge devices such as SIMATIC 227E, SIMATIC 427E, or SIMATIC S7-1500 TM NPU. In addition, in the scenario of mixed workpiece sorting, new workpieces with the same shape as old workpieces but one length or width is different and the other is the same are often encountered. Existing technologies usually require frequent retraining of AI models using real workpiece images (including real new workpieces and real old workpieces). However, frequent retraining of AI modules means more cost and time.

[0076] The embodiment of the present invention proposes a lightweight workpiece detection solution that is easy to implement at the edge device. In addition, for a new workpiece with the same shape as the old workpiece but one length or width is different and the other is the same, there is no need to frequently retrain the model, which can detect the new workpiece faster.

[0077] Figure 1 FIG. 1 is an exemplary flow chart of a workpiece detection method according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0078] Step 101: Perform data augmentation on the original image of the original workpiece.

[0079] A workpiece refers to an object to be processed during a machining process, which may be a single part or a combination of several parts fixed together. The original image means that the size of the original workpiece contained in the original image is not changed compared to the new workpiece image obtained by changing the size of the workpiece. The original workpiece is the workpiece contained in the original image. Preferably, the original image is a real captured image of the original workpiece.

[0080] Here, performing data augmentation on the original image may include:

[0081] (1) Based on the original image, a new workpiece image included in the workpiece image group is generated, wherein the workpiece in the new workpiece image has the same shape as the original workpiece. In addition, the workpiece in the new workpiece image has the same length as the original workpiece, but the workpiece in the new workpiece image has a different width from the original workpiece.

[0082] (2) Based on the original image, a new workpiece image included in the workpiece image group is generated, wherein the workpiece in the new workpiece image has the same shape as the original workpiece. In addition, the workpiece in the new workpiece image has the same width as the original workpiece, but the workpiece in the new workpiece image has a different length from the original workpiece.

[0083] Preferably, the workpiece image group may also include original images.

[0084] Figure 2A is a first schematic diagram of exemplary workpieces having the same shape and different lengths according to an embodiment of the present invention. Figure 2A , the workpieces 10 , 11 , 12 , and 13 all have the same shape and width, and the workpieces 10 , 11 , 12 , and 13 each have a different length. Figure 2B is a second schematic diagram of exemplary workpieces having the same shape and different lengths according to an embodiment of the present invention. Figure 2B , the workpieces 14 , 15 , and 16 all have the same shape and width, and the workpieces 14 , 15 , and 16 each have a different length.

[0085] In one embodiment, step 101 includes:

[0086] Sub-step (1): dividing the original workpiece into a first region with invariant features and a second region with variable features;

[0087] Sub-step (2): changing the length of the second region to generate a second region of changed length; alternatively, changing the width of the second region to generate a second region of changed width.

[0088] Sub-step (3): combining the first region and the second region whose length is changed into a new workpiece image. Alternatively, combining the first region and the second region whose width is changed to create a new workpiece image.

[0089] For example, a first region may be implemented as a textured region, while a second region may be implemented as a less textured region.

[0090] Figure 32 is an exemplary schematic diagram of performing data enhancement processing on a workpiece image according to an embodiment of the present invention. The workpiece image 20 includes a first region 21, 23 and a second region 24. In the first region 21, there is also a symbol 22 (for example, implemented as a workpiece mark). The first regions 21 and 23 have a specific shape and a constant length. The second region 24 is a region with less texture whose length can be changed. First, the workpiece image 20 is divided into the first region 21, 23 and the second region 24. Next, the second region 24 is evenly divided into several sub-blocks 241, such as 5 sub-blocks. Next, the sub-block 241 can be copied to increase the length of the second region 24, or the sub-block 241 can be deleted to reduce the length of the second region 24. Finally, the second region 24 is recombined with the first region 21 and 23 after the length is changed to obtain a new workpiece image. For example, a sub-block 241 (a total of 6 sub-blocks 241) is added to the second region 24, and then the second region 24 containing 6 sub-blocks 241 is recombined with the first region 21 and 23 to obtain a new workpiece image 25.

[0091] exist Figure 3 In the example of increasing the length of the workpiece to achieve data expansion, an exemplary embodiment is shown, and in fact, the length of the workpiece can also be reduced to achieve data expansion. Optionally, the length of the workpiece can remain unchanged, and the width of the workpiece can be changed to achieve data expansion.

[0092] Step 102: Based on the workpiece image group obtained by data expansion, a neural network model including multiple feature extraction branches is trained to obtain a workpiece detection model.

[0093] Here, the artifact image group may include: an artifact image (original image) before the data enhancement in step 101 and a new artifact image generated based on the data enhancement in step 101 .

[0094] In one implementation, the process of obtaining a neural network model including multiple feature extraction branches specifically includes: obtaining a meta-transformer; replacing the feature extraction layer of the meta-transformer with multiple feature extraction branches, wherein the multiple feature extraction branches include at least one of the following: a shortcut to an adder; a 1x1 convolution kernel connected in parallel with a pooling unit; a 1x1 convolution kernel connected in parallel with a convolution unit; and determining the replaced meta-transformer as a neural network model.

[0095] Meta-Transformer is a new multimodal learning framework for processing and correlating information from multiple modalities. Despite the inherent gaps between various data, Meta-Transformer leverages a frozen encoder to extract high-level semantic features from input data in a shared labeled space without the need for paired multimodal training data. This framework consists of a unified data landmark, a modality-shared encoder, and a task header for various downstream tasks.

[0096] Figure 4A Schematic diagram of the feature extraction layer in the meta-transformer. The feature extraction layer in the meta-transformer includes: a first batch normalization (BN) unit 31, a pooling unit 32, a discarding unit 33, a second BN unit 35, a convolution unit 36, an adder 34 connected between the discarding unit 33 and the second BN unit 35, and an adder 37 after the convolution unit 36. Among them, the data input of the pooling unit 32, the data input of the discarding unit 33, and the data input of the convolution unit 36 ​​all require 2 storage spaces (2x memory).

[0097] Currently, there are few feature extraction branches in the feature extraction layer of the meta-transformer, making it difficult to ensure the detection effect. By replacing the feature extraction layer of the meta-transformer with multiple feature extraction branches, the detection effect can be improved. Multiple feature extraction branches can be implemented by various methods, such as adding at least one of the following: a shortcut to the adder; a 1x1 convolution kernel in parallel with the pooling unit; a 1x1 convolution kernel in parallel with the convolution unit, etc.

[0098] Figure 4B Schematic diagram of the feature extraction layer participating in training according to an embodiment of the present invention. Figure 4B , the feature extraction layer 80 includes: a first BN unit 41, a first adder 42, a pooling unit 43, a first 1x1 convolution kernel 44, a second adder 45, a discarding unit 46, a third adder 47, a second BN unit 48, a fourth adder 49, a convolution unit 50 (such as a 3x3 convolution kernel), a second 1x1 convolution kernel 51 and a fifth adder 52. In addition, the output of the first adder 42 has a shortcut to the input of the second adder 45. The output of the second adder 45 has a shortcut to the input of the third adder 47. The output of the third adder 47 has a shortcut to the input of the fourth adder 49. The output of the fourth adder 49 has a shortcut to the input of the fifth adder 52. The first 1x1 convolution kernel 44 is parallel to the pooling unit 43, and the second 1x1 convolution kernel 51 is parallel to the convolution unit 50.

[0099] Compared with the feature extraction layer in the meta-transformer, the feature extraction layer 80 involved in training has more feature extraction branches and thus has better feature extraction performance.

[0100] Based on a group of workpiece images obtained after data augmentation (including original workpiece images before data augmentation and new workpiece images generated based on data augmentation), a meta-transformer with a replaced feature extraction layer is trained to obtain a workpiece detection model suitable for detecting workpieces having the same shape and at least one different size as the original workpiece.

[0101] Step 103: Convert the workpiece detection model into a lightweight workpiece detection model.

[0102] Here, based on the identity transformation method, the artifact detection model in step 102 is converted into a lightweight artifact detection model, which is conducive to deploying the artifact detection model at the edge device.

[0103] Figure 4C Schematic diagram of a lightweight feature extraction layer according to an embodiment of the present invention. Figure 4C As shown, a lightweight transformation can include:

[0104] (1) Based on the same transformation method, the first BN unit 41, the first adder 42 connected to the first BN unit 41, and the shortcut from the model input to the first adder 42 are converted into a new first BN unit 60.

[0105] (2) Based on the same transformation method, the pooling unit 43, the second adder 45 connected to the pooling unit 43, the 1x1 convolution kernel 44 connected in parallel with the pooling unit 43, and the shortcut from the first adder 43 to the second adder 45 are converted into a new pooling unit 61.

[0106] (3) Based on the same conversion method, the discarding unit 46, the third adder 47 connected to the discarding unit 46, and the shortcut from the input of the discarding unit 46 to the third adder 47 are converted into a new discarding unit 62.

[0107] (4) Based on the same transformation method, the second BN unit 48, the fourth adder 49 connected to the second BN unit 48, and the shortcut from the input of the second BN unit 48 to the fourth adder 49 are converted into a new second BN unit 63.

[0108] (5) Based on the same transformation method, the convolution unit 50, the fifth adder 52, the 1x1 convolution kernel 51 connected in parallel with the convolution unit 50, and the shortcut from the input of the convolution unit 50 to the fifth adder 52 are converted into a new convolution unit 64.

[0109] Figure 6 is an exemplary schematic diagram of the same transformation according to an embodiment of the present invention. Figure 6 In the process, the pooling unit 43, the second adder 45 connected to the pooling unit 43, the 1x1 convolution kernel 44 connected in parallel with the pooling unit 43, and the shortcut from the first adder 43 to the second adder 43l are converted into a new pooling unit 61.

[0110] The new first BN unit 60, the new pooling unit 61, the new discarding unit 62, the new second BN unit 63 and the new convolution unit 64 are sequentially connected to form a feature detection layer 90 in the lightweight workpiece detection model. In the feature detection layer 90 of the lightweight workpiece detection model, a storage space (1x memory) is required for data storage, thereby reducing storage pressure.

[0111] Figure 5 7 is a schematic diagram of a lightweight artifact detection model obtained through training according to an embodiment of the present invention. The meta-transformer architecture 70 includes a model input 71, a feature extraction layer 72, and a classification layer 73. For example, the feature extraction layer 72 has Figure 4A The structure shown.

[0112] First, use Figure 4B The feature extraction layer 80 in the meta-transformer architecture 70 replaces the feature extraction layer 72 in the meta-transformer architecture 70. Then, the meta-transformer architecture 70 (where the feature extraction layer 72 is replaced by the feature extraction layer 80) is trained using the artifact images in the artifact image group. The artifact image group includes the original artifact image and the new artifact image, where the new artifact image has the same shape as the original artifact and a different width or length. The feature extraction layer 75 in the trained meta-transformer architecture 74 is transformed into Figure 4C The lightweight feature extraction layer 90 in is used to obtain a lightweight workpiece detection model. The lightweight workpiece detection model can then be deployed to an industrial edge device to detect mixed workpieces with the same shape and different lengths or widths.

[0113] Step 104: Perform detection on a workpiece having the same shape and at least one different size as the original workpiece based on the lightweight workpiece detection model.

[0114] Here, detecting the workpiece may include identifying the workpiece, classifying the workpiece, locating the workpiece, and the like.

[0115] In one embodiment, step 102 includes: training a neural network model based on a workpiece image group to obtain a workpiece detection model capable of detecting the first and second regions, wherein the workpiece image group contains images with local labels of the first and second regions. The method includes: deploying a lightweight workpiece detection model in an edge device; inputting a reference workpiece image including preset grab point coordinates into the lightweight workpiece detection model to output the center coordinates of the first region and the center coordinates of the second region in the reference workpiece image; storing the center coordinates of the first region in the reference workpiece image, the center coordinates of the second region in the reference workpiece image, and the preset grab point coordinates in association; inputting an image of the workpiece to be detected into the lightweight workpiece detection model to output the center coordinates of the first region in the image of the workpiece to be detected and the center coordinates of the second region in the image of the workpiece to be detected; determining the grab point coordinates in the image of the workpiece to be detected based on the center coordinates of the first region in the reference workpiece image, the center coordinates of the second region in the reference workpiece image, the preset grab point coordinates, the center coordinates of the first region in the image of the workpiece to be detected, and the center coordinates of the second region in the image of the workpiece to be detected.

[0116] For example, the user calibrates the grasping point in the reference workpiece image. Assume that the grasping point coordinates are (x b ,y b The reference workpiece image is input into the lightweight workpiece detection model in the edge device to detect the center coordinates (x a ,y a ) and the center coordinates (x c ,y c Then, the center coordinates (x a ,y a ), the center coordinates of the second region (x c ,y c ) and the coordinates of the grab point (x b ,y b ).

[0117] Next, an image of the workpiece to be detected (e.g., a workpiece in the image of the workpiece to be detected that has the same shape and different length as the reference workpiece in the reference workpiece image) is input into the lightweight workpiece detection model. The lightweight workpiece detection model outputs the center coordinates (x newa ,y newa ) and the center coordinates (x newc ,y newc ).

[0118] Then, the coordinates of the grasping point in the image of the workpiece to be inspected (x newb ,y newb ) can be calculated as:

[0119]

[0120] Therefore, it is convenient and flexible to quickly obtain the coordinates of the gripping point of a new workpiece, which is especially suitable for sorting scenarios of mixed workpieces with the same shape but different lengths or widths.

[0121] In one embodiment, step 102 includes: training a neural network model based on a global label of the workpiece image, a local label of the first region, and a local label of the second region to obtain a workpiece detection model. The workpiece detection model can detect the overall contour of the workpiece image, and the corresponding contours of the first region and the second region in the workpiece image. Step 104 includes: inputting a test image containing multiple workpieces with overlapping relationships into the lightweight workpiece detection model; obtaining a detection result of each workpiece from the lightweight workpiece detection model, wherein the detection result of each workpiece includes its corresponding first region and second region. The overall contour to which the component belongs can be determined based on the overall contour with the maximum overlap among multiple overall contours determined by the components (including the first region and the second region) and the global label.

[0122] For example, if three workpieces in an image have an overlapping relationship, the workpiece detection model identifies: the overall contour 1 of the first workpiece, the overall contour 2 of the second workpiece, and the overall contour 3 of the third workpiece. The workpiece detection model also identifies: the first region 1, the first region 2, and the first region 3 of the three workpieces; the second region 1, the second region 2, and the second region 3 of the three workpieces. If the overall contour that overlaps most with the first region 1 is the overall contour 1, and the overall contour that overlaps most with the second region 2 is the overall contour 1, then it is determined that the first region 1 and the second region 2 are components of the overall contour 1. Similarly, the corresponding components of the overall contour 2 and the overall contour 3 are determined.

[0123] The artifact detection model trained based on global and local labels can accurately detect multiple artifacts with overlapping relationships separately.

[0124] Figure 7 9 is a comparison of the effects between the embodiments of the present invention and the prior art. The workpiece detection model in the prior art is trained by the original workpiece image 84 and the original workpiece image 86. The original workpiece image 84, the original workpiece image 86, and the workpiece images 83 and 85 having the same shape and different lengths as the original workpiece image are input into the workpiece detection model of the prior art, and a detection result 92 is obtained. According to the detection result 92, only the original workpiece image 84 and the original workpiece image 86 can be recognized, while the workpiece images 83 and 85 cannot be recognized.

[0125] In an embodiment of the present invention, data augmentation processing is performed on the original workpiece image 84 (or the original workpiece image 86) to generate workpiece images 83 and 85 having the same shape and different lengths as the original workpiece image. The workpiece detection model is trained by the original workpiece image 84, the original workpiece image 86, and the workpiece images 83 and 85. The original workpiece image 84, the original workpiece image 86, and the workpiece images 83 and 85 having the same shape and different lengths as the original workpiece image are input into the workpiece detection model of the present invention to obtain the detection result of the embodiment of the present invention. In the detection result, the original workpiece images 84 and 86 and the workpiece images 83 and 85 can all be identified.

[0126] Figure 8 FIG. 1 is an exemplary schematic diagram of an apparatus for detecting a workpiece according to an embodiment of the present invention. Figure 8 As shown, the device 800 for detecting a workpiece includes:

[0127] A data expansion module 801, which is configured to perform data expansion on an original image of an original workpiece; a training module 802, which is configured to perform training on a neural network model including a plurality of feature extraction branches based on a workpiece image group obtained through data expansion to obtain a workpiece detection model; a conversion module 803, which is configured to convert the workpiece detection model into a lightweight workpiece detection model; and a detection module 804, which is configured to perform detection on a workpiece having the same shape and at least one different size as the original workpiece based on the lightweight workpiece detection model.

[0128] In one embodiment, the data expansion module 801 is configured to generate a new workpiece image in the workpiece image group based on the original image, wherein the workpiece in the new workpiece image has the same shape as the original workpiece; wherein the length of the workpiece in the new workpiece image is the same as the length of the original workpiece, and the width of the workpiece in the new workpiece image is different from the width of the original workpiece; or wherein the width of the workpiece in the new workpiece image is the same as the width of the original workpiece, and the length of the workpiece in the new workpiece image is different from the length of the original workpiece.

[0129] In one embodiment, the data expansion module 801 is configured to divide the original workpiece in the original image into a first region with invariant features and a second region with variable features; change the length of the second region to generate a second region with a changed length; change the width of the second region to generate a second region with a changed width; combine the first region and the second region with a changed length to form a new workpiece image; or combine the first region and the second region with a changed width to form a new workpiece image.

[0130] In one embodiment, the training module 802 is configured to obtain a meta-transformer; replace a feature extraction layer of the meta-transformer with a plurality of feature extraction branches, wherein the plurality of feature extraction branches include at least one of: a shortcut to an adder; a 1x1 convolution kernel in parallel with a pooling unit; a 1x1 convolution kernel in parallel with a convolution unit; and determine the replaced meta-transformer as a neural network model.

[0131] In one embodiment, the conversion module 803 is configured to convert the first batch normalization unit in the workpiece detection model, the first adder connected to the first batch normalization unit, and the shortcut from the model input to the first adder into a new first batch normalization unit based on the same transformation method; convert the pooling unit in the workpiece detection model, the second adder connected to the pooling unit, the 1x1 convolution kernel connected in parallel with the pooling unit, and the shortcut from the first adder to the second adder into a new pooling unit based on the same transformation method; convert the discarding unit in the workpiece detection model, the third adder connected to the discarding unit, and the shortcut from the input of the discarding unit to the third adder into A new discard unit; based on the same transformation method, a second batch normalization unit in the workpiece detection model, a fourth adder connected to the second batch normalization unit, and a shortcut from the input of the second batch normalization unit to the fourth adder are converted into a new second batch normalization unit; based on the same transformation method, a convolution unit, a fifth adder, a 1x1 convolution kernel connected in parallel with the convolution unit, and a shortcut from the input of the convolution unit to the fifth adder in the workpiece detection model are converted into a new convolution unit; a new first batch normalization unit, a new pooling unit, a new discard unit, a new second batch normalization unit, and a new convolution unit are sequentially connected to form a feature detection layer in the lightweight workpiece detection model.

[0132] In one embodiment, the training module 802 is configured to train a neural network model based on an image of an artifact including a local label of a first region and a local label of a second region in the artifact image group; the detection module 804 is configured to deploy a lightweight artifact detection model in an edge device; a reference artifact image including preset grasping point coordinates is input into the lightweight artifact detection model to output the center coordinates of the first region and the center coordinates of the second region in the reference artifact image; the center coordinates of the first region in the reference artifact image, the center coordinates of the second region in the reference artifact image, and the preset grasping point coordinates are stored in association; the image of the artifact to be detected is input into the lightweight artifact detection model to output the center coordinates of the first region in the image of the artifact to be detected and the center coordinates of the second region in the image of the artifact to be detected; the grasping point coordinates in the image of the artifact to be detected are determined based on the center coordinates of the first region in the reference artifact image, the center coordinates of the second region in the reference artifact image, the preset grasping point coordinates, the center coordinates of the first region in the image of the artifact to be detected, and the center coordinates of the second region in the image of the artifact to be detected.

[0133] In one embodiment, the training module 802 is configured to train a neural network model based on a workpiece image in a workpiece image group that includes a local label of a first region, a local label of a second region, and a global label of the workpiece image; the detection module 804 is configured to input an image including multiple workpieces with overlapping relationships into a lightweight workpiece detection model; and obtain a detection result of each of the multiple workpieces from the lightweight workpiece detection model, wherein each detection result of each workpiece includes the first region and the second region of the corresponding workpiece.

[0134] An embodiment of the present invention further provides an electronic device having a processor-memory architecture. Fig. 9 is an exemplary structural diagram of an electronic device having a processor memory architecture according to an embodiment of the present invention. Fig. 9 As shown, the electronic device 900 includes a processor 901, a memory 902, and a computer program stored in the memory 902 and can be run on the processor 901. When the computer program is executed by the processor 901, the workpiece detection method described in any of the above methods is implemented. Among them, the memory 902 can be implemented as various storage media, such as an electrically erasable programmable read-only memory (EEPROM), a flash memory, a programmable program read-only memory (PROM), etc. The processor 901 can be implemented as including one or more central processing units or one or more field programmable gate arrays, wherein the field programmable gate array integrates one or more central processing unit cores. Specifically, the central processing unit or core can be implemented as a CPU, an MCU, a DSP, etc.

[0135] It should be noted that all steps and modules in the above process and structure diagram are not all necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed, and can be adjusted as needed. The division of each module is just for the convenience of describing the used functional division. In the actual implementation scheme, the module can be divided into multiple modules, and the functions of multiple modules can also be implemented by the same module. These modules can be located in the same device or in different devices.

[0136] The hardware module in each embodiment can be implemented mechanically or electronically. For example, the hardware module can include a specially designed permanent circuit or logic device (such as a special processor, such as an FPGA or ASIC) to complete a specific operation. The hardware module can also include a programmable logic device or circuit (such as a general purpose processor or other programmable processor) temporarily configured by software for performing a specific operation. As for the specific use of mechanical methods, it can be determined based on cost and time considerations whether a dedicated permanent circuit or a temporarily configured circuit (such as software configuration) is used to implement the hardware module.

[0137] The above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting a workpiece, the method comprising: performing (101) data augmentation on the original image of the original workpiece; Based on the workpiece image group obtained by the data augmentation, a neural network model including a plurality of feature extraction branches is trained (102) to obtain a workpiece detection model; converting (103) the artifact detection model into a lightweight artifact detection model; and An inspection is performed (104) on a workpiece having the same shape and at least one different size as the original workpiece based on the lightweight workpiece inspection model.

2. The method of claim 1, wherein performing (101) data augmentation on the original image of the original workpiece comprises: generating a new workpiece image in the workpiece image group based on the original image, wherein the workpiece in the new workpiece image has the same shape as the original workpiece; wherein the length of the workpiece in the new workpiece image is the same as the length of the original workpiece, and the width of the workpiece in the new workpiece image is different from the width of the original workpiece; or The width of the workpiece in the new workpiece image is the same as the width of the original workpiece, and the length of the workpiece in the new workpiece image is different from the length of the original workpiece.

3. The method according to claim 2, wherein generating a new artifact image in the artifact image group based on the original image comprises: Dividing the original workpiece in the original image into a first region having an invariant feature and a second region having a variable feature; changing the length of the second region to generate a second region of changed length; varying a width of the second region to produce a second region of varying width; The first region and the second region with a changed length are combined to form the new workpiece image; or the first region and the second region with a changed width are combined to form the new workpiece image.

4. The method according to claim 1, comprising: Get a meta transformer; Replacing the feature extraction layer of the meta-transformer with a plurality of feature extraction branches, wherein the plurality of feature extraction branches include at least one of: a shortcut to an adder; a 1x1 convolution kernel in parallel with a pooling unit; a 1x1 convolution kernel in parallel with a convolution unit; The replaced meta-transformer is determined as the neural network model.

5. The method according to claim 4, wherein converting (103) the artifact detection model into a lightweight artifact detection model comprises: Based on the same transformation method, converting a first batch normalization unit in the workpiece detection model, a first adder connected to the first batch normalization unit, and a shortcut from the model input to the first adder into a new first batch normalization unit; Based on the same transformation method, the pooling unit in the workpiece detection model, the second adder connected to the pooling unit, the 1x1 convolution kernel connected in parallel to the pooling unit, and the shortcut from the first adder to the second adder are converted into a new pooling unit; Based on the same transformation method, converting the discarding unit in the workpiece detection model, the third adder connected to the discarding unit, and the shortcut from the input of the discarding unit to the third adder into a new discarding unit; Based on the same transformation method, converting a second batch normalization unit in the workpiece detection model, a fourth adder connected to the second batch normalization unit, and a shortcut from an input of the second batch normalization unit to the fourth adder into a new second batch normalization unit; Based on the same transformation method, the convolution unit, the fifth adder, the 1x1 convolution kernel connected in parallel with the convolution unit, and the shortcut from the input of the convolution unit to the fifth adder in the workpiece detection model are converted into a new convolution unit; The new first batch normalization unit, the new pooling unit, the new discarding unit, the new second batch normalization unit, and the new convolution unit are sequentially connected to form a feature detection layer in the lightweight artifact detection model.

6. The method of claim 3, wherein performing training (102) on the neural network model comprises: The neural network model is trained based on the workpiece image including the local label of the first region and the local label of the second region in the workpiece image group, wherein the method comprises: Deploying the lightweight artifact detection model in an edge device; Inputting a reference workpiece image including preset grab point coordinates into the lightweight workpiece detection model to output the center coordinates of the first area and the center coordinates of the second area in the reference workpiece image; storing the center coordinates of the first area in the reference workpiece image, the center coordinates of the second area in the reference workpiece image, and the preset grab point coordinates in association; Inputting the image of the workpiece to be detected into the lightweight workpiece detection model to output the center coordinates of the first area in the image of the workpiece to be detected and the center coordinates of the second area in the image of the workpiece to be detected; The grab point coordinates in the image of the workpiece to be detected are determined based on the center coordinates of the first area in the reference workpiece image, the center coordinates of the second area in the reference workpiece image, the preset grab point coordinates, the center coordinates of the first area in the image of the workpiece to be detected, and the center coordinates of the second area in the image of the workpiece to be detected.

7. The method of claim 3, wherein performing training (102) on the neural network model comprises: The neural network model is trained based on an artifact image in the artifact image group that includes a local label of the first region, a local label of the second region, and a global label of the artifact image, wherein the method comprises: inputting an image including a plurality of artifacts having overlapping relationships into the lightweight artifact detection model; An inspection result of each of the plurality of workpieces is obtained from the lightweight workpiece inspection model, wherein each inspection result of each workpiece includes a first region and a second region of the corresponding workpiece.

8. A device for detecting a workpiece, the device comprising: A data expansion module (801), the data expansion module being configured to perform data expansion on an original image of an original workpiece; A training module (802), the training module being configured to perform training on a neural network model including a plurality of feature extraction branches based on the workpiece image group obtained by the data augmentation to obtain a workpiece detection model; A conversion module (803), the conversion module being configured to convert the artifact detection model into a lightweight artifact detection model; as well as A detection module (804) is configured to perform detection on a workpiece having the same shape and at least one different size as the original workpiece based on the lightweight workpiece detection model.

9. An electronic device comprising a processor (901) and a memory (902), wherein an application program executable by the processor (901) is stored in the memory (902) so that the processor (901) executes the method for detecting a workpiece according to any one of claims 1 to 7. 10 . A computer-readable medium comprising computer-readable instructions stored thereon, wherein the computer-readable instructions are used to execute the method for inspecting a workpiece according to any one of claims 1 to 7. 11 . A computer program product, comprising a computer program, for executing the method for inspecting a workpiece according to claim 1 when the computer program is executed by a processor.

Images