METHOD, DEVICE, APPARATUS AND MEDIUM FOR DETECTING A WORKPIECE

Description

FIELD

[0001] The present invention relates to the technical field of artificial intelligence (AI) technology, in particular to a method, apparatus, device, and medium for detecting workpiece.BACKGROUND

[0002] With the improvement of industrial automation, intelligent industrial robots are gradually replacing manual sorting tasks in various industrial scenarios, which not only improves work efficiency but also effectively saves production costs. To achieve intelligent sorting, robots need to use machine vision technology to automatically identify and locate the sorted mixture. With significant breakthroughs in feature extraction in deep learning technology, object detection technology based on deep learning can automatically extract comprehensive image features. However, in order to achieve high detection accuracy, deep learning-based object detection networks usually have deep network structures designed, which leads to a significant increase in their spatial and temporal costs.

[0003] In general, performing AI based workpiece detection processing on image processors (GPUs) takes a long time and affects speed of entire process, making it difficult to implement on industrial edge (IE) devices.

[0004] CAO HU et al., "Efficient Grasp Detection Network With Gaussian-Based Grasp Representation for Robotic Manipulation", IEEE / ASME Trans. Mechatronics, vol. 28, no. 3, June 2023, pages 1384-1394, relates to robotic grasp detection, in which a deep learning model is trained to detect grasp positions on objects using RGB or depth images. The deep learning model is a lightweight generative network architecture for grasp detection in one stage. Online data augmentation is conducted to train the network on small datasets. Specifically, random crops, zooms, and rotations are used to improve the diversity of these datasets.

[0005] DING XIAOHAN et al., "RepVGG: Making VGG-style ConvNets Great Again", 2021 IEEE / CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 20 June 2021, pages 13728-13737, relates to convolutional neural network architecture re-parametrisation.

[0006] CHEN WEI et al., "FS-Net: Fast Shape-based Network for Category-Level 6D Object Pose Estimation with Decoupled Rotation Mechanism", 2021 IEEE / CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 20 June 2021, pages 1581-1590, relates to object pose estimation from RGB-D data.SUMMARY

[0007] Embodiments of the present invention propose a method, apparatus, device, and medium for detecting workpiece.

[0008] In a first aspect, a method for detecting workpiece is provided. The method comprising: performing data augmentation on an original image of an original workpiece; performing training on a neural network model comprising multiple feature extraction branches to obtain a workpiece detection model based on a workpiece image group obtained through the data augmentation; converting the workpiece detection model into a lightweight workpiece detection model; and performing detection on workpieces with the same shape and at least one different dimension as the original workpiece based on the lightweight workpiece detection model.

[0009] Therefore, new workpieces of different dimensions can be detected quickly based on model trained on data augmentation. Multiple feature extraction branches can improve detection performance. Moreover, lightweight workpiece detection model facilitates deployment work and reduces resource pressure on deployment side, especially suitable for edge devices.

[0010] Preferably, wherein performing data augmentation on an original image of an original workpiece comprises: generating a new workpiece image in the workpiece image group based on the original image, a workpiece in the new workpiece image has the same shape as the original workpiece; wherein the length of workpiece in the new workpiece image is the same as that of the original workpiece, the width of workpiece in the new workpiece image is different from that of the original workpiece; or wherein the width of workpiece in the new workpiece image is the same as that of the original workpiece, the length of workpiece in the new workpiece image is different from that of the original workpiece.

[0011] Therefore, embodiments of the present invention can identify new workpieces with the same shape but different lengths or widths, especially suitable for mixed sorting scenarios of workpieces.

[0012] Preferably, wherein generating a new workpiece image in the workpiece image group based on the original image comprises: dividing the original workpiece in the original image into a first region with invariant features and a second region with variable features; changing the length of the second region to generate a second region with changed length; changing the width of the second region to generate a second region with changed width; combining the first region and the second region with changed length to form the new workpiece image; or combining the first region and the second region with changed width to form the new workpiece image.

[0013] Therefore, by changing the length or width of the second region with variable features, many types of new workpiece images can be easily generated.

[0014] Preferably, comprising: obtaining a meta transformer; replacing a feature extraction layer of the meta transformer with multiple feature extraction branches, wherein the multiple feature extraction branches comprise at least one of the following: shortcut to adder; 1x1 convolutional kernel in parallel with pooling unit; 1x1 convolutional kernel in parallel with convolutional unit; determining the replaced meta transformer as the neural network model.

[0015] Therefore, replacing the feature extraction layer with multiple feature extraction branches can improve detection performance.

[0016] Preferably, wherein converting the workpiece detection model into a lightweight workpiece detection model comprises: converting a first batch normalization unit, a first adder connected to the first batch normalization unit, and a shortcut from model input to the first adder in the workpiece detection model into a new first batch normalization unit, based on an identical transformation method; converting a pooling unit, a second adder connected to the pooling unit, a 1x1 convolutional kernel in parallel with the pooling unit, and a shortcut from the first adder to the second adder in the workpiece detection model into a new pooling unit, based on an identical transformation method; converting a dropout unit, a third adder connected to the dropout unit, and a shortcut from the input of the dropout unit to the third adder in the workpiece detection model into a new dropout unit, based on an identical transformation method; converting a second batch normalization unit, a fourth adder connected to the second batch normalization unit, and a shortcut from the input of the second batch normalization unit to the fourth adder in the workpiece detection model into a new second batch normalization unit, based on an identical transformation method; converting a convolutional unit, a fifth adder, a 1x1 convolutional kernel in parallel with the convolutional unit, and a shortcut from the input of the convolutional unit to the fifth adder in the workpiece detection model into a new convolutional unit, based on an identical transformation method; connecting the new first batch normalization unit, the new pooling unit, the new dropout unit, the new second batch normalization unit, and the new convolution unit in sequence to form a feature detection layer in the lightweight workpiece detection model.

[0017] Therefore, replacing the components in the feature extraction layer with lightweight components through identity transformation can improve detection speed and reduce storage requirements, especially suitable for edge devices.

[0018] Preferably, wherein performing training on a neural network model comprising: training the neural network model based on a workpiece image in the workpiece image group comprising a local label of the first region and a local label of the second region, wherein the method comprises: deploying the lightweight workpiece detection model in an edge device; inputting a reference workpiece image comprising preset grasping point coordinates into the lightweight workpiece detection model to output center coordinates of the first region and center coordinates of the second region in the reference workpiece image; storing associatively the center coordinates of the first region in the reference workpiece image, the center coordinates of the second region in the reference workpiece image, and the preset gripping point coordinates; inputting an image of workpiece to be detected into the lightweight workpiece detection model to output center coordinates of the first region in the image of workpiece to be detected and center coordinates of the second region in the image of workpiece to be detected; determining gripping point coordinates in the image of workpiece to be detected based on the center coordinates of the first region in the reference workpiece image, the center coordinates of the second region in the reference workpiece image, the preset gripping point coordinates, the center coordinates of the first region in the image of workpiece to be detected, and the center coordinates of the second region in the image of workpiece to be detected.

[0019] Therefore, it is convenient and flexible to quickly obtain gripping point coordinates of new workpieces, especially suitable for sorting scenarios of mixed workpieces with the same shape but different lengths or widths.

[0020] Preferably, wherein performing training on a neural network model comprising: training the neural network model based on a workpiece image in the workpiece image group comprising a local label of the first region, a local label of the second region and a global label of the workpiece image, wherein the method comprises: inputting an image comprising multiple workpieces with overlapping relationships into the lightweight workpiece detection model; obtaining detection results for each of the multiple workpieces from the lightweight workpiece detection model, wherein each detection result for each workpiece comprises first region and second region of corresponding workpiece.

[0021] The workpiece detection model trained on global and local labels can accurately perform detection for multiple workpieces with overlapping relationships.

[0022] In a second aspect, an apparatus for detecting workpiece is provided. The apparatus comprising: a data augmentation module, configured to perform data augmentation on an original image of an original workpiece; a training module, configured to perform training on a neural network model comprising multiple feature extraction branches to obtain a workpiece detection model based on a workpiece image group obtained through the data augmentation; a converting module, configured to convert the workpiece detection model into a lightweight workpiece detection model; and a detecting module, configured to perform detection on workpieces with the same shape and at least one different dimension as the original workpiece based on the lightweight workpiece detection model.

[0023] In a third aspect, an electronic device is provided. The electronic device comprising a processor and a memory, wherein an application program executable by the processor is stored in the memory for causing the processor to execute a method for detecting workpiece as described in any of the above.

[0024] In a fourth aspect, a computer-readable medium comprising computer-readable instructions stored thereon is provided, wherein the computer-readable instructions, when executed by a processor, implement a method for detecting workpiece as described in any of the above.

[0025] In a fifth aspect, a computer program product comprising a computer program, when the computer program is executed by a processor for executing a method for detecting workpiece as described in any of the above.BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to make technical solutions of examples of the present disclosure clearer, accompanying drawings to be used in description of the examples will be simply introduced hereinafter. Obviously, the accompanying drawings to be described hereinafter are only some examples of the present disclosure. Those skilled in the art may obtain other drawings according to these accompanying drawings without creative labor. FIG.1 is an exemplary flowchart of a workpiece detection method according to an embodiment of the present invention. FIG.2A is a first schematic diagram of exemplary workpieces with the same shape and different lengths according to an embodiment of the present invention. FIG.2B is a second schematic diagram of exemplary workpieces with the same shape and different lengths according to an embodiment of the present invention. FIG.3 is an exemplary schematic diagram of performing data augmentation processing on a workpiece image according to an embodiment of the present invention. FIG.4A is a schematic diagram of the feature extraction layer in a meta transformer. FIG.4B is a schematic diagram of feature extraction layer participating in training according to an embodiment of the present invention. FIG.4C is a schematic diagram of feature extraction layer after lightweight according to an embodiment of the present invention. FIG.5 is a schematic diagram of lightweight workpiece detection model obtained through training according to an embodiment of the present invention. FIG.6 is an exemplary schematic diagram of identical transformation according to the embodiment of the present invention. FIG.7 is a comparison of the effect between embodiments of the present invention and the prior art. FIG.8 is an exemplary schematic diagram of an apparatus for detecting workpiece according to an embodiment of the present invention. FIG.9 is an exemplary structural diagram of an electronic device according to an embodiment of the present invention. List of reference numbers: reference numbersmeanings101∼104steps10,11,12,13workpieces of the same shape with different lengths14,15workpieces of the same shape with different lengths20original workpiece21,23first region24second region22symbol241subblock25new workpiece image31first batch normalization unit32pooling unit33dropout unit34,37adder35second batch normalization unit36convolutional unit80feature extraction layer during training process41first batch normalization unit42first adder43pooling unit44first 1x1 convolutional kernel45second adder46dropout unit47third adder48second batch normalization unit49fourth adder50convolutional unit51second 1x1 convolutional kernel52fifth adder90lightweight feature extraction layer61new first batch normalization unit62new dropout unit63new second batch normalization unit64new convolutional unit70meta converter architecture71model input72feature extraction layer73classification layer74workpiece detection model75feature extraction layer84,86original image of original workpiece83,85images of workpieces with the same shape and different lengths as the original workpiece92detection results of prior art93detection results of embodiments of present invention800apparatus for detecting workpiece801data augmentation module802training module803converting module804detecting module900electronic device901processor902memory DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme, and advantages of the invention clearer, the following examples are given to further explain the invention in detail.

[0028] In order to be concise and intuitive in description, the scheme of the invention is described below by describing several representative embodiments. Many details in the embodiments are only used to help understand the scheme of the invention. However, it is obvious that the technical scheme of the invention can be realized without being limited to these details. In order to avoid unnecessarily blurring the scheme of the invention, some embodiments are not described in detail, but only the framework is given. Hereinafter, "including" refers to "including but not limited to", "according to..." refers to "at least according to..., but not limited to...". Due to the language habits of Chinese, when the number of an element is not specifically indicated below, it means that the element can be one or more, or can be understood as at least one.

[0029] In the industrial field, picking and placing workpieces for self-assembly, printed circuit boards (PCBs) for automatic testing, and electronic waste classification often rely on AI based workpiece detection. AI based workpiece detection requires a long execution time on the GPU and affects the speed of entire process, making it difficult to perform workpiece detection work on industrial edge devices such as SIMATIC 227E, SIMATIC 427E, or SIMATIC S7-1500 TM NPU. In addition, in the scenario of mixed sorting of workpieces, it is common to encounter new workpieces that have the same shape as the old one, but have a different length or width and another that is the same. Existing technologies often require the use of real workpiece images, including real new workpieces and real old workpieces, to frequently retrain AI models. However, frequent retraining of AI modules means more cost and time.

[0030] Embodiments of the present invention propose a lightweight workpiece detection scheme, which is convenient for implementation at edge devices. Moreover, for new workpieces with the same shape as the old one, but with one different length or width and another identical, there is no need to frequently retrain the model, which can detect new workpieces faster.

[0031] FIG.1 is an exemplary flowchart of a workpiece detection method according to an embodiment of the present invention. As shown in FIG.1, the method comprises: Step 101: performing data augmentation on an original image of an original workpiece.

[0032] Workpiece refers to the processing object in the mechanical processing process, which can be a single part or a combination of several parts fixed together. The meaning of the original image is: compared to the new workpiece image obtained by changing size of workpiece, the size of original workpiece containing in the original image has not been changed. The original workpiece is the workpiece contained in the original image. Preferably, the original image is a real captured image of the original workpiece.

[0033] Here, performing data augmentation on the original image can include: (1) Based on the original image, generate a new workpiece image comprised in the workpiece image group, where the workpiece in the new workpiece image has the same shape as the original workpiece. Moreover, the workpiece in the new workpiece image has the same length as the original workpiece, while the workpiece in the new workpiece image has a different width from the original workpiece. (2) Based on the original image, generate a new workpiece image comprised in the workpiece image group, where the workpiece in the new workpiece image has the same shape as the original workpiece. Moreover, the workpiece in the new workpiece image has the same width as the original workpiece, while the workpiece in the new workpiece image has a different length from the original workpiece.

[0034] Preferably, the workpiece image group may also comprise the original image.

[0035] FIG.2A is a first schematic diagram of exemplary workpieces with the same shape and different lengths according to an embodiment of the present invention. In Figure 2A, workpiece 10, workpiece 11, workpiece 12, and workpiece 13 all have the same shape and width, and workpiece 10, workpiece 11, workpiece 12, and workpiece 13 each have different lengths. Figure 2B is a second schematic diagram of exemplary workpieces with the same shape and different lengths according to the embodiment of the present invention. In Figure 2B, workpiece 14, workpiece 15, and workpiece 16 all have the same shape and width, and workpiece 14, workpiece 15, and workpiece 16 each have different lengths.

[0036] In one embodiment, step 101 includes: Sub step (1): Divide the original workpiece into a first region with invariant features and a second region with variable features; Sub step (2): Change the length of the second region to generate a second region with changed length; Alternatively, change the width of the second region to generate a second region with changed width. Sub step (3): Combine the first region and the second region with changed length into a new workpiece image. Alternatively, combine the first region and the second region with changed width to create a new workpiece image.

[0037] For example, the first region can be implemented as a textured region, while the second region can be implemented as a texture less region.

[0038] Figure 3 is an exemplary schematic diagram of performing data enhancement processing on a workpiece image according to an embodiment of the present invention. The workpiece image 20 includes first regions 21, 23, and second region 24. In the first region 21, there is also a symbol 22 (for example, implemented as a workpiece Logo). The first regions 21 and 23 have a specific shape and are not altered in length. The second region 24 is a texture less region that can be changed in length. Firstly, divide the workpiece image 20 into first regions 21, 23, and second region 24. Next, divide the second region 24 evenly into several sub blocks 241, such as 5 sub blocks. Next, sub block 241 can be copied to increase the length of the second region 24, or sub block 241 can be deleted to reduce the length of the second region 24. Finally, recombine the second region 24 with the first regions 21 and 23 after the length change to obtain a new workpiece image. For example, adding a sub block 241 in the second region 24 (totaling 6 sub blocks 241), and then recombining the second region 24 containing 6 sub blocks 241 with the first regions 21 and 23 to obtain a new workpiece image 25.

[0039] In Figure 3, an exemplary example of increasing the length of the workpiece to achieve data augmentation is illustrated, and in fact, the length of the workpiece can also be reduced to achieve data augmentation. Optionally, the length of the workpiece can be kept constant and the width of the workpiece can be changed to achieve data augmentation.

[0040] Step 102: performing training on a neural network model comprising multiple feature extraction branches to obtain a workpiece detection model based on a workpiece image group obtained through the data augmentation.

[0041] Here, the workpiece image group may include: the workpiece image (original image) before data enhancement in step 101 and the new workpiece image generated based on data enhancement in step 101.

[0042] In one implementation, the process of obtaining a neural network model comprising multiple feature extraction branches specifically includes: obtaining a meta transformer; Replace the feature extraction layer of the meta transformer with multiple feature extraction branches, including at least one of the following: short cut to adder; 1x1 convolutional kernel in parallel with pooling units; 1x1 convolutional kernel in parallel with convolutional units; determining the replaced meta transformer as a neural network model.

[0043] Meta transformer is a new framework for multimodal learning, which is used to process and associate information from multiple modalities. Although there are inherent gaps between various data, the meta transformer utilizes a frozen encoder to extract advanced semantic features from input data in a shared tag space, without the need for paired multimodal training data. This framework consists of a unified data marker, a pattern sharing encoder, and task headers for various downstream tasks.

[0044] Figure 4A is a schematic diagram of the feature extraction layer in a meta converter. The feature extraction layer in the meta transformer includes: first batch normalization (BN) unit 31, pooling unit 32, drop out unit 33, second BN unit 35, convolution unit 36, adder 33 connected between drop out unit 33 and second BN unit 35, and adder 37 after convolution unit 36. Among them, the data input of pooling unit 32, data input of drop out unit 33, and data input of convolution unit 36 all require 2 storage spaces (2 x memory).

[0045] At present, there are few feature extraction branches in the feature extraction layer of the meta transformer, which makes it difficult to ensure detection effect. By replacing the feature extraction layer of the meta transformer with multiple feature extraction branches, the detection effect can be improved. Multiple feature extraction branches can be achieved through various methods, such as adding at least one of the following: short cut to adder; 1x1 convolutional kernel in parallel with pooling unit; 1x1 convolutional kernel in parallel with convolutional unit, and so on.

[0046] Figure 4B is a schematic diagram of the feature extraction layer participating in training according to an embodiment of the present invention. In Figure 4B, the feature extraction layer 80 includes: first BN unit 41, first adder 42, pooling unit 43, first 1x1 convolution core 44, second adder 45, drop out unit 46, third adder 47, second BN unit 48, fourth adder 49, convolution unit 50 (such as 3x3 convolution core), second 1x1 convolution core 51, and fifth adder 52. Moreover, the output of the first adder 42 has a short cut to the input of the second adder 45. The output of the second adder 45 has a short cut to the input of the third adder 47. The output of the third adder 47 has a short cut to the input of the fourth adder 49. The output of the fourth adder 49 has a short cut to the input of the fifth adder 52. The first 1x1 convolution kernel 44 is parallel to pooling unit 43, and the second 1x1 convolution kernel 51 is parallel to the convolution unit 50.

[0047] Compared to the feature extraction layer in the meta converter, the feature extraction layer 80 participating in the training has more feature extraction branches, thus having better feature extraction performance.

[0048] Based on the workpiece image group obtained after data augmentation (including the original workpiece image before data augmentation and the new workpiece images generated based on data augmentation), a meta transformer with a replaced feature extraction layer is trained to obtain a workpiece detection model suitable for detecting workpieces with the same shape and at least one different dimension as the original workpiece.

[0049] Step 103: converting the workpiece detection model into a lightweight workpiece detection model.

[0050] Here, based on identity transformation method, the workpiece detection model in step 102 is transformed into a lightweight workpiece detection model, which facilitates the deployment of workpiece detection model at edge devices.

[0051] FIG.4C is a schematic diagram of feature extraction layer after lightweight according to an embodiment of the present invention. As shown in Figure 4C, lightweight conversion can include: (1) Based on identical transformation method, the first BN unit 41, the first adder 42 connected to the first BN unit 41, and a short cut from the model input to the first adder 42 are converted into a new first BN unit 60. (2) Based on identical transformation method, pooling unit 43, the second adder 45 connected to pooling unit 43, 1x1 convolutional kernel 44 in parallel with pooling unit 43, and a short cut from the first adder 43 to the second adder 43 are converted into a new pooling unit 61. (3) Based on identical transformation method, drop out unit 46, third adder 47 connected to the drop out unit 46, and a short cut from the input of the drop out unit 46 to third adder 47 are converted into a new drop out unit 62. (4) Based on identical transformation method, second BN unit 48, fourth adder 49 connected to the second BN unit 48, and short cut from the input of the second BN unit 48 to fourth adder 49 are converted into a new second BN unit 63. (5) Based on identical transformation method, convolutional unit 50, fifth adder 52, 1x1 convolutional kernel 51 in parallel with convolutional unit 50, and short cut from the input of convolutional unit 50 to the fifth adder 52 are converted into a new convolutional unit 64.

[0052] FIG.6 is an exemplary schematic diagram of identical transformation according to the embodiment of the present invention. In Figure 6, pooling unit 43, second adder 45 connected to pooling unit 43, 1x1 convolutional kernel 44 in parallel with pooling unit 43, and short cut from the first adder 43 to the second adder 43 l are converted into a new pooling unit 61.

[0053] Connect the new first BN unit 60, the new pooling unit 61, the new drop out unit 62, the new second BN unit 63, and the new convolution unit 64 in sequence to form a feature detection layer 90 in the lightweight workpiece detection model. In the feature detection layer 90 of the lightweight workpiece detection model, one storage space (1 x memory) is required for data storage, thus reducing storage pressure.

[0054] FIG.5 is a schematic diagram of lightweight workpiece detection model obtained through training according to an embodiment of the present invention. The meta transformer architecture 70 includes model input 71, feature extraction layer 72, and classification layer 73. For example, the feature extraction layer 72 has a structure shown in Figure 4A.

[0055] Firstly, replace the feature extraction layer 72 in the meta transformer architecture 70 with the feature extraction layer 80 in Figure 4B. Then, using workpiece images in the workpiece image group, to train the meta transformer architecture 70 (where the feature extraction layer 72 is replaced by the feature extraction layer 80). The workpiece image group includes original workpiece image and new workpiece images, where new workpiece images have the same shape and different widths or lengths as the original workpiece. The feature extraction layer 75 in the trained meta converter architecture 74 is transformed into lightweight feature extraction layer 90 in Figure 4C through identity transformation, thereby obtaining a lightweight workpiece detection model. Then, the lightweight workpiece detection model can be deployed to industrial edge devices to detect mixed workpieces with the same shape and different lengths or widths.

[0056] Step 104: performing detection on workpieces with the same shape and at least one different dimension as the original workpiece based on the lightweight workpiece detection model.

[0057] Here, detecting workpieces can include identifying workpieces, classifying workpieces, and locating workpieces, among others.

[0058] In one embodiment, step 102 includes: training a neural network model based on a workpiece image group containing images with local labels of the first region and the second region therein, to obtain a workpiece detection model with the ability to detect the first and second regions. This method includes: deploying the lightweight workpiece detection model in an edge device; inputting a reference workpiece image comprising preset grasping point coordinates into the lightweight workpiece detection model to output center coordinates of the first region and center coordinates of the second region in the reference workpiece image; storing associatively the center coordinates of the first region in the reference workpiece image, the center coordinates of the second region in the reference workpiece image, and the preset gripping point coordinates; inputting an image of workpiece to be detected into the lightweight workpiece detection model to output center coordinates of the first region in the image of workpiece to be detected and center coordinates of the second region in the image of workpiece to be detected; determining gripping point coordinates in the image of workpiece to be detected based on the center coordinates of the first region in the reference workpiece image, the center coordinates of the second region in the reference workpiece image, the preset gripping point coordinates, the center coordinates of the first region in the image of workpiece to be detected, and the center coordinates of the second region in the image of workpiece to be detected.

[0059] For example, user calibrates gripping point in the reference workpiece image, assuming that the gripping point coordinates are(x b , y b ). Input the reference workpiece image into the lightweight workpiece detection model in edge device to detect center coordinates(x a , y a ) of the first region in the reference workpiece image and center coordinates (x c , y c ) of the second region in the reference workpiece image. Then, associatively store center coordinates of the first region(x a , y a ), center coordinates of the second region (x c , y c ), and coordinates of the grasping point (x b , y b ).

[0060] Next, input an image of workpiece to be detected (such as the workpiece in the image of the workpiece to be detected having the same shape and different length as the reference workpiece in the reference workpiece image) into the lightweight workpiece detection model. The lightweight workpiece detection model outputs center coordinates (x newa , y newa ) of the first region in the workpiece image to be detected and center coordinates (x newc , y newc ) of the second region in the workpiece image to be detected by the lightweight workpiece detection model.

[0061] Then, the coordinates (x newb , y newb )of the gripping point in the image of the workpiece to be detected can be calculated as: x newb = x newa + x b − x a x c − x b ∗ x newc 1 + x b − x a x c − x b ; y newb = y newa + y b − y a y c − y b ∗ y newc 1 + y b − y a y c − y b

[0062] Therefore, it is convenient and flexible to quickly obtain the gripping point coordinates of new workpieces, especially suitable for sorting scenarios of mixed workpieces with the same shape but different lengths or widths.

[0063] In one embodiment, step 102 includes: training the neural network model based on global label of workpiece image, local label of the first region, and local label of the second region to obtain the workpiece detection model. The workpiece detection model can detect the overall contour of the workpiece image, as well as the respective contours of the first and second regions in the workpiece image. Step 104 includes: inputting the tested image containing multiple workpieces with overlapping relationships into a lightweight workpiece detection model; Obtain the detection results of each workpiece from a lightweight workpiece detection model, where the detection results of each workpiece include their respective first region and second region. The overall contour to which the component belongs can be determined based on the overall contour with the maximum overlap among multiple overall contours determined by the component (including the first and second regions) and the global label.

[0064] For example, if three workpieces in the image have overlapping relationships, the workpiece detection model recognizes: overall contour 1 of the first workpiece, overall contour 2 of the second workpiece, and overall contour 3 of the third workpiece. The workpiece detection model also recognizes: first region 1, first region 2, and first region 3 in the three workpieces; second region 1, second region 2, and second region 3 in the three workpieces. If the overall contour that most overlaps with the first region 1 is overall contour 1, and the overall contour that most overlaps with the second region 2 is overall contour 1. Then determine that the first region 1 and the second region 2 are components of the overall contour 1. Similarly, determine the respective components of overall contour 2 and overall contour 3.

[0065] The workpiece detection model based on global and local label training can accurately detect multiple workpieces with overlapping relationships separately.

[0066] FIG.7 is a comparison of the effect between embodiments of the present invention and the prior art. A workpiece detection model in prior art is trained through original workpiece image 84 and original workpiece 86 image. Input original workpiece image 84, original workpiece image 86, and workpiece images 83 and 85 with the same shape and different lengths as original workpiece image into workpiece detection model in prior art, and obtain detection results 92. According to detection results 92, only original workpiece image 84 and the original workpiece image 86 can be recognized, but the workpiece images 83 and 85 cannot be recognized.

[0067] In embodiments of the present invention, data augmentation processing is performed on the original workpiece image 84 (or original workpiece image 86) to generate workpiece images 83, 85 with the same shape and different lengths as the original workpiece image. A workpiece detection model is trained through original workpiece image 84, original workpiece 86 image, and workpiece images 83, 85. Input original workpiece image 84, original workpiece image 86, and workpiece images 83 and 85 with the same shape and different lengths as the original workpiece image into the workpiece detection model of present invention to obtain detection results of embodiments of present invention. In the detection results, both original workpiece images 84,86 and workpiece images 83 and 85 can be recognized.

[0068] FIG.8 is an exemplary schematic diagram of an apparatus for detecting workpiece according to an embodiment of the present invention. As shown in FIG.8, apparatus 800 for detecting workpiece comprising: a data augmentation module 801, configured to perform data augmentation on an original image of an original workpiece; a training module 802, configured to perform training on a neural network model comprising multiple feature extraction branches to obtain a workpiece detection model based on a workpiece image group obtained through the data augmentation; a converting module 803, configured to convert the workpiece detection model into a lightweight workpiece detection model; and a detecting module 804, configured to perform detection on workpieces with the same shape and at least one different dimension as the original workpiece based on the lightweight workpiece detection model.

[0069] In one embodiment, data augmentation module 801, configured to generate a new workpiece image in the workpiece image group based on the original image, a workpiece in the new workpiece image has the same shape as the original workpiece; wherein the length of workpiece in the new workpiece image is the same as that of the original workpiece, the width of workpiece in the new workpiece image is different from that of the original workpiece; or wherein the width of workpiece in the new workpiece image is the same as that of the original workpiece, the length of workpiece in the new workpiece image is different from that of the original workpiece.

[0070] In one embodiment, data augmentation module 801, configured to divide the original workpiece in the original image into a first region with invariant features and a second region with variable features; change the length of the second region to generate a second region with changed length; change the width of the second region to generate a second region with changed width; combine the first region and the second region with changed length to form the new workpiece image; or combining the first region and the second region with changed width to form the new workpiece image.

[0071] In one embodiment, training module 802, configured to obtain a meta transformer; replace a feature extraction layer of the meta transformer with multiple feature extraction branches, wherein the multiple feature extraction branches comprise at least one of the following: shortcut to adder; 1x1 convolutional kernel in parallel with pooling unit; 1x1 convolutional kernel in parallel with convolutional unit; determine the replaced meta transformer as the neural network model.

[0072] In one embodiment, converting module 803, configured to convert a first batch normalization unit, a first adder connected to the first batch normalization unit, and a shortcut from model input to the first adder in the workpiece detection model into a new first batch normalization unit, based on an identical transformation method; convert a pooling unit, a second adder connected to the pooling unit, a 1x1 convolutional kernel in parallel with the pooling unit, and a shortcut from the first adder to the second adder in the workpiece detection model into a new pooling unit, based on an identical transformation method; convert a dropout unit, a third adder connected to the dropout unit, and a shortcut from the input of the dropout unit to the third adder in the workpiece detection model into a new dropout unit, based on an identical transformation method; convert a second batch normalization unit, a fourth adder connected to the second batch normalization unit, and a shortcut from the input of the second batch normalization unit to the fourth adder in the workpiece detection model into a new second batch normalization unit, based on an identical transformation method; convert a convolutional unit, a fifth adder, a 1x1 convolutional kernel in parallel with the convolutional unit, and a shortcut from the input of the convolutional unit to the fifth adder in the workpiece detection model into a new convolutional unit, based on an identical transformation method; connect the new first batch normalization unit, the new pooling unit, the new dropout unit, the new second batch normalization unit, and the new convolution unit in sequence to form a feature detection layer in the lightweight workpiece detection model.

[0073] In one embodiment, training module 802, configured to train the neural network model based on a workpiece image in the workpiece image group comprising a local label of the first region and a local label of the second region; detecting module 804, configured to deploy the lightweight workpiece detection model in an edge device; input a reference workpiece image comprising preset grasping point coordinates into the lightweight workpiece detection model to output center coordinates of the first region and center coordinates of the second region in the reference workpiece image; store associatively the center coordinates of the first region in the reference workpiece image, the center coordinates of the second region in the reference workpiece image, and the preset gripping point coordinates; input an image of workpiece to be detected into the lightweight workpiece detection model to output center coordinates of the first region in the image of workpiece to be detected and center coordinates of the second region in the image of workpiece to be detected; determine gripping point coordinates in the image of workpiece to be detected based on the center coordinates of the first region in the reference workpiece image, the center coordinates of the second region in the reference workpiece image, the preset gripping point coordinates, the center coordinates of the first region in the image of workpiece to be detected, and the center coordinates of the second region in the image of workpiece to be detected.

[0074] In one embodiment, training module 802, configured to train the neural network model based on a workpiece image in the workpiece image group comprising a local label of the first region, a local label of the second region and a global label of the workpiece image; detecting module 804, configured to input an image comprising multiple workpieces with overlapping relationships into the lightweight workpiece detection model; obtain detection results for each of the multiple workpieces from the lightweight workpiece detection model, wherein each detection result for each workpiece comprises first region and second region of corresponding workpiece.

[0075] The embodiment of the present invention also proposes an electronic device with a processor memory architecture. Figure 9 is an exemplary structural diagram of an electronic device with a processor memory architecture according to an embodiment of the present invention. As shown in Figure 9, electronic device 900 includes a processor 901, a memory 902, and a computer program stored on memory 902 that can run on processor 901. When the computer program is executed by processor 901, the workpiece detection method as described in either of the above methods is implemented. Among them, memory 902 can be implemented as various storage media such as electrically erasable programmable read-only memory (EEPROM), flash memory, programmable program read-only memory (PROM), etc. Processor 901 can be implemented to include one or more central processors or one or more field programmable gate arrays, wherein the field programmable gate array integrates one or more central processor cores. Specifically, the central processing unit or core can be implemented as a CPU, MCU, DSP, and so on.

[0076] It should be noted that not all steps and modules in the above processes and structural diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution sequence of each step is not fixed and can be adjusted as needed. The division of each module is only for the convenience of describing the functional division used. In actual implementation, a module can be divided into multiple modules, and the functions of multiple modules can also be implemented by the same module. These modules can be in the same device or different devices.

[0077] The hardware modules in each implementation can be implemented mechanically or electronically. For example, a hardware module can include specially designed permanent circuits or logic devices (such as dedicated processors, such as FPGA or ASIC) to complete specific operations. Hardware modules can also include programmable logic devices or circuits temporarily configured by software (such as general-purpose processors or other programmable processors) for performing specific operations. As for the specific use of mechanical methods, either dedicated permanent circuits or temporarily configured circuits (such as software configuration) to implement hardware modules, it can be determined based on cost and time considerations.

Claims

1. A method for detecting workpieces, comprising: performing (101) data augmentation on an original image of an original workpiece by generating a workpiece image group of new workpiece images, wherein each new workpiece image comprises a new workpiece with the same shape as the original workpiece but with either a different length or a different width; performing training (102) on a neural network model (70) comprising a feature extraction layer (80) with multiple feature extraction branches to obtain a workpiece detection model based on the workpiece image group obtained through the data augmentation; converting (103) the workpiece detection model (74) into a lightweight workpiece detection model by replacing multi-branch segments of a trained multi-branch feature extraction layer (75) of the workpiece detection model (70) by single-branch segments applying an identical transformation as the replaced multi-branch segments to convert the trained multi-branch feature extraction layer (75) into a lightweight feature extraction layer (90); and performing (104) detection on workpieces with the same shape as the original workpiece but either a different length or a different width than the original workpiece based on the lightweight workpiece detection model.

2. The method of claim 1, wherein generating a new workpiece image in the workpiece image group based on the original image comprises: dividing the original workpiece in the original image into a first region with invariant features and a second region with variable features; changing the length of the second region to generate a second region with changed length, or alternatively changing the width of the second region to generate a second region with changed width; combining the first region and the second region with changed length to form the new workpiece image; or alternatively combining the first region and the second region with changed width to form the new workpiece image.

3. The method of claim 1, comprising: obtaining a multilayer neural network model (70) comprising a feature extraction layer (72); replacing a feature extraction layer (72) of the multilayer neural network model by a multibranch feature extraction layer (80) with multiple feature extraction branches, wherein the multiple feature extraction branches comprise at least one of the following: shortcut to adder; 1x1 convolutional kernel in parallel with pooling unit; 1x1 convolutional kernel in parallel with convolutional unit; determining the replaced multilayer neural network model (70) as the neural network model.

4. The method of claim 3, wherein converting (103) the workpiece detection model into a lightweight workpiece detection model comprises: converting a first batch normalization unit, a first adder connected to the first batch normalization unit, and a shortcut from model input to the first adder in the workpiece detection model into a new first batch normalization unit, based on an identical transformation method which results in a unit applying an identical transformation; converting a pooling unit, a second adder connected to the pooling unit, a 1x1 convolutional kernel in parallel with the pooling unit, and a shortcut from the first adder to the second adder in the workpiece detection model into a new pooling unit, based on an identical transformation method which results in a unit applying an identical transformation; converting a dropout unit, a third adder connected to the dropout unit, and a shortcut from the input of the dropout unit to the third adder in the workpiece detection model into a new dropout unit, based on an identical transformation method which results in a unit applying an identical transformation; converting a second batch normalization unit, a fourth adder connected to the second batch normalization unit, and a shortcut from the input of the second batch normalization unit to the fourth adder in the workpiece detection model into a new second batch normalization unit, based on an identical transformation method which results in a unit applying an identical transformation; converting a convolutional unit, a fifth adder, a 1x1 convolutional kernel in parallel with the convolutional unit, and a shortcut from the input of the convolutional unit to the fifth adder in the workpiece detection model into a new convolutional unit, based on an identical transformation method which results in a unit applying an identical transformation; connecting the new first batch normalization unit, the new pooling unit, the new dropout unit, the new second batch normalization unit, and the new convolution unit in sequence to form a feature detection layer in the lightweight workpiece detection model.

5. The method of claim 2, wherein performing training (102) on a neural network model comprising: training the neural network model based on a workpiece image in the workpiece image group comprising a local label of the first region and a local label of the second region, wherein the method comprises: deploying the lightweight workpiece detection model in an edge device; inputting a reference workpiece image comprising preset grasping point coordinates into the lightweight workpiece detection model to output center coordinates of the first region and center coordinates of the second region in the reference workpiece image; storing associatively the center coordinates of the first region in the reference workpiece image, the center coordinates of the second region in the reference workpiece image, and the preset gripping point coordinates; inputting an image of workpiece to be detected into the lightweight workpiece detection model to output center coordinates of the first region in the image of workpiece to be detected and center coordinates of the second region in the image of workpiece to be detected; determining gripping point coordinates in the image of workpiece to be detected based on the center coordinates of the first region in the reference workpiece image, the center coordinates of the second region in the reference workpiece image, the preset gripping point coordinates, the center coordinates of the first region in the image of workpiece to be detected, and the center coordinates of the second region in the image of workpiece to be detected.

6. The method of claim 2, wherein performing training (102) on a neural network model comprising: training the neural network model based on a workpiece image in the workpiece image group comprising a local label of the first region, a local label of the second region and a global label of the workpiece image, wherein the method comprises: inputting an image comprising multiple workpieces with overlapping relationships into the lightweight workpiece detection model, wherein an overlapping relationship refers to the case where contours of multiple workpieces overlap partially in the image; obtaining detection results for each of the multiple workpieces from the lightweight workpiece detection model, wherein each detection result for each workpiece comprises first region and second region of corresponding workpiece.

7. An apparatus for detecting workpieces, comprising: a data augmentation module (801), configured to perform data augmentation on an original image of an original workpiece by generating a workpiece image group of new workpiece images, wherein each new workpiece image comprises a new workpiece with the same shape as the original workpiece but with either a different length or a different width; a training module (802), configured to perform training on a neural network model comprising a feature extraction layer (80) with multiple feature extraction branches to obtain a workpiece detection model based on the workpiece image group obtained through the data augmentation; a converting module (803), configured to convert the workpiece detection model into a lightweight workpiece detection model by replacing multi-branch segments of a trained multi-branch feature extraction layer (75) of the workpiece detection model (70) by single-branch segments applying an identical transformation as the replaced multi-branch segments to convert the trained multi-branch feature extraction layer (75) into a lightweight feature extraction layer (90); and a detecting module (804), configured to perform detection on workpieces with the same shape as the original workpiece but either a different length or a different width than the original workpiece based on the lightweight workpiece detection model.

8. An electronic device, comprising a processor (901) and a memory (902), wherein an application program executable by the processor (901) is stored in the memory (902) for causing the processor (901) to execute a method for detecting workpiece according to any one of claims 1-6.

9. A computer-readable medium comprising computer-readable instructions stored thereon, wherein the computer-readable instructions for executing a method for detecting workpiece according to any one of claims 1-6.

10. A computer program product comprising a computer program, upon the computer program is executed by a processor for executing a method for detecting workpiece according to any one of claims 1-6.