Model training method, chip tray detection method and device, equipment and product

Abstract

The invention relates to a model training method, a chip tray detection method and device, equipment and a product, and relates to the technical field of semiconductor testing. The method comprises the following steps: acquiring a pre-constructed initial model and a sample detection image; generating a second channel convolution feature by the initial model based on the original input feature of the sample detection image and the first channel convolution feature; performing maximum average attention feature extraction on the second channel convolution feature to obtain a first initial intermediate feature; performing enhanced receptive field feature extraction on the first initial intermediate feature to obtain a second initial intermediate feature; performing feature extraction processing on the first initial intermediate feature and the second initial intermediate feature to obtain a sample image detection result; and training based on a difference value between the sample image detection result and the tray area identifier to obtain a chip tray detection model. The present disclosure may provide a detection capability for a slit or groove of a low-texture black tray.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of semiconductor testing, and in particular, to a model training method, a chip tray detection method, a model training device, a chip tray detection device, an electronic device, and a computer program product. BACKGROUND

[0002] The dynamic random access memory (DRAM) post-package testing stage is a core link for guaranteeing product functions, performance, and reliability. In related schemes, the DRAM post-package taping process mainly relies on a combination of manual visual inspection and mechanical auxiliary detection.

[0003] The manual visual inspection method refers to checking the tray quantity (10 full trays + 1 empty tray / bundle) and the gap by the naked eye, and checking the tray quantity and the stacking gap. In order to partially make up for the defects of manual work, mechanical protection measures are introduced, for example, a scale line can be pasted on the side of the discharge workbench for comparison, and a tray stacking height limiting jig can be installed at the entry of the taping machine to realize the checking operation of the tray quantity and the stacking gap.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The purpose of the present disclosure is to provide a model training method, a chip tray detection method, a model training device, a chip tray detection device, an electronic device, a computer readable storage medium, and a computer program product, thereby at least partially overcoming the problems of low efficiency of manual inspection, inability of mechanical protection to detect chip tray quantity deviation, and tray gap defects caused by chip jumping in the tray in related schemes.

[0006] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, a model training method is provided, comprising: obtaining a pre-constructed initial model and a sample detection image, the sample detection image comprising a tray area identifier, the tray area identifier comprising a tray position identifier and a tray slot identifier; generating, by the initial model, a second channel convolution feature based on a first channel convolution feature and an original input feature of the sample detection image, the first channel convolution feature being obtained by performing residual separable channel convolution processing on the original input feature; performing maximum average attention feature extraction on the second channel convolution feature to obtain a first initial intermediate feature; performing enhanced receptive field feature extraction on the first initial intermediate feature to obtain a second initial intermediate feature; performing feature extraction processing on the first initial intermediate feature and the second initial intermediate feature to obtain a sample image detection result, the sample image detection result comprising a tray number detection result and a tray slot detection result; training the initial model based on a difference between the sample image detection result and the tray area identifier to obtain a chip tray detection model.

[0008] According to a second aspect of the present disclosure, a chip tray detection method is provided, comprising: obtaining a to-be-detected image corresponding to a to-be-detected chip tray; obtaining a pre-trained chip tray detection model, the chip tray detection model being trained based on a model training method; performing, by the chip tray detection model, tray detection processing on the to-be-detected image to obtain a model prediction result; performing number detection processing and slot detection processing on the model prediction result, respectively, to obtain a tray number detection result and a tray slot detection result; and determining a tray detection result based on the tray number detection result and the tray slot detection result.

[0009] According to a third aspect of the present disclosure, a model training apparatus is provided, comprising: a sample image acquisition module configured to acquire an initial model pre-constructed and a sample detection image, the sample detection image comprising a tray area identifier, the tray area identifier comprising a tray position identifier and a tray slot identifier; a channel convolution feature determination module configured to generate a second channel convolution feature based on an original input feature of the sample detection image and a first channel convolution feature by the initial model, the first channel convolution feature being obtained by performing residual separable channel convolution processing on the original input feature; a first intermediate feature extraction module configured to perform maximum average attention feature extraction on the second channel convolution feature to obtain a first initial intermediate feature; a second intermediate feature extraction module configured to perform enhanced receptive field feature extraction on the first initial intermediate feature to obtain a second initial intermediate feature; a sample detection result determination module configured to perform feature extraction processing on the first initial intermediate feature and the second initial intermediate feature to obtain a sample image detection result, the sample image detection result comprising a tray number detection result and a tray slot detection result; and a model training module configured to train the initial model based on a difference between the sample image detection result and the tray area identifier to obtain a chip tray detection model.

[0010] According to a fourth aspect of the present disclosure, a chip tray detection apparatus is provided, comprising: an image acquisition module configured to acquire a to-be-detected image corresponding to a to-be-detected chip tray; a model acquisition module configured to acquire a pre-trained chip tray detection model, the chip tray detection model being trained based on any one of the model training methods described above; a model prediction module configured to perform tray detection processing on the to-be-detected image by the chip tray detection model to obtain a model prediction result; a number and slot detection module configured to perform number detection processing and slot detection processing on the model prediction result respectively to obtain a tray number detection result and a tray slot detection result; and a detection result determination module configured to determine a tray detection result based on the tray number detection result and the tray slot detection result.

[0011] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement any one of the model training methods or chip tray detection methods described above.

[0012] According to a sixth aspect of the present disclosure, a computer readable storage medium is provided, having a computer program stored thereon, the computer program being executed by a processor to implement any one of the model training methods or chip tray detection methods described above.

[0013] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the model training method or the chip tray detection method according to any one of the above.

[0014] The technical solutions provided by the present disclosure can have the following beneficial effects: In the model training method in the exemplary embodiments of the present disclosure, on the one hand, the first channel convolution feature extracted is fused with the original input feature through the residual separable channel convolution, which can solve the problem of information loss between channels caused by the classic deep separable convolution module. On the other hand, the maximum average attention mechanism can combine the significant information and the average information of the chip tray, more accurately allocate the weight, and improve the detection capability of the gap or groove. On the other hand, the enhanced receptive field module is added in the model, which can combine the global background information, edge information and significant information in the image, and effectively distinguish the tray itself and the abnormal gap.

[0015] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0016] The drawings incorporated into the specification and forming part of the specification, show embodiments consistent with the present disclosure, and together with the specification, serve to explain the principles of the present disclosure. It is obvious that the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained from these drawings without creative labor for those skilled in the art. In the drawings: Figure 1 The schematic diagram shows the related scheme of pasting scale lines on the side of the discharge workbench and installing the tray stacking height limiting jig at the belt feeding port of the belt labeling machine.

[0017] Figure 2 The flowchart of the model training method according to the exemplary embodiments of the present disclosure is schematically shown.

[0018] Figure 3 The structure diagram of the chip tray detection model according to the exemplary embodiments of the present disclosure is schematically shown.

[0019] Figure 4 The network structure diagram of the residual separable channel feature extraction layer according to the exemplary embodiments of the present disclosure is schematically shown; wherein, Figure 4 (a) in FIG. 1 is the overall structure diagram of the residual separable channel feature extraction layer (RSC Layer); Figure 4 (b) in FIG. 1 is the network structure diagram of the RSC-Conv module in the RSC Layer; Figure 4(c) is a network structure diagram of the MA-CA module in the RSC Layer.

[0020] Figure 5 A network structure diagram of an SPPF-based enhanced receptive field module according to an example embodiment of the present disclosure is schematically shown.

[0021] Figure 6 A flowchart of a chip tray detection method according to an example embodiment of the present disclosure is schematically shown.

[0022] Figure 7 A hardware installation schematic diagram of a tape dispenser of a chip tray detection method according to an example embodiment of the present disclosure is shown.

[0023] Figure 8 A structure schematic diagram of a control system of a chip tray detection method according to an example embodiment of the present disclosure is shown.

[0024] Figure 9 A schematic diagram of chip tray placement in a tape dispenser according to an example embodiment of the present disclosure is shown.

[0025] Figure 10 A system workflow diagram of a chip tray detection method according to an example embodiment of the present disclosure is shown.

[0026] Figure 11 A schematic diagram of tray quantity detection and gap detection according to an example embodiment of the present disclosure is shown; wherein, Figure 11 (a) in is a schematic diagram of a tray detection frame obtained after target detection on a front view of a chip tray to be detected; Figure 11 (b) in is a schematic diagram of a tray detection frame obtained after target detection on a rear view of a chip tray to be detected; Figure 11 (c) in shows a schematic diagram of a chip tray without abnormal gap; Figure 11 (d) in shows a schematic diagram of a chip tray with abnormal gap.

[0027] Figure 12 A block diagram of a model training device according to an example embodiment of the present disclosure is schematically shown.

[0028] Figure 13 A block diagram of a chip tray detection device according to an example embodiment of the present disclosure is schematically shown.

[0029] Figure 14 A block diagram of an electronic device according to an example embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0030] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments, however, can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the description. Repetitive descriptions of like elements will be omitted for sake of brevity.

[0031] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the

[0032] The block diagrams in the drawings show functions and functionality as they can be implemented in software or computer programs. Implementing the functionality in a given way is not meant to limit implementation of the functions described. Rather, specific arrangements of functions and the like can be implemented in hardware, software or a combination of hardware and software.

[0033] For the DRAM post-process tape labeling process, the artificial visual inspection method has the problems of easy missed detection and low efficiency, such as a missed detection rate > 5% and a single inspection time > 30 seconds / bundle. In order to partially make up for the artificial defects, a mechanical auxiliary detection method is also proposed, which is referred to as Figure 1 , Figure 1 The related scheme is shown in the schematic diagram of the scale line pasted on the side of the discharge workbench and the tray stacking height limiting jig installed at the tape inlet of the tape labeling machine.

[0034] From Figure 1As can be seen, the automated visual inspection (AVI) device 110 delivers the chip tray to the discharge workbench 120 through the discharge port, the mechanical auxiliary detection mode compares the scale line 121 (such as a 10+1 tray scale line) pasted on the discharge workbench 120, and the tray stacking height limiting jig 131 (such as a 10+1 tray height limiting jig) installed at the entry of the bundling machine 130 is used to realize auxiliary detection, and finally the vacuum packaging device 140 is used to complete the packaging operation. However, this scheme can only meet the standard delivery mode of 10+1 / bundle, and for non-standard delivery, it relies on manual inspection, which often leads to the event that the operator skips the calibration step and directly delivers. Therefore, the DRAM post-bundling process is prone to problems such as high manual dependence, poor quality control, and rising hidden costs.

[0035] Based on this, in the example embodiment, first, a model training method is provided, which can be implemented by a server or a terminal device. The terminal described in the disclosure can include mobile terminals such as mobile phones, tablets, laptops, palmtop computers, personal digital assistants (PDA), and fixed terminals such as desktop computers. Figure 2 The schematic diagram of the model training method flow according to some embodiments of the disclosure is shown schematically. Referring to Figure 2 The model training method can include the following steps: Step S210, an initial model constructed in advance and a sample detection image are obtained, the sample detection image includes a tray area identifier, and the tray area identifier includes a tray position identifier and a tray gap identifier; Step S220, the initial model generates a second channel convolution feature based on the original input feature and the first channel convolution feature of the sample detection image, and the first channel convolution feature is obtained by performing residual separable channel convolution processing on the original input feature; Step S230, maximum average attention feature extraction is performed on the second channel convolution feature to obtain a first initial intermediate feature; Step S240, enhanced receptive field feature extraction is performed on the first initial intermediate feature to obtain a second initial intermediate feature; Step S250, feature extraction processing is performed on the first initial intermediate feature and the second initial intermediate feature to obtain a sample image detection result, and the sample image detection result includes a tray number detection result and a tray gap detection result; Step S260, based on the difference between the sample image detection result and the tray area identifier, the initial model is trained to obtain a chip tray detection model.

[0036] According to the model training method in the example embodiment, on the one hand, the first channel convolution feature extracted is fused with the original input feature through the residual separable channel convolution, which can solve the problem of information loss between channels caused by the classic deep separable convolution module. On the other hand, the maximum average attention mechanism can combine the significant information and average information of the chip tray, more accurately allocate the weight, and improve the detection capability of the gap or groove. On the other hand, the enhanced receptive field module is added to the model, which can combine the global background information, edge information and significant information in the image, and effectively distinguish the tray itself and the abnormal gap.

[0037] In the following, the model training method in the example embodiment will be further described.

[0038] In an example embodiment of the disclosure, in step S210, an initial model constructed in advance is acquired together with a sample detection image, and the sample detection image includes a tray area identifier, and the tray area identifier includes a tray position identifier and a tray gap identifier.

[0039] The sample detection image can be an image in which a region of interest (ROI) of a chip tray is pre-labeled. The tray area identifier can be a tray position identifier and an abnormal gap identifier in the sample detection image. The tray position identifier can be a detection box identifier of a position area of the chip tray in the sample detection image. The abnormal gap identifier can be a detection box identifier corresponding to the abnormal gap in the sample detection image.

[0040] After the initial sample image of the chip tray is acquired through the image acquisition device, image preprocessing is performed on the initial sample image to obtain a sample detection image. The image preprocessing can include, but is not limited to, template matching processing, perspective transformation processing, ROI cropping processing, etc. The initial model in the disclosure is a network model constructed on the basis of a You Only Look Once (YOLO) network structure, and the model is trained based on the labeled ROI image to obtain a chip detection model, which is used to realize the number and abnormal gap detection of the chip tray.

[0041] In an example embodiment of the disclosure, for step S220, the initial model generates a second channel convolution feature based on the original input feature and the first channel convolution feature of the sample detection image, including: performing initial feature extraction processing on the sample detection image through a backbone network to obtain the original input feature; performing residual separable channel convolution processing on the original input feature of the sample detection image to obtain the first channel convolution feature; and performing feature fusion processing on the original input feature and the first channel convolution feature to obtain the second channel convolution feature.

[0042] The original input features may be image features obtained by performing initial feature extraction on the sample detection image by the backbone network of the initial model. The first channel convolution features may be features extracted by the Residual Separable Channel–Convolution (RSC-Conv) module in the Residual Separable Channel Layer (RSC Layer) of the initial model. The second channel convolution features may be features obtained by fusion processing of the original input features with the first channel convolution features.

[0043] refer to Figure 3 , Figure 3 The structure of a chip tray detection model according to an exemplary embodiment of the present disclosure is schematically shown. Figure 3 The initial model in the

[15] includes a backbone network (backbone), a neck network (neck), and a detection head (head). The sample detection image is input into the initial model, and the backbone network of the initial model performs initial feature extraction on the sample detection image to obtain the original input features. Specifically, the backbone network may include a focus structure layer and a Conv2D_BN_SiLU layer.

[0044] The original input image, i.e., the sample detection image, is received through the Focus layer, and preliminary feature extraction and spatial downsampling are performed on it. Then, feature extraction, batch normalization, and nonlinear transformation are completed through the Conv2D_BN_SiLU feature extraction layer (abbreviated as Conv2D_BN_SiLU layer) to obtain the original input features. Among them, the Conv2D_BN_SiLU layer is composed of two-dimensional convolution (Conv2D) + batch normalization (BN) + Sigmoid-weighted linear unit (SiLU) activation function.

[0045] In order to reduce the amount of computation, the traditional YOLO network usually adopts the method of dimensionality reduction to reduce the number of channels and reduce the feature information. In order to solve the problem of information loss between channels in the classic depth-separable convolution structure, this paper adds the RSC Layer containing the RSC-Conv module to the initial model. Figure 3 , the original input features are transmitted to the RSC Layer in the backbone network for feature extraction.

[0046] refer to Figure 4 , Figure 4 The network structure diagram of the residual separable channel feature extraction layer according to an exemplary embodiment of the present disclosure is schematically shown.Figure 4 (a) is the overall structure diagram of the residual separable channel feature extraction layer (RSC Layer) in (a), the RSC Layer can include a specified number (such as n) of RSC-Conv modules, a 1x1 convolutional layer, and a Max-Avg-Coordinate Attention (MA-CA) module. In the RSC Layer, the original input feature X is taken as the input of the RSC Layer, and the original input feature is subjected to residual separable channel convolution processing by the feature extraction layer composed of n RSC-Conv modules to obtain the first channel convolution feature X1.

[0047] Then the first channel convolution feature X1 is added to the original input feature X, and the two features are subjected to feature fusion processing to obtain the second channel convolution feature. By performing feature fusion processing on the first channel convolution feature and the original input feature, the purpose of preventing gradient disappearance and enhancing feature layer information can be achieved.

[0048] In an exemplary embodiment of the present disclosure, residual separable channel convolution processing is performed on the original input feature of the sample detection image to obtain the first channel convolution feature, including: performing grouped convolution processing on the original input feature based on the channel dimension to obtain an initial convolution feature, the initial convolution feature lacking inter-channel continuous information; performing feature fusion processing on the initial convolution feature and the original input feature to obtain a first fusion feature, the first fusion feature including inter-channel continuous information; and performing point-by-point convolution processing on the first fusion feature to obtain the first channel convolution feature.

[0049] Reference is made to (b) of Figure 4 , Figure 4 The network structure of the RSC-Conv module is shown in (b). Referring to the residual network feature, the input of the RSC-Conv module can be an input feature layer X0. Taking the input feature layer X0 as an example, one branch of the RSC-Conv module can perform grouped convolution processing on the input feature layer X0 based on the channel dimension, as shown in Figure 4 (b) performs layer-by-layer convolution on the input feature layer X0 to obtain an initial convolution feature X1. Since the initial convolution feature X1 is obtained by layer-by-layer convolution processing based on the channel dimension, the initial convolution feature X1 lacks inter-channel continuous information.

[0050] To supplement the missing inter-channel continuous information, the RSC-Conv module adds the input feature layer X0 to the feature layer in which the inter-channel continuous information is missing, i.e., the initial convolutional feature X1, and performs feature fusion processing on the two feature layers. Since the original input feature has continuous information between channels without grouped convolution in the channel dimension, the original input feature and the feature obtained after grouped convolution in the channel are used to supplement the missing inter-channel continuous information by addition operation, and a feature layer X2 with inter-channel continuous information, i.e., a first fused feature X2, is obtained. The first fused feature X2 includes inter-channel continuous information.

[0051] Then, the information between the channels of the first fused feature X2 is integrated through convolution processing, and a feature layer containing more information is extracted as a first channel convolutional feature. Figure 4 In the RSC-Conv module, point-by-point convolution processing can be performed on the first fused feature X2 to obtain the first channel convolutional feature. The first channel convolutional feature obtained by feature extraction supplements the missing inter-channel continuous information.

[0052] Further, since the RSC Layer can include one or more RSC-Conv modules, when a RSC-Conv module completes feature extraction processing, the output feature of the RSC-Conv module can be used as the input of the next RSC-Conv module, and the output of the last RSC-Conv module can be used as the first channel convolutional feature.

[0053] In an exemplary embodiment of the present disclosure, the second channel convolutional feature is subjected to maximum average attention feature extraction to obtain a first initial intermediate feature, including: performing convolution processing on the second channel convolutional feature to obtain a first convolutional feature; performing average pooling processing and maximum pooling processing on the height dimension feature and the width dimension feature of the first convolutional feature, respectively, to obtain an average pooling feature and a maximum pooling feature; obtaining a pre-constructed weight configuration function, determining the feature weight corresponding to each of the average pooling feature and the maximum pooling feature based on the weight configuration function; and obtaining the first initial intermediate feature based on the feature weight and the first convolutional feature.

[0054] With reference to (a) in Figure 4 In (a) in the RSC Layer, after obtaining the second channel convolutional feature, convolution processing is performed on the second channel convolutional feature, such as 1x1 convolution kernel, to obtain a first convolutional feature X2. Then, the MA-CA module is used to perform average pooling processing and maximum pooling processing on the height dimension feature and the width dimension feature of the first convolutional feature, respectively, to obtain an average pooling feature and a maximum pooling feature.

[0055] Finally, the feature weight of each feature in the feature layer is calculated by the delta function, and then the feature weight is multiplied by the original feature layer in the MA-CA module to obtain an output feature layer with non-dimensional feature extraction; wherein the original feature layer in the MA-CA module can be the first convolutional feature, and the non-dimensional feature can be understood as indicating whether a certain pixel position has a feature. The MA-CA module can combine the salient information and the average information in the chip tray to more accurately allocate the weight and improve the detection capability of the gap or groove.

[0056] The feature layer extracted by the RSC Layer is subjected to the Conv2D_BN_SiLU layer and the Darknet network structure (such as Dark2) to obtain a feature layer subjected to deep feature extraction, spatial down-sampling and an increase in the number of channels, i.e. the first initial intermediate feature.

[0057] In an exemplary embodiment of the present disclosure, the height dimension feature and the width dimension feature of the first convolutional feature are respectively subjected to average pooling processing and maximum pooling processing to obtain average pooling features and maximum pooling features, comprising: the height dimension feature is respectively subjected to average pooling processing and maximum pooling processing to obtain average height pooling features and maximum height pooling features; the width dimension feature is respectively subjected to average pooling processing and maximum pooling processing to obtain average width pooling features and maximum width pooling features; the average height pooling features and the average width pooling features are subjected to feature fusion processing to obtain initial average pooling features; the maximum height pooling features and the maximum width pooling features are subjected to feature fusion processing to obtain initial maximum pooling features; the initial average pooling features are subjected to average pooling processing based on the width dimension to obtain the average pooling features; the initial maximum pooling features are subjected to maximum pooling processing based on the height dimension to obtain the maximum pooling features.

[0058] With reference to (c) in Figure 4 , (c) in Figure 4 shows a network structure diagram of the MA-CA module. The first convolutional feature is taken as the input layer of the MA-CA module, the input layer is split along the height and width directions to obtain height dimension features and width dimension features, and the above features are subjected to pooling processing in two pooling modes (i.e. average pooling and maximum pooling) to generate four feature layers. Specifically, the height dimension features are respectively subjected to average pooling processing and maximum pooling processing to obtain average height pooling features (also referred to as average pooling layer along height) and maximum height pooling features (also referred to as maximum pooling layer along height); the width dimension features are respectively subjected to average pooling processing (also referred to as average pooling layer along width) and maximum pooling processing to obtain average width pooling features and maximum width pooling features (also referred to as maximum pooling layer along width).

[0059] The average pooling layer along the height and the average pooling layer along the width are reconstructed by outer product to obtain an average pooling feature layer, i.e., an initial average pooling feature; the maximum pooling layer along the height and the maximum pooling layer along the width are reconstructed by outer product to obtain a maximum pooling feature layer, i.e., an initial maximum pooling feature. Then, the initial average pooling feature is subjected to average pooling processing along the width again to obtain an average pooling feature layer of CxHx1, i.e., an average pooling feature; the initial maximum pooling feature is subjected to maximum pooling processing along the height again to obtain a maximum pooling feature layer of Cx1xW, i.e., a maximum pooling feature. By adding the MA-CA module, the model can distinguish the average information (such as stacking gaps) of the tray itself and the salient information (such as abnormal gaps or taking and placing grooves) of the tray, and then combine the salient information and the average information to more accurately allocate the weight and improve the detection capability of the gap or the groove.

[0060] With reference to the foregoing description, the output feature layer extracted by the RSC Layer is subjected to feature extraction by the Conv2D_BN_SiLU layer and the Dark2 structure to obtain a first initial intermediate feature, and the first initial intermediate feature can be further subjected to feature extraction by the RSC Layer, the Conv2D_BN_SiLU layer and the Dark3 structure as the input of the neck network. Similarly, the Dark4 output can be obtained in combination with the foregoing feature extraction process of the model. Figure 3

[0061] In an exemplary embodiment of the present disclosure, for step S240, the first initial intermediate feature is subjected to enhanced receptive field feature extraction to obtain a second initial intermediate feature, including: the first initial intermediate feature is subjected to maximum pooling processing and average pooling processing respectively to obtain a first maximum pooling feature and a first average pooling feature; the first initial intermediate feature is subjected to maximum pooling processing based on a pre-configured maximum pooling layer number to obtain a second maximum pooling feature; and the first maximum pooling feature, the first average pooling feature and the second maximum pooling feature are subjected to feature fusion processing to obtain the second initial intermediate feature.

[0062] In order to overcome the problem that the traditional YOLO algorithm has a limited receptive field and cannot distinguish between global background information (the whole tray) and salient information (abnormal gaps) in a black low-texture tray, the present disclosure proposes to add an enhanced receptive field module (ERF-SPPF) in the model structure to effectively distinguish between the tray itself and abnormal gaps by combining global background information, edge information and salient information. After the Dark4 output is extracted by the RSC Layer, it is input into the enhanced receptive field module based on SPPF (Enhanced Receptive Field – Space Pooling Pyramid Fast, ERF-SPPF) for feature extraction. Referring to Figure 5 ,​Figure 5 A network structure diagram of an SPPF-based enhanced receptive field module according to an example embodiment of the present disclosure is schematically shown.

[0063] In the present disclosure, the features extracted via the RSC Layer can be taken as first initial intermediate features. From the first initial intermediate features, Figure 5 As can be seen, the ERF-SPPF module respectively performs maximum pooling processing and average pooling processing on the first initial intermediate features to obtain first maximum pooling features and first average pooling features. In addition, a specified number of maximum pooling feature extraction layers, i.e., the number of maximum pooling layers, are configured in another branch of the ERF-SPPF module, and the number of maximum pooling layers n of the input layer can be set according to the size of the input layer.

[0064] Based on the pre-configured number of maximum pooling layers, the first initial intermediate features are subjected to maximum pooling processing to obtain second maximum pooling features. After obtaining the second maximum pooling features, the first maximum pooling features, the first average pooling features and the second maximum pooling features are subjected to feature fusion processing, and the fused features are subjected to feature extraction processing by the Conv2D_BN_SiLU feature extraction layer to obtain second initial intermediate features. After passing through the RSC Layer, the ERF-SPPF module is used again, global average pooling layers and global maximum pooling layers are mainly used, some global background information and edge information are added on the basis of the ERF-SPPF module, the global background information and the edge information are used to better detect significant information, and then the Dark5 is used to obtain the output of the layer. The enhanced receptive field module can combine the global background information, the edge information and the significant information to achieve the detection effect of effectively distinguishing the tray itself and the abnormal gap.

[0065] For ease of distinction, the present disclosure can also take the features extracted via the RSC Layer and the ERF-SPPF module as second initial intermediate features; and then take the first initial intermediate features and the second initial intermediate features as inputs of a neck network. For example, the first initial intermediate features can be the Dark3 output and the Dark4 output, the second initial intermediate features can be the Dark5 output, and the Dark3 output, the Dark4 output and the Dark5 output are taken as inputs of the neck network.

[0066] In an example embodiment of the present disclosure, the first initial intermediate features and the second initial intermediate features are subjected to feature extraction processing to obtain a sample image detection result, including: the first initial intermediate features and the second initial intermediate features are subjected to feature fusion processing to obtain second fused features; the second fused features are subjected to residual separable channel convolution processing and maximum average attention feature extraction by the residual separable channel extraction layer to obtain neck output features; and the neck output features are input to a detection head, and the sample image detection result is output by the detection head.

[0067] With reference back to Figure 3 , the neck network of the present disclosure is basically the same as the original structure of YOLO, mainly adding RSCLayer, i.e., residual separable channel extraction layer, for dimensionless feature extraction, which strengthens the extraction ability of channel information and significant information features. The Dark3 output, Dark4 output and Dark5 output are the three inputs of the neck network, and the Dark5 output is subjected to feature extraction by the Conv2D_BN_SiLU feature extraction layer and the UpSample layer, and then the output of the UpSample layer is subjected to feature fusion processing with the Dark4 output to obtain a second fusion feature.

[0068] The second fusion feature is input to the RSC Layer, which includes the RSC-Conv module and the MA-CA module, so that the second fusion feature can be subjected to residual separable channel convolution processing and maximum average attention feature extraction by the RSC Layer to obtain the output of the RSC Layer.

[0069] Then, the output of the RSC Layer is subjected to feature extraction processing by the Conv2D_BN_SiLU feature extraction layer and the UpSample layer, and the extracted features are subjected to feature fusion processing with the Dark3 output to obtain a fusion feature, and then the fusion feature is subjected to feature extraction processing by the RSC Layer as a neck output feature. With reference back to Figure 3 , two other neck output features are obtained based on the feature extraction layer of the neck network, such as downsampling the first neck output feature, and performing feature fusion processing between the downsampled feature and the feature output by the Conv2D_BN_SiLU layer, and then processing the fusion feature by the RSC Layer to obtain a second neck output feature; similarly, a third neck output feature is obtained.

[0070] The three feature layers output by the neck are input to the detection head, which is basically the same as the original structure of YOLO, and three detection heads (YOLO-HEAD) are used to obtain prediction results based on the neck output features as sample image detection results. The sample image detection results can include tray number detection results and tray gap detection results, and the tray number detection results are mainly obtained according to the number of tray detection boxes output by the detection head, and the tray number detection results include the detection box of the position of each chip tray, and can also include the tray statistical number calculated according to the tray detection box; the tray gap detection result can be determined by the abnormal gap detection box output by the detection head, and then the sample image detection result is used as the data basis for model training.

[0071] After obtaining the sample image detection result, the difference between the sample image detection result and the tray area label can be calculated, such as comparing the tray position label with the chip tray detection box output by the model, and comparing the tray gap label with the abnormal gap detection box, constructing a loss function based on the above differences, and training the initial model based on the constructed loss function until the model training end condition is met, to obtain the chip tray detection model.

[0072] The model parameters and the like of the chip tray detection model are stored in a pre-training weight file, so that the microcomputer in the tape labeling machine can call the pre-training weight file to perform tray number detection and gap anomaly detection and the like on the chip tray to be detected in the subsequent operation.

[0073] In summary, the model training method of the present disclosure, on the one hand, the first channel convolution feature extracted is fused with the original input feature through the residual separable channel convolution, which can solve the problem of information loss between channels caused by the classic deep separable convolution module. On the other hand, the maximum average attention mechanism can combine the significant information and the average information of the chip tray, more accurately allocate the weight, and improve the detection ability of the gap or groove. On the other hand, the enhanced receptive field module is added to the model, which can combine the global background information, edge information and significant information in the image, effectively distinguish the tray itself and the abnormal gap. On the other hand, through the above-mentioned model training method, a network model for chip tray number detection and abnormal gap detection is trained.

[0074] In addition, the present disclosure also provides a chip tray detection method, which can use a server to implement the chip tray detection method of the present disclosure, or a terminal device to implement the method described in the present disclosure. The terminal described in the present disclosure can include mobile terminals such as mobile phones, tablet computers, notebook computers, palmtop computers, personal digital assistants (Personal Digital Assistant, PDA), and fixed terminals such as desktop computers. Figure 6 The schematic diagram of the chip tray detection method flow according to some embodiments of the present disclosure is schematically shown. Referring to Figure 6 The chip tray detection method can include the following steps: Step S610, obtaining the image to be detected corresponding to the chip tray to be detected; Step S620, obtaining a pre-trained chip tray detection model, the chip tray detection model being trained based on a model training method; Step S630, performing tray detection processing on the image to be detected by the chip tray detection model to obtain a model prediction result; Step S640, performing number detection processing and gap detection processing on the model prediction result respectively to obtain a tray number detection result and a tray gap detection result; Step S650, based on the tray quantity detection result and the tray gap detection result, determine the tray detection result.

[0075] According to the chip tray detection method in the example embodiment, on the one hand, the number of stacked layers of the chip tray is accurately identified by AI vision, which can realize accurate counting. On the other hand, through the gap detection process, the tray stacking gap is identified, and the sensitivity can reach 0.1mm level, which can intercept the risk of chip solder ball being pressed. On the other hand, through automatic visual detection, the detection time can be greatly reduced, the detection efficiency can be improved, and the production line beat delay can be avoided.

[0076] In the following, the chip tray detection method in the example embodiment will be further described.

[0077] The delivery process of the tray can include steps such as laser marking (Laser Marking), baking (Baking), automated visual inspection (Automated Visual Inspection, AVI), bundling (Bundler), packing (Packing), etc. The present disclosure focuses on the chip tray bundling process after the AVI site, and proposes an automatic detection scheme combining artificial intelligence (Artificial Intelligence, AI) visual recognition and Internet of Things (Internet of Things, IoT) technology. Through high-precision image acquisition and real-time analysis of the chip tray, the tray quantity verification, 0.1mm level gap identification and manufacturing execution system (Manufacturing Execution System, MES) closed-loop control are realized.

[0078] In an example embodiment of the present disclosure, for step S610, obtaining the to-be-detected image corresponding to the to-be-detected chip tray includes: obtaining a product batch number collected by an image collection device, performing batch number matching processing on the product batch number to obtain a batch number matching result; when the batch number matching result is matching success, collecting an initial image through the image collection device; performing template matching processing on the initial image to obtain a template matching result; when the template matching result is that the initial image is a chip tray image, performing image correction processing on the initial image to obtain a corrected image; performing region of interest cropping processing on the corrected image to obtain the to-be-detected image.

[0079] The image acquisition device may be a device for capturing images of a chip tray. The product batch number may be the product batch number corresponding to the chip tray to be inspected before the bundling process. The batch number matching result may be the result obtained by performing consistency matching processing on the product batch number of the chip tray to be inspected and the reference product batch number obtained from the MES. The initial image may be an image of multiple views of the chip tray to be inspected, such as front and back views, captured by the image acquisition device. The template matching result may be a detection result determining whether the initial detection image is an image of the chip tray. The chip tray image may be an image containing the chip tray. The image correction processing may be a process of correcting the initial image to a front-facing image. The corrected image may be a front-facing image of the chip tray to be inspected obtained after the image correction processing. The image to be inspected may be an image obtained by cropping the corrected image with a region of interest.

[0080] refer to Figure 7 , Figure 7 A schematic diagram showing the hardware installation of a tape taping machine according to a chip tray detection method according to an exemplary embodiment of the present disclosure is shown. Figure 7 In the embodiment, image acquisition devices, such as high-resolution industrial cameras or high-resolution cameras, are installed on the top and both sides of the strapping machine 700, including a first camera 710 installed on the top, a second camera 720 and a third camera 730 installed on both sides, which are used to collect images of the product batch number and multiple views of the chip tray before strapping in real time, complete stereo calibration and image correction, and ensure coverage of the entire area of ​​the tray, such as Figure 7 The image acquisition device can capture images of the chip tray to be inspected from the front and back views. In addition, the specific installation position of the image acquisition device can be adjusted according to the structure of the tape machine and the placement position of the chip tray to achieve image acquisition.

[0081] In addition, the tape machine is also equipped with an AI detection result display panel 740, and a mini computer is installed inside the tape machine. The mini computer is connected to the tape triggering circuit and sensor of the tape machine through a relay, and controls the operation of the tape machine. Figure 8 , Figure 8 A schematic structural diagram of a control system of a chip tray detection method according to an exemplary embodiment of the present disclosure is shown. Figure 8 The edge computing unit of the mini computer 810 integrates a trigger signal linked to the strapping machine's tape output sensor 820 to ensure that image acquisition and strapping actions are completed in sequence; wherein, the tape output sensor 820 is installed at the tape inlet of the strapping machine 700, and the mini computer 810 can transmit signals to the tape output sensor 820 through the relay 830; the mini computer 810 can also interact with the MES system 850 through the server 840.

[0082] After the hardware installation is completed, the above hardware device can be used to detect the chip tray. Reference Figure 9 , Figure 9 A schematic diagram of chip tray placement in a taping machine according to an example embodiment of the present disclosure is shown. A chip tray 910 is placed in a taping machine 700, and then the chip tray detection method of the present disclosure is used to detect the number of trays and the gap of the tray, to obtain the corresponding tray detection result.

[0083] Specifically, reference Figure 10 , Figure 10 A system workflow diagram of a chip tray detection method according to an example embodiment of the present disclosure is shown. After the program initialization in the host computer is completed and the taping machine is placed by the object to be detected, the image acquisition device (such as the first camera 710) in the taping machine 700 will acquire the image of the object to be detected that has been placed, and obtain the product batch number of the object to be detected.

[0084] After the image acquisition device acquires the product batch number, it obtains the reference product batch number from the MES database, which stores the product batch number of the chip product to be banded and shipped out of the warehouse in the current period. The product batch number is matched with the reference product batch number, the correctness of the product batch number acquired by the image acquisition device is judged, and the batch number matching result is obtained. If the physical disk identifier (PDID) is not queried from the MES, it is considered that the batch number matching result is a matching failure, and the subsequent operation will not be performed. When the batch number matching result is a matching success, the initial image is acquired by the image acquisition device (such as the second camera 720 and the third camera 730), and the initial image can be the front and rear views of the chip tray image.

[0085] The acquired initial image is preprocessed, and the image preprocessing specifically includes the following steps: template matching processing is performed on the initial image to determine whether the object to be detected placed in the taping machine is a chip tray (target Tray), and a template matching result is obtained. The Scale-invariant feature transform (SIFT) algorithm is used to extract the four corners of the target Tray if the target Tray exists, and the subsequent process is not continued if the target Tray does not exist. The template matching operation can realize the tray target detection.

[0086] When the template matching result is that the initial image is a chip tray image, the front view and the rear view of the chip tray are subjected to image correction processing to obtain a corrected image, and the problem of improper placement of the chip tray is eliminated. For example, the image is corrected by perspective transformation, and the image is corrected to a front view based on the detected corner points to calculate a homography matrix.

[0087] Then, the corrected image is subjected to ROI clipping processing to clip the to-be-detected subject tray, prevent environmental factors such as reflection from affecting, and take the obtained ROI image as the to-be-detected image, that is, as the input of the chip tray detection model. By performing the above preprocessing operation on the collected initial image, the perspective distortion caused by non-standard placement of the tray can be eliminated, and an ROI image with a front view and a rear view is obtained, which is beneficial to the model to perform quantity and gap detection processing.

[0088] In an exemplary embodiment of the present disclosure, for step S630, the tray detection processing of the to-be-detected image is performed by the chip tray detection model to obtain a model prediction result, including: performing residual separable channel convolution processing on the original model input feature of the to-be-detected image by the backbone network to obtain a first image channel convolution feature; generating a second image channel convolution feature based on the original model input feature and the first image channel convolution feature; performing maximum average attention feature extraction on the second image channel convolution feature to obtain a first intermediate feature; performing enhanced receptive field feature extraction on the first intermediate feature to obtain a second intermediate feature; performing feature extraction processing on the first intermediate feature and the second intermediate feature by the neck network to obtain a neck output image feature; inputting the neck output image feature into the detection head, and outputting the model prediction result by the detection head.

[0089] The original model input feature can be an image feature obtained by performing feature extraction on the to-be-detected image after inputting the to-be-detected image into the chip tray detection model. The model prediction result can be an image prediction result output by the chip tray detection model, and the model prediction result can include multiple detection boxes, including a tray position detection box and a tray gap detection box.

[0090] After obtaining the to-be-detected image, the to-be-detected image is input into the chip tray detection model, which can include a backbone network, a neck network, and a detection head. The backbone network includes an RSC Layer, and the RSC Layer includes an RSC-Conv module and an MA-CA module. The original model input feature of the to-be-detected image is subjected to residual separable channel convolution processing by the RSC-Conv module in the RSC Layer to obtain a first image channel convolution feature; and the original model input feature and the first image channel convolution feature are subjected to feature fusion processing to generate a second image channel convolution feature.

[0091] The MA-CA module then performs maximum average attention feature extraction on the convolutional features of the second image channel to obtain the first intermediate feature. This first intermediate feature is then input into the ERF-SPPF module, which performs enhanced receptive field feature extraction on the first intermediate feature to obtain the second intermediate feature. Both the first and second intermediate features serve as the output of the backbone network and serve as the input of the neck network, which performs feature extraction on the first and second intermediate features to obtain the neck output image feature. This neck output image feature is then input into the detection head, which outputs the model prediction result. The model prediction result can include the tray annotation box and the gap annotation box in the image, which serve as the data basis for subsequent chip tray quantity detection and gap detection.

[0092] The processing process of the chip tray detection model outputting the model prediction result in this embodiment is the same as the processing process of the initial model performing feature extraction on the sample detection image to obtain the sample image detection result, and this disclosure will not elaborate on this.

[0093] In step S640 , quantity detection processing and gap detection processing are performed on the model prediction results to obtain pallet quantity detection results and pallet gap detection results.

[0094] After obtaining the model prediction results, the model prediction results are processed by quantity detection and gap detection respectively. Figure 11 , Figure 11 A schematic diagram illustrating tray quantity and gap detection according to an exemplary embodiment of the present disclosure is shown. Quantity detection includes performing target detection on the pick-and-place grooves, counting the target number of the pick-and-place grooves, and verifying and comparing the statistical results with information read from the MES to obtain a chip tray quantity statistical result.

[0095] In this embodiment, the image acquisition device obtains two views of the tray of the chip to be tested as the images to be tested, and the placement grooves for the chip to be tested include a groove on each of the left and right sides of the tray in the front and rear views, which facilitates removing or placing the tray. Figure 11 (a) and Figure 11 (b) in the figure shows the model prediction results for the front and back views of the chip tray to be inspected, respectively, which is a schematic diagram of the tray detection frame obtained after target detection. After performing target detection on the front and back views of the access groove, the predicted results for the tray detection frame are obtained, and the number of target access grooves is calculated based on the model prediction results. After obtaining the prediction results for the left and right grooves in the front and back views, that is, the number of detection frames, the final number of targets is determined based on the number of detection frames. For example, the mode of these four number statistics is used as the final number result, that is, the number of targets.

[0096] In addition, the target detection of the abnormal gap is used to detect whether the chip tray to be detected has an abnormal gap, to determine whether the chip has jumped, and if the abnormal gap exists, the abnormal gap is marked as the tray gap detection result. For example, Figure 11 (c) and Figure 11 (d) in FIG. 2 respectively show the schematic diagrams of the two cases of the chip tray without abnormal gap and the chip tray with abnormal gap.

[0097] In an exemplary embodiment of the present disclosure, when the tray detection result is that the tray state is normal, the chip tray to be detected is subjected to a banding operation; when the tray detection result is that the tray state is abnormal, the banding machine operation is interrupted, the chip tray to be detected is determined as an abnormal chip tray, and an alarm signal is sent; and the abnormal chip tray is pushed to a product management system.

[0098] The tray state normal means that the tray quantity of the chip tray is consistent with the tray shipment quantity in the MES system and there is no tray gap. The abnormal chip tray can be a chip tray with at least one abnormal state such as the tray quantity inconsistent with the tray shipment quantity in the MES system or the existence of the tray gap. The product management system can be a control system for bundling processing or abnormal processing of the detected chip tray.

[0099] After obtaining the tray detection result, the batch information is interactively verified with the MES system through the IoT gateway, specifically including preset parameter verification, such as 10 trays + 1 top cover / bundle quantity comparison; then the gap detection result is compared with the standard threshold, and the comparison result is stored in the MES system in real time. If the tray state is normal, the banding work can be performed. If the tray state is abnormal, such as information verification inconsistency or detection abnormality (quantity inconsistency or tray stacking gap exceeding the standard), an alarm is immediately reminded and the sensor is interrupted to send an opening signal to interrupt the banding machine work, the batch is marked as a problem product, and is pushed to the MES production management system. Through the above steps, the present disclosure realizes an automatic detection system combining AI visual recognition and IoT technology, realizes tray quantity verification, 0.1 millimeter level gap identification and MES system closed loop control through high precision image acquisition and real time analysis.

[0100] It should be noted that the terms "first", "second", etc. used in the present disclosure are only used to distinguish different channel convolution features, initial intermediate features, convolution features and maximum pooling features, and should not impose any limitation on the present disclosure.

[0101] To sum up, the chip tray detection method of the present disclosure, on the one hand, can realize accurate counting through AI vision accurate identification of the stacking number of chip trays. On the other hand, through the gap detection process, the stacking gap of the tray is identified, and the sensitivity can reach the level of 0.1 mm, which can intercept the risk of chip solder ball being pressed. On the other hand, through automatic visual detection, the detection time can be greatly reduced from the original 30s / time to within 1s, and the detection efficiency can be improved by 99% to avoid production line beat delay. On the other hand, the DRAM post-process can realize full automation closed-loop management and control, reduce manual intervention, and reduce labor cost.

[0102] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired result. In addition or alternatively, some steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps, etc.

[0103] Next, the present disclosure also provides a model training device. Referring to Figure 12 The model training device 1200 can include a sample image acquisition module 1210, a channel convolution feature determination module 1220, a first intermediate feature extraction module 1230, a second intermediate feature extraction module 1240, a sample detection result determination module 1250, and a model training module 1260.

[0104] Specifically, the sample image acquisition module 1210 is configured to acquire a pre-constructed initial model and a sample detection image, the sample detection image including a tray area identifier, the tray area identifier including a tray position identifier and a tray gap identifier; the channel convolution feature determination module 1220 is configured to generate a second channel convolution feature based on the original input feature and the first channel convolution feature of the sample detection image from the initial model, the first channel convolution feature being obtained by residual separable channel convolution processing on the original input feature; the first intermediate feature extraction module 1230 is configured to extract a first initial intermediate feature by maximum average attention feature extraction on the second channel convolution feature; the second intermediate feature extraction module 1240 is configured to extract a second initial intermediate feature by enhanced receptive field feature extraction on the first initial intermediate feature; the sample detection result determination module 1250 is configured to obtain a sample image detection result by feature extraction processing on the first initial intermediate feature and the second initial intermediate feature, the sample image detection result including a tray quantity detection result and a tray gap detection result; and the model training module 1260 is configured to train the initial model based on the difference between the sample image detection result and the tray area identifier to obtain a chip tray detection model.

[0105] In an example implementation of the present disclosure, the initial model comprises a backbone network, the channel convolution feature determination module 1220 comprises a channel convolution feature determination unit configured to: perform initial feature extraction processing on the sample detection image by the backbone network to obtain original input features; perform residual separable channel convolution processing on the original input features of the sample detection image to obtain first channel convolution features; and perform feature fusion processing on the original input features and the first channel convolution features to obtain second channel convolution features.

[0106] In an example implementation of the present disclosure, the channel convolution feature determination unit comprises a channel convolution feature determination subunit configured to: perform grouped convolution processing on the original input features based on a channel dimension to obtain initial convolution features, the initial convolution features lacking inter-channel continuous information; perform feature fusion processing on the initial convolution features and the original input features to obtain first fusion features, the first fusion features comprising inter-channel continuous information; and perform point-by-point convolution processing on the first fusion features to obtain the first channel convolution features.

[0107] In an example implementation of the present disclosure, the first intermediate feature extraction module 1230 comprises a first intermediate feature extraction unit configured to: perform convolution processing on the second channel convolution features to obtain first convolution features; perform average pooling processing and maximum pooling processing on height dimension features and width dimension features of the first convolution features, respectively, to obtain average pooling features and maximum pooling features; obtain a pre-constructed weight configuration function, determine respective feature weights of each of the average pooling features and the maximum pooling features based on the weight configuration function; and obtain first initial intermediate features according to the feature weights and the first convolution features.

[0108] In an example implementation of the present disclosure, the first intermediate feature extraction unit comprises a feature extraction subunit configured to: perform average pooling processing and maximum pooling processing on the height dimension features to obtain average height pooling features and maximum height pooling features; perform average pooling processing and maximum pooling processing on the width dimension features to obtain average width pooling features and maximum width pooling features; perform feature fusion processing on the average height pooling features and the average width pooling features to obtain initial average pooling features; perform feature fusion processing on the maximum height pooling features and the maximum width pooling features to obtain initial maximum pooling features; perform average pooling processing on the initial average pooling features based on the width dimension to obtain the average pooling features; and perform maximum pooling processing on the initial maximum pooling features based on the height dimension to obtain the maximum pooling features.

[0109] In an example embodiment of the present disclosure, the second intermediate feature extraction module 1240 includes a second intermediate feature extraction unit configured to: perform maximum pooling processing and average pooling processing on the first initial intermediate feature respectively to obtain a first maximum pooled feature and a first average pooled feature; perform maximum pooling processing on the first initial intermediate feature based on a preconfigured maximum pooling layer number to obtain a second maximum pooled feature; and perform feature fusion processing on the first maximum pooled feature, the first average pooled feature, and the second maximum pooled feature to obtain a second initial intermediate feature.

[0110] In an example embodiment of the present disclosure, the initial model includes a neck network and a detection head, the neck network includes a residual separable channel extraction layer, and the sample detection result determination module 1250 includes a sample detection result determination unit configured to: perform feature fusion processing on the first initial intermediate feature and the second initial intermediate feature to obtain a second fusion feature; perform residual separable channel convolution processing and maximum average attention feature extraction on the second fusion feature by the residual separable channel extraction layer to obtain a neck output feature; and input the neck output feature into the detection head to output a sample image detection result by the detection head.

[0111] In addition, the present disclosure also provides a chip tray detection device. Referring to Figure 13 The chip tray detection device can include an image acquisition module 1310, a model acquisition module 1320, a model prediction module 1330, a quantity and gap detection module 1340, and a detection result determination module 1350.

[0112] Specifically, the image acquisition module 1310 is configured to acquire a to-be-detected image corresponding to a to-be-detected chip tray; the model acquisition module 1320 is configured to acquire a pre-trained chip tray detection model, which is trained based on a model training method; the model prediction module 1330 is configured to perform tray detection processing on the to-be-detected image by the chip tray detection model to obtain a model prediction result; the quantity and gap detection module 1340 is configured to perform quantity detection processing and gap detection processing on the model prediction result respectively to obtain a tray quantity detection result and a tray gap detection result; and the detection result determination module 1350 is configured to determine a tray detection result based on the tray quantity detection result and the tray gap detection result.

[0113] In an example embodiment of the present disclosure, the image acquisition module 1310 comprises an image acquisition unit configured to: acquire a product batch number captured by an image capturing device, perform batch number matching processing on the product batch number to obtain a batch number matching result; when the batch number matching result is a matching success, capture an initial image by the image capturing device; perform template matching processing on the initial image to obtain a template matching result; when the template matching result is that the initial image is a chip tray image, perform image correction processing on the initial image to obtain a corrected image; and perform region of interest cropping processing on the corrected image to obtain a to-be-detected image.

[0114] In an example embodiment of the present disclosure, the chip tray detection model comprises a backbone network, a neck network and a detection head; the model prediction module 1330 comprises a model prediction unit configured to: perform residual separable channel convolution processing on original model input features of the to-be-detected image by the backbone network to obtain first image channel convolution features; generate second image channel convolution features based on the original model input features and the first image channel convolution features; perform maximum average attention feature extraction on the second image channel convolution features to obtain first intermediate features; perform enhanced receptive field feature extraction on the first intermediate features to obtain second intermediate features; perform feature extraction processing on the first intermediate features and the second intermediate features by the neck network to obtain neck output image features; and input the neck output image features to the detection head, and output a model prediction result by the detection head.

[0115] In an example embodiment of the present disclosure, the chip tray detection device 1300 further comprises a chip tray processing module configured to: when the tray detection result is that the tray state is normal, perform a banding operation on the to-be-detected chip tray; when the tray detection result is that the tray state is abnormal, interrupt the operation of the banding machine, determine the to-be-detected chip tray as an abnormal chip tray, and issue an alarm signal; and push the abnormal chip tray to a product management system.

[0116] The specific details of the virtual modules of the above-mentioned model training device and chip tray detection device have been described in detail in the corresponding model training method and chip tray detection method, and thus will not be described here again.

[0117] It should be noted that, although several modules or units of the model training device and the chip tray detection device are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units.

[0118] In addition, in the exemplary embodiments of the present disclosure, an electronic device capable of implementing the above-mentioned model training method or chip tray detection method is also provided.

[0119] Those skilled in the art can understand that various aspects of the present application can be implemented as a system, a method or a program product. Therefore, various aspects of the present application can be embodied as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" here.

[0120] As shown in Figure 14 The electronic device 1400 is in the form of a general computing device. The components of the electronic device 1400 can include, but are not limited to, the above-mentioned at least one processing unit 1410, the above-mentioned at least one storage unit 1420, a bus 1430 connecting different system components (including the storage unit 1420 and the processing unit 1410), a display unit 1440.

[0121] The storage unit stores program code which can be executed by the processing unit 1410, so that the processing unit 1410 performs the steps according to various exemplary embodiments of the present disclosure described in the above "exemplary method" part of the present specification.

[0122] The storage unit 1420 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 1421 and / or a cache memory unit 1422, and can further include a read-only memory (ROM) 1423.

[0123] The storage unit 1420 can also include program / utilities 1424 having a set of (at least one) program modules 1425, such as an operating system, one or more application programs, other program modules, and program data, each of which or some combination of which can include the implementation of a network environment.

[0124] The bus 1430 can represent one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0125] The electronic device 1400 can also communicate with one or more external devices 1470 such as a keyboard or pointing device, a Bluetooth device, or a database, and / or one or more devices that enable a user to interact with the electronic device 1400 and / or one or more devices (e.g., a router, a modem, a server, etc.) that enable the electronic device 1400 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface(s) 1450. Still yet, the electronic device 1400 can communicate with one or more networks (such as one or more local area networks (LANs), wide area networks (WANs), and / or the Internet) through network adapter 1460. As depicted, network adapter 1460 communicates with the other components of the electronic device 1400 via bus 1430. It should be appreciated that although not shown, other hardware and / or software components could be used in conjunction with the electronic device 1400. These include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0126] Through the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0127] In the example embodiments of the present disclosure, a computer readable storage medium is also provided, which stores a program product capable of implementing the above-mentioned method of the present disclosure. In some possible embodiments, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code for causing a terminal device to perform the steps described in the above-mentioned “example method” section according to various example embodiments of the present disclosure when the program product is run on the terminal device.

[0128] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the example embodiments of the present disclosure, and are not for limiting purposes. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0129] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

[0130] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A model training method, characterized in that: include: Acquire a pre-built initial model and a sample detection image, wherein the sample detection image includes a pallet area identifier, and the pallet area identifier includes a pallet position identifier and a pallet gap identifier; The initial model generates a second channel convolution feature based on the original input feature of the sample detection image and the first channel convolution feature, where the first channel convolution feature is obtained by performing residual separable channel convolution on the original input feature; Performing maximum average attention feature extraction on the second channel convolution feature to obtain a first initial intermediate feature; Performing enhanced receptive field feature extraction on the first initial intermediate feature to obtain a second initial intermediate feature; Performing feature extraction processing on the first initial intermediate feature and the second initial intermediate feature to obtain a sample image detection result, wherein the sample image detection result includes a pallet quantity detection result and a pallet gap detection result; The initial model is trained based on the difference between the sample image detection result and the tray area identifier to obtain a chip tray detection model.

2. The method according to claim 1, characterized in that The initial model includes a backbone network, and the initial model generates a second channel convolution feature based on the original input feature of the sample detection image and the first channel convolution feature, including: Performing initial feature extraction processing on the sample detection image through the backbone network to obtain the original input features; Performing residual separable channel convolution processing on the original input features of the sample detection image to obtain the first channel convolution features; The original input feature is fused with the first channel convolution feature to obtain the second channel convolution feature.

3. The method according to claim 2, characterized in that The performing residual separable channel convolution processing on the original input features of the sample detection image to obtain the first channel convolution features includes: Performing group convolution processing on the original input features based on the channel dimension to obtain initial convolution features, where the initial convolution features lack continuous information between channels; Performing feature fusion processing on the initial convolution feature and the original input feature to obtain a first fused feature, where the first fused feature includes inter-channel continuous information; Perform point-by-point convolution processing on the first fusion feature to obtain the first channel convolution feature.

4. The method according to claim 1, wherein The performing maximum average attention feature extraction on the second channel convolution feature to obtain a first initial intermediate feature includes: Performing convolution processing on the second channel convolution feature to obtain a first convolution feature; Performing average pooling and maximum pooling on the height dimension features and width dimension features of the first convolutional features, respectively, to obtain average pooling features and maximum pooling features; Obtain a pre-built weight configuration function, and determine the feature weight corresponding to each feature in the average pooling feature and the maximum pooling feature based on the weight configuration function; The first initial intermediate feature is obtained according to the feature weight and the first convolution feature.

5. The method according to claim 4, characterized in that The performing average pooling processing and maximum pooling processing on the height dimension feature and the width dimension feature of the first convolution feature respectively to obtain the average pooling feature and the maximum pooling feature includes: Performing average pooling and maximum pooling on the height dimension features respectively to obtain average height pooling features and maximum height pooling features; Performing average pooling and maximum pooling on the width dimension features respectively to obtain average width pooling features and maximum width pooling features; Performing feature fusion processing on the average height pooling feature and the average width pooling feature to obtain an initial average pooling feature; Performing feature fusion processing on the maximum height pooling feature and the maximum width pooling feature to obtain an initial maximum pooling feature; Based on the width dimension, performing average pooling processing on the initial average pooling feature to obtain the average pooling feature; Based on the height dimension, the initial maximum pooling feature is subjected to maximum pooling processing to obtain the maximum pooling feature.

6. The method according to claim 1, characterized in that The step of performing enhanced receptive field feature extraction on the first initial intermediate feature to obtain a second initial intermediate feature includes: Performing maximum pooling and average pooling on the first initial intermediate features to obtain a first maximum pooling feature and a first average pooling feature; Based on a pre-configured maximum pooling layer number, performing maximum pooling processing on the first initial intermediate feature to obtain a second maximum pooling feature; Perform feature fusion processing on the first maximum pooling feature, the first average pooling feature, and the second maximum pooling feature to obtain the second initial intermediate feature.

7. The method according to claim 1, characterized in that The initial model includes a neck network and a detection head, the neck network includes a residual separable channel extraction layer, and the feature extraction processing is performed on the first initial intermediate feature and the second initial intermediate feature to obtain a sample image detection result, including: performing feature fusion processing on the first initial intermediate feature and the second initial intermediate feature to obtain a second fused feature; The residual separable channel extraction layer performs residual separable channel convolution processing and maximum average attention feature extraction on the second fusion feature to obtain a neck output feature; The neck output feature is input to the detection head, and the detection head outputs the sample image detection result.

8. A chip tray detection method, characterized in that: include: Obtain an image to be inspected corresponding to the chip tray to be inspected; Obtain a pre-trained chip tray detection model, wherein the chip tray detection model is trained based on the model training method according to any one of claims 1 to 7; The chip tray detection model performs tray detection processing on the image to be detected to obtain a model prediction result; Performing quantity detection processing and gap detection processing on the model prediction results respectively to obtain pallet quantity detection results and pallet gap detection results; A pallet detection result is determined based on the pallet quantity detection result and the pallet gap detection result.

9. The method according to claim 8, characterized in that The step of obtaining the image to be detected corresponding to the chip tray to be detected includes: Acquire the product batch number captured by the image acquisition device, perform batch number matching processing on the product batch number, and obtain a batch number matching result; When the batch number matching result is a successful match, an initial image is captured by the image capture device; Performing template matching processing on the initial image to obtain a template matching result; When the template matching result indicates that the initial image is a chip tray image, performing image correction processing on the initial image to obtain a corrected image; The corrected image is subjected to a region of interest cropping process to obtain the image to be detected.

10. The method according to claim 8, characterized in that The chip tray detection model includes a backbone network, a neck network, and a detection head; the chip tray detection model performs tray detection processing on the image to be detected to obtain a model prediction result, including: Performing residual separable channel convolution processing on the original model input features of the image to be detected through the backbone network to obtain a first image channel convolution feature; Generating a second image channel convolution feature based on the original model input feature and the first image channel convolution feature; Performing maximum average attention feature extraction on the second image channel convolution feature to obtain a first intermediate feature; Performing enhanced receptive field feature extraction on the first intermediate feature to obtain a second intermediate feature; The neck network performs feature extraction processing on the first intermediate feature and the second intermediate feature to obtain a neck output image feature; The neck output image features are input to the detection head, and the detection head outputs the model prediction result.

11. The method according to any one of claims 8 to 10, characterized in that: The method further comprises: When the tray detection result shows that the tray state is normal, performing a taping operation on the chip tray to be detected; When the tray detection result shows that the tray state is abnormal, the operation of the strapping machine is interrupted, the chip tray to be detected is determined to be an abnormal chip tray, and an alarm signal is issued; The abnormal chip tray is pushed to the product management and control system.

12. A model training device, characterized in that: include: A sample image acquisition module, configured to acquire a pre-built initial model and a sample detection image, wherein the sample detection image includes a pallet area identifier, and the pallet area identifier includes a pallet position identifier and a pallet gap identifier; a channel convolution feature determination module, configured to generate, by the initial model, a second channel convolution feature based on the original input feature of the sample detection image and a first channel convolution feature, wherein the first channel convolution feature is obtained by performing residual separable channel convolution processing on the original input feature; A first intermediate feature extraction module is used to perform maximum average attention feature extraction on the second channel convolution feature to obtain a first initial intermediate feature; A second intermediate feature extraction module is used to perform enhanced receptive field feature extraction on the first initial intermediate feature to obtain a second initial intermediate feature; a sample detection result determination module, configured to perform feature extraction processing on the first initial intermediate feature and the second initial intermediate feature to obtain a sample image detection result, wherein the sample image detection result includes a pallet quantity detection result and a pallet gap detection result; The model training module is used to train the initial model based on the difference between the sample image detection result and the tray area identifier to obtain a chip tray detection model.

13. A chip tray detection device, characterized in that: include: An image acquisition module is used to acquire an image to be detected corresponding to the chip tray to be detected; A model acquisition module, configured to acquire a pre-trained chip tray detection model, wherein the chip tray detection model is trained based on the model training method according to any one of claims 1 to 7; A model prediction module, configured to perform tray detection processing on the image to be detected by the chip tray detection model to obtain a model prediction result; A quantity and gap detection module is used to perform quantity detection processing and gap detection processing on the model prediction results to obtain pallet quantity detection results and pallet gap detection results; The detection result determination module is used to determine the pallet detection result based on the pallet quantity detection result and the pallet gap detection result.

14. An electronic device, characterized in that: include: processor; as well as A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the model training method according to any one of claims 1 to 7, or implement the chip tray detection method according to any one of claims 8 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the model training method according to any one of claims 1 to 7 is implemented, or the chip tray detection method according to any one of claims 8 to 11 is implemented.

Images