Model training method, chip tray detection method and device, equipment and product

Abstract

The present disclosure relates to a model training method, a chip tray detection method and device, equipment and product, and relates to the technical field of semiconductor testing. The method comprises: obtaining a pre-constructed initial model and a sample detection image; generating, by the initial model, second channel convolution features based on original input features and first channel convolution features of the sample detection image; performing maximum average attention feature extraction on the second channel convolution features to obtain first initial intermediate features; performing enhanced receptive field feature extraction on the first initial intermediate features to obtain second initial intermediate features; performing feature extraction processing on the first initial intermediate features and the second initial intermediate features to obtain a sample image detection result; and training based on a difference between the sample image detection result and a tray region identifier to obtain a chip tray detection model. The present disclosure can provide detection capability for the gaps or grooves of low-texture black trays.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of semiconductor testing, and in particular, to a model training method, a chip tray detection method, a model training device, a chip tray detection device, an electronic device, and a computer program product. BACKGROUND

[0002] The dynamic random access memory (DRAM) post-package testing stage is a core link for guaranteeing product functions, performance, and reliability. In related schemes, the DRAM post-package taping process mainly relies on a combination of manual visual inspection and mechanical auxiliary detection.

[0003] The manual visual inspection method refers to checking the tray quantity (10 full trays + 1 empty tray / bundle) and the gap by the naked eye, and checking the tray quantity and the stacking gap. In order to partially make up for the defects of manual work, mechanical protection measures are introduced, for example, a scale line can be pasted on the side of the discharge workbench for comparison, and a tray stacking height limiting jig can be installed at the entry of the taping machine to realize the checking operation of the tray quantity and the stacking gap.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The purpose of the present disclosure is to provide a model training method, a chip tray detection method, a model training device, a chip tray detection device, an electronic device, a computer readable storage medium, and a computer program product, thereby at least partially overcoming the problems of low efficiency of manual inspection, inability of mechanical protection to detect chip tray quantity deviation, and tray gap defects caused by chip jumping in the tray in related schemes.

[0006] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, a model training method is provided, comprising: obtaining a pre-constructed initial model and a sample detection image, the sample detection image comprising a tray area identifier, the tray area identifier comprising a tray position identifier and a tray slot identifier; generating, by the initial model, a second channel convolution feature based on a first channel convolution feature and an original input feature of the sample detection image, the first channel convolution feature being obtained by performing residual separable channel convolution processing on the original input feature; performing maximum average attention feature extraction on the second channel convolution feature to obtain a first initial intermediate feature; performing enhanced receptive field feature extraction on the first initial intermediate feature to obtain a second initial intermediate feature; performing feature extraction processing on the first initial intermediate feature and the second initial intermediate feature to obtain a sample image detection result, the sample image detection result comprising a tray number detection result and a tray slot detection result; training the initial model based on a difference between the sample image detection result and the tray area identifier to obtain a chip tray detection model.

[0008] According to a second aspect of the present disclosure, a chip tray detection method is provided, comprising: obtaining a to-be-detected image corresponding to a to-be-detected chip tray; obtaining a pre-trained chip tray detection model, the chip tray detection model being trained based on a model training method; performing, by the chip tray detection model, tray detection processing on the to-be-detected image to obtain a model prediction result; performing number detection processing and slot detection processing on the model prediction result, respectively, to obtain a tray number detection result and a tray slot detection result; and determining a tray detection result based on the tray number detection result and the tray slot detection result.

[0009] According to a third aspect of the present disclosure, a model training apparatus is provided, comprising: a sample image acquisition module configured to acquire an initial model pre-constructed and a sample detection image, the sample detection image comprising a tray area identifier, the tray area identifier comprising a tray position identifier and a tray slot identifier; a channel convolution feature determination module configured to generate a second channel convolution feature based on an original input feature of the sample detection image and a first channel convolution feature by the initial model, the first channel convolution feature being obtained by performing residual separable channel convolution processing on the original input feature; a first intermediate feature extraction module configured to perform maximum average attention feature extraction on the second channel convolution feature to obtain a first initial intermediate feature; a second intermediate feature extraction module configured to perform enhanced receptive field feature extraction on the first initial intermediate feature to obtain a second initial intermediate feature; a sample detection result determination module configured to perform feature extraction processing on the first initial intermediate feature and the second initial intermediate feature to obtain a sample image detection result, the sample image detection result comprising a tray number detection result and a tray slot detection result; and a model training module configured to train the initial model based on a difference between the sample image detection result and the tray area identifier to obtain a chip tray detection model.

[0010] According to a fourth aspect of the present disclosure, a chip tray detection apparatus is provided, comprising: an image acquisition module configured to acquire a to-be-detected image corresponding to a to-be-detected chip tray; a model acquisition module configured to acquire a pre-trained chip tray detection model, the chip tray detection model being trained based on any one of the model training methods described above; a model prediction module configured to perform tray detection processing on the to-be-detected image by the chip tray detection model to obtain a model prediction result; a number and slot detection module configured to perform number detection processing and slot detection processing on the model prediction result respectively to obtain a tray number detection result and a tray slot detection result; and a detection result determination module configured to determine a tray detection result based on the tray number detection result and the tray slot detection result.

[0011] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement any one of the model training methods or chip tray detection methods described above.

[0012] According to a sixth aspect of the present disclosure, a computer readable storage medium is provided, having a computer program stored thereon, the computer program being executed by a processor to implement any one of the model training methods or chip tray detection methods described above.

[0013] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the model training method or the chip tray detection method according to any one of the preceding aspects.

[0014] The technical solutions provided by the present disclosure can have the following beneficial effects:

[0015] In the model training method in the exemplary embodiments of the present disclosure, on the one hand, the first channel convolution feature extracted is fused with the original input feature through the residual separable channel convolution, which can solve the problem of information loss between channels caused by the classic deep separable convolution module. On the other hand, the maximum average attention mechanism can combine the significant information and the average information of the chip tray, more accurately allocate the weight, and improve the detection capability of the gap or groove. On the other hand, the enhanced receptive field module is added in the model, which can combine the global background information, edge information and significant information in the image, and effectively distinguish the tray itself and the abnormal gap.

[0016] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained from these drawings without creative labor for those skilled in the art. In the drawings:

[0018] Figure 1 The schematic diagram shows that the relevant scheme pastes a scale line on the side of the discharge workbench and installs a tray stack height limiting jig at the belt feeding port of the belt labeling machine.

[0019] Figure 2 The flowchart of the model training method according to the exemplary embodiments of the present disclosure is schematically shown.

[0020] Figure 3 The structure diagram of the chip tray detection model according to the exemplary embodiments of the present disclosure is schematically shown.

[0021] Figure 4 The network structure diagram of the residual separable channel feature extraction layer according to the exemplary embodiments of the present disclosure is schematically shown; wherein, Figure 4 (a) in FIG. 1 is the overall structure diagram of the residual separable channel feature extraction layer (RSC Layer); Figure 4 (b) in FIG. 1 is the network structure diagram of the RSC-Conv module in the RSC Layer;Figure 4 (c) is a network structure diagram of the MA-CA module in the RSC Layer.

[0022] Figure 5 A network structure diagram of an SPPF-based enhanced receptive field module according to an example embodiment of the present disclosure is schematically shown.

[0023] Figure 6 A flowchart of a chip tray detection method according to an example embodiment of the present disclosure is schematically shown.

[0024] Figure 7 A hardware installation schematic diagram of a tape dispenser of a chip tray detection method according to an example embodiment of the present disclosure is shown.

[0025] Figure 8 A structure schematic diagram of a control system of a chip tray detection method according to an example embodiment of the present disclosure is shown.

[0026] Figure 9 A schematic diagram of chip tray placement in a tape dispenser according to an example embodiment of the present disclosure is shown.

[0027] Figure 10 A system workflow diagram of a chip tray detection method according to an example embodiment of the present disclosure is shown.

[0028] Figure 11 A schematic diagram of tray quantity detection and gap detection according to an example embodiment of the present disclosure is shown; wherein, Figure 11 (a) in is a schematic diagram of a tray detection frame obtained after target detection on a front view of a chip tray to be detected; Figure 11 (b) in is a schematic diagram of a tray detection frame obtained after target detection on a rear view of a chip tray to be detected; Figure 11 (c) in shows a schematic diagram of a chip tray without abnormal gap; Figure 11 (d) in shows a schematic diagram of a chip tray with abnormal gap.

[0029] Figure 12 A block diagram of a model training apparatus according to an example embodiment of the present disclosure is schematically shown.

[0030] Figure 13 A block diagram of a chip tray detection apparatus according to an example embodiment of the present disclosure is schematically shown.

[0031] Figure 14 A block diagram of an electronic device according to an example embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0032] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments, however, can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the description. Repetitive descriptions of like elements will be omitted for sake of brevity.

[0033] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the

[0034] The block diagrams in the drawings show functions and functionality, which can be implemented in software, hardware, or a combination thereof. As used herein, the term "software" encompasses firmware, resident software, microcode, etc. Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the

[0035] For the DRAM post-process tape-out procedure, the artificial visual inspection method has the problems of easy missed detection and low efficiency, such as a missed detection rate > 5% and a single inspection time > 30 seconds / bundle. In order to partially make up for the artificial defects, a mechanical auxiliary detection method is also proposed, which is referred to as Figure 1 , Figure 1 The schematic diagram of the related scheme is shown in the side of the discharge workbench, which pastes the scale line and installs the tray stacking height limiting jig in the tape inlet of the tape machine.

[0036] From Figure 1As can be seen, the automated visual inspection (AVI) device 110 delivers the chip tray to the discharge workbench 120 through the discharge port, the mechanical auxiliary detection mode compares the scale line 121 (such as a 10+1 tray scale line) pasted on the discharge workbench 120, and the tray stacking height limiting jig 131 (such as a 10+1 tray height limiting jig) installed at the entry of the bundling machine 130 is used to realize auxiliary detection, and finally the vacuum packaging device 140 is used to complete the packaging operation. However, this scheme can only meet the standard delivery mode of 10+1 / bundle, and for non-standard delivery, it relies on manual inspection, which often leads to the event that the operator skips the calibration step and directly delivers. Therefore, the DRAM post-delivery process is prone to problems such as high manual dependence, poor quality control, and rising hidden costs.

[0037] Based on this, in the example embodiment, first, a model training method is provided, which can be implemented by a server or a terminal device. The terminal described in the disclosure can include mobile terminals such as mobile phones, tablets, laptops, palmtop computers, personal digital assistants (PDAs), and fixed terminals such as desktop computers. Figure 2 The model training method according to some embodiments of the disclosure is schematically shown in the schematic diagram. Referring to Figure 2 The model training method can include the following steps:

[0038] Step S210, an initial model constructed in advance and a sample detection image are obtained, the sample detection image includes a tray area identifier, and the tray area identifier includes a tray position identifier and a tray gap identifier;

[0039] Step S220, the initial model generates a second channel convolution feature based on the original input feature and the first channel convolution feature of the sample detection image, and the first channel convolution feature is obtained by performing residual separable channel convolution processing on the original input feature;

[0040] Step S230, maximum average attention feature extraction is performed on the second channel convolution feature to obtain a first initial intermediate feature;

[0041] Step S240, enhanced receptive field feature extraction is performed on the first initial intermediate feature to obtain a second initial intermediate feature;

[0042] Step S250, feature extraction processing is performed on the first initial intermediate feature and the second initial intermediate feature to obtain a sample image detection result, and the sample image detection result includes a tray number detection result and a tray gap detection result;

[0043] Step S260, training the initial model based on the difference between the sample image detection result and the tray area identifier, to obtain a chip tray detection model.

[0044] According to the model training method in the example embodiment, on the one hand, the first channel convolution feature extracted is fused with the original input feature through the residual separable channel convolution, which can solve the problem of information loss between channels caused by the classic deep separable convolution module. On the other hand, the maximum average attention mechanism can combine the significant information and the average information of the chip tray, more accurately allocate the weight, and improve the detection ability of the gap or groove. On the other hand, the enhanced receptive field module is added in the model, which can combine the global background information, edge information and significant information in the image, and effectively distinguish the tray itself and the abnormal gap.

[0045] In the following, the model training method in the example embodiment will be further described.

[0046] In an example embodiment of the disclosure, in step S210, an initial model constructed in advance and a sample detection image are obtained, the sample detection image includes a tray area identifier, and the tray area identifier includes a tray position identifier and a tray gap identifier.

[0047] The sample detection image can be an image in which a region of interest (ROI) of a chip tray is pre-labeled. The tray area identifier can be a tray position identifier and an abnormal gap identifier in the sample detection image. The tray position identifier can be a bounding box identifier of a position region of the chip tray in the sample detection image. The abnormal gap identifier can be a bounding box identifier corresponding to the abnormal gap in the sample detection image.

[0048] After obtaining the initial sample image of the chip tray through the image acquisition device, image preprocessing is performed on the initial sample image to obtain a sample detection image. The image preprocessing can include, but is not limited to, template matching processing, perspective transformation processing, ROI cropping processing, etc. The initial model in the disclosure is a network model constructed on the basis of a You Only Look Once (YOLO) network structure. The model is trained based on the labeled ROI image to obtain a chip detection model, which is used to realize the number and abnormal gap detection of the chip tray.

[0049] In an example embodiment of the present disclosure, for step S220, the initial model generates the second channel convolution feature based on the original input feature of the sample detection image and the first channel convolution feature, comprising: performing initial feature extraction processing on the sample detection image by the backbone network to obtain the original input feature; performing residual separable channel convolution processing on the original input feature of the sample detection image to obtain the first channel convolution feature; and performing feature fusion processing on the original input feature and the first channel convolution feature to obtain the second channel convolution feature.

[0050] The original input feature can be an image feature obtained after initial feature extraction of the sample detection image by the backbone network of the initial model. The first channel convolution feature can be a feature obtained after feature extraction by a residual separable channel convolution (RSC-Conv) module in a residual separable channel feature extraction layer (RSC Layer) of the initial model. The second channel convolution feature can be a feature obtained after feature fusion processing on the original input feature and the first channel convolution feature.

[0051] Reference Figure 3 , Figure 3 An example structure diagram of a chip tray detection model according to an example embodiment of the present disclosure is schematically shown. Figure 3 The initial model in the example embodiment of the present disclosure includes a backbone network (backbone), a neck network (neck), and a detection head (head), and the sample detection image is input into the initial model. The initial feature extraction processing is performed on the sample detection image by the backbone network of the initial model to obtain the original input feature. Specifically, the backbone network can include a Focus structure layer and a Conv2D_BN_SiLU layer.

[0052] The original input image, i.e., the sample detection image, is received by the Focus layer, and the original input image is preliminarily feature-extracted and spatially down-sampled. Then, the Conv2D_BN_SiLU feature extraction layer (referred to as Conv2D_BN_SiLU layer) is used to complete one-time feature extraction, batch normalization, and nonlinear transformation to obtain the original input feature. The Conv2D_BN_SiLU layer is composed of a two-dimensional convolution (Conv2D) + batch normalization (BN) + Sigmoid-Weighted Linear Unit (SiLU) activation function.

[0053] To reduce the amount of calculation, the traditional YOLO network usually adopts the way of dimension reduction to reduce the channel number and reduce the feature information. In order to solve the problem of information loss between channels in the classical deep separable convolution structure, the RSC Layer containing the RSC-Conv module is added in the initial model. Referring to Figure 3 The original input feature is transmitted to the RSC Layer in the backbone network for feature extraction processing.

[0054] Referring to Figure 4 , Figure 4 The network structure diagram of the residual separable channel feature extraction layer according to the example embodiment of the present disclosure is schematically shown. Figure 4 (a) in FIG. 1 is a whole structure diagram of the residual separable channel feature extraction layer (RSC Layer), which can include a specified number (such as n) of RSC-Conv modules, a 1x1 convolution layer, and a Max-Avg-Coordinate Attention (MA-CA) module. In the RSC Layer, the original input feature X is taken as the input of the RSC Layer, and the residual separable channel convolution processing of the original input feature is performed by the feature extraction layer composed of n RSC-Conv modules to obtain the first channel convolution feature X1.

[0055] Then the first channel convolution feature X1 is added to the original input feature X, and the feature fusion processing of the two features is performed to obtain the second channel convolution feature. By performing the feature fusion processing of the first channel convolution feature and the original input feature, the purpose of preventing gradient disappearance and enhancing feature layer information can be achieved.

[0056] In an example embodiment of the present disclosure, the residual separable channel convolution processing of the original input feature of the sample detection image is performed to obtain the first channel convolution feature, including: performing grouped convolution processing on the original input feature based on the channel dimension to obtain the initial convolution feature, the initial convolution feature lacking inter-channel continuous information; performing feature fusion processing on the initial convolution feature and the original input feature to obtain the first fusion feature, the first fusion feature including inter-channel continuous information; and performing point-by-point convolution processing on the first fusion feature to obtain the first channel convolution feature.

[0057] Referring to Figure 4 , Figure 4 (b) in FIG. 1 shows the network structure of the RSC-Conv module. Referring to the residual network feature, the input of the RSC-Conv module can be the input feature layer X0. Taking the input feature layer X0 as the original input feature X, one branch of the RSC-Conv module can perform grouped convolution processing on the input feature layer X0 based on the channel dimension, such as Figure 4(b) in the RSC Layer, performing layer-by-layer convolution on the input feature layer X0 to obtain an initial convolutional feature X1, since the initial convolutional feature X1 is obtained by layer-by-layer convolution based on the channel dimension, the initial convolutional feature X1 lacks inter-channel continuous information.

[0058] To supplement the inter-channel missing continuous information, the RSC-Conv module adds the input feature layer X0 to the feature layer lacking inter-channel continuous information, i.e., the initial convolutional feature X1, and performs feature fusion processing on the two feature layers. Since the original input feature does not perform grouped convolution on the channel dimension, it has continuous information between channels, and therefore, the original input feature and the feature obtained after grouped convolution on the channel are used to supplement the inter-channel missing continuous information by addition operation, to obtain a feature layer X2 having inter-channel continuous information, i.e., a first fused feature X2, the first fused feature X2 including inter-channel continuous information.

[0059] Then, the information between the channels of the first fused feature X2 is integrated by convolution processing, and a feature is extracted to obtain an output feature layer containing more information as a first channel convolutional feature. Figure 4 In the RSC Layer, the RSC-Conv module can perform point-by-point convolution processing on the first fused feature X2 to obtain the first channel convolutional feature. The first channel convolutional feature obtained by feature extraction supplements the inter-channel missing continuous information.

[0060] Further, since the RSC Layer can include one or more RSC-Conv modules, when one RSC-Conv module completes feature extraction processing, the output feature of the RSC-Conv module can be used as the input of the next RSC-Conv module, and the output of the last RSC-Conv module can be used as the first channel convolutional feature.

[0061] In an exemplary embodiment of the present disclosure, the second channel convolutional feature is subjected to maximum average attention feature extraction to obtain a first initial intermediate feature, including: performing convolution processing on the second channel convolutional feature to obtain a first convolutional feature; performing average pooling processing and maximum pooling processing on the height dimension feature and the width dimension feature of the first convolutional feature, respectively, to obtain an average pooling feature and a maximum pooling feature; obtaining a pre-constructed weight configuration function, and determining the feature weight corresponding to each of the average pooling feature and the maximum pooling feature based on the weight configuration function; and obtaining the first initial intermediate feature based on the feature weight and the first convolutional feature.

[0062] Continuing to refer to Figure 4In step (a), after obtaining the second channel convolutional feature, the second channel convolutional feature is convolved, for example, by using a 1×1 convolutional kernel to convolve it, to obtain the first convolutional feature X2. Then, the MA-CA module is used to perform average pooling and max pooling on the height and width dimensions of the first convolutional feature, respectively, to obtain the average pooling feature and max pooling feature.

[0063] Finally, the feature weights of each feature in the feature layer are calculated using the delta function. These weights are then multiplied by the original feature layer in the MA-CA module to obtain the output feature layer with dimensionless feature extraction. The original feature layer in the MA-CA module can be the first convolutional feature layer, and the dimensionless features can be understood as indicators of whether a pixel location has a feature. The MA-CA module can combine salient and average information from the chip tray to more accurately allocate weights, improving the detection capability of gaps or grooves.

[0064] The feature layer extracted by the RSC Layer is processed through the Conv2D_BN_SiLU layer and the Darknet network structure (such as Dark2) to obtain the feature layer after deep feature extraction, spatial downsampling and channel number increase, which is the first initial intermediate feature.

[0065] In one exemplary embodiment of this disclosure, average pooling and max pooling are performed on the height and width dimensions of the first convolutional feature, respectively, to obtain average pooled features and max pooled features. This includes: performing average pooling and max pooling on the height dimension feature to obtain average height pooled features and max height pooled features; performing average pooling and max pooling on the width dimension feature to obtain average width pooled features and max width pooled features; fusing the average height pooled features and average width pooled features to obtain initial average pooled features; fusing the max height pooled features and max width pooled features to obtain initial max pooled features; performing average pooling on the initial average pooled features based on the width dimension to obtain average pooled features; and performing max pooling on the initial max pooled features based on the height dimension to obtain max pooled features.

[0066] Continue to refer to Figure 4 (c) Figure 4(c) in FIG. 4 shows a network structure diagram of the MA-CA module. The first convolutional features are taken as the input layer of the MA-CA module, and the input layer is split along the height and width directions to obtain height-dimension features and width-dimension features, and the above features are processed by two pooling modes (i.e. average pooling and maximum pooling) to generate four feature layers. Specifically, the height-dimension features are processed by average pooling and maximum pooling respectively to obtain average height-pooling features (also referred to as average height-pooling layer along height) and maximum height-pooling features (also referred to as maximum height-pooling layer along height); the width-dimension features are processed by average pooling (also referred to as average width-pooling layer along width) and maximum pooling respectively to obtain average width-pooling features and maximum width-pooling features (also referred to as maximum width-pooling layer along width).

[0067] The average height-pooling layer along height and the average width-pooling layer along width are reconstructed by outer product to obtain an average pooling feature layer, i.e. initial average pooling features; the maximum height-pooling layer along height and the maximum width-pooling layer along width are reconstructed by outer product to obtain a maximum pooling feature layer, i.e. initial maximum pooling features. Then, the initial average pooling features are processed by average pooling along width again to obtain a CxHx1 average pooling feature layer, i.e. average pooling features; the initial maximum pooling features are processed by maximum pooling along height again to obtain a Cx1xW maximum pooling feature layer, i.e. maximum pooling features. By adding the MA-CA module, the model can distinguish the average information of the tray itself (such as stacking gaps) and the salient information of the tray (such as abnormal gaps or taking and placing grooves), and then combine the salient information and the average information to more accurately allocate weights and improve the detection capability of the gaps or grooves.

[0068] With reference to Figure 3 For the output feature layer extracted by the RSC Layer, the Conv2D_BN_SiLU layer and the Dark2 structure are used for feature extraction to obtain first initial intermediate features, and the first initial intermediate features can be further extracted by the RSC Layer, the Conv2D_BN_SiLU layer and the Dark3 structure as the input of the neck network. Similarly, the Dark4 output can be obtained by combining the above model feature extraction process.

[0069] In an example embodiment of the present disclosure, for step S240, the first initial intermediate feature is subjected to enhanced receptive field feature extraction to obtain a second initial intermediate feature, including: subjecting the first initial intermediate feature to maximum pooling processing and average pooling processing respectively to obtain a first maximum pooling feature and a first average pooling feature; subjecting the first initial intermediate feature to maximum pooling processing based on a pre-configured maximum pooling layer number to obtain a second maximum pooling feature; and subjecting the first maximum pooling feature, the first average pooling feature and the second maximum pooling feature to feature fusion processing to obtain the second initial intermediate feature.

[0070] To overcome the problem that the traditional YOLO algorithm has a limited receptive field and cannot distinguish between global background information (the whole tray) and significant information (abnormal gaps) in a black low-texture tray, the present disclosure proposes to add an enhanced receptive field module (ERF-SPPF) in the model structure to combine global background information, edge information and significant information to effectively distinguish between the tray itself and abnormal gaps. After being extracted via the RSC Layer, the Dark4 output is input into the SPPF-based enhanced receptive field module (Enhanced Receptive Field - Space Pooling Pyramid Fast, ERF-SPPF) for feature extraction. Referring to Figure 5 , Figure 5 The network structure diagram of the SPPF-based enhanced receptive field module according to an example embodiment of the present disclosure is schematically shown.

[0071] In the present disclosure, the feature obtained via the RSC Layer can be taken as the first initial intermediate feature. From Figure 5 As can be seen from FIG. 4, the ERF-SPPF module subjects the first initial intermediate feature to maximum pooling processing and average pooling processing respectively to obtain a first maximum pooling feature and a first average pooling feature. Moreover, a specified number of maximum pooling feature extraction layers, i.e. a maximum pooling layer number, are configured in another branch of the ERF-SPPF module, and the maximum pooling layer number n of the input layer can be set according to the input layer size.

[0072] The first initial intermediate feature is subjected to maximum pooling processing based on a preconfigured maximum number of pooling layers to obtain a second maximum pooled feature. After obtaining the second maximum pooled feature, the first maximum pooled feature, the first average pooled feature, and the second maximum pooled feature are subjected to feature fusion processing, and the fused features are subjected to feature extraction processing by a Conv2D_BN_SiLU feature extraction layer to obtain a second initial intermediate feature. After passing through the RSC Layer, the ERF-SPPF module is used again. The global average pooling layer and the global maximum pooling layer are mainly used. Some global background information and edge information are added to the ERF-SPPF module. The global background information and the edge information are used to better detect significant information. After passing through the Dark5, the output of the layer is obtained. The enhanced receptive field module can combine the global background information, the edge information, and the significant information to effectively distinguish the tray itself and the abnormal gap.

[0073] For ease of distinction, the present disclosure can also take the feature obtained by the RSC Layer and the ERF-SPPF module as the second initial intermediate feature. Then, the first initial intermediate feature and the second initial intermediate feature are taken as the input of the neck network. For example, the first initial intermediate feature can be the Dark3 output and the Dark4 output, the second initial intermediate feature can be the Dark5 output, and the Dark3 output, the Dark4 output, and the Dark5 output are taken as the input of the neck network.

[0074] In an exemplary embodiment of the present disclosure, the first initial intermediate feature and the second initial intermediate feature are subjected to feature extraction processing to obtain a sample image detection result, including: the first initial intermediate feature and the second initial intermediate feature are subjected to feature fusion processing to obtain a second fusion feature; the residual separable channel extraction layer is used to perform residual separable channel convolution processing and maximum average attention feature extraction on the second fusion feature to obtain a neck output feature; and the neck output feature is input to the detection head, and the detection head outputs the sample image detection result.

[0075] Continuing to refer to Figure 3 The neck network of the present disclosure is basically the same as the original structure of YOLO. The RSCLayer, i.e., the residual separable channel extraction layer, is mainly added for non-dimensional feature extraction, which strengthens the extraction ability of channel information and significant information features. The Dark3 output, the Dark4 output, and the Dark5 output are taken as the three inputs of the neck network. After the Dark5 output is subjected to feature extraction by the Conv2D_BN_SiLU feature extraction layer and the up-sampling (UpSample) layer, the output of the UpSample layer and the Dark4 output are subjected to feature fusion processing to obtain a second fusion feature.

[0076] The second fusion feature is input to the RSC Layer, which includes an RSC-Conv module and an MA-CA module. Therefore, through the RSC Layer, the second fusion feature can be subjected to residual separable channel convolution processing and maximum average attention feature extraction, and the output of the RSC Layer is obtained.

[0077] Then, the output of the RSC Layer is subjected to feature extraction processing through a Conv2D_BN_SiLU feature extraction layer and an UpSample layer, the extracted features are subjected to feature fusion processing with the Dark3 output, and the fusion features are obtained. The fusion features are subjected to feature extraction processing through the RSC Layer as the neck output features. With reference to Figure 3 , based on the feature extraction layer of the neck network, two other neck output features are obtained. For example, the first neck output feature is subjected to down-sampling processing, and the features obtained by down-sampling are subjected to feature fusion processing with the features output by the Conv2D_BN_SiLU layer. The fusion features are processed by the RSC Layer to obtain the second neck output feature. Similarly, the third neck output feature is obtained.

[0078] The three feature layers output by the neck are input to the detection head, which is basically the same as the original structure of YOLO. The three detection heads (YOLO-HEAD) obtain prediction results based on the neck output features as sample image detection results. The sample image detection results can include tray number detection results and tray gap detection results. The tray number detection results are mainly obtained according to the number of tray detection boxes output by the detection head. The tray number detection results include the detection box of the position of each chip tray, and can also include the tray statistical number calculated according to the tray detection box. The tray gap detection results can be determined by the abnormal gap detection box output by the detection head. The sample image detection results are used as the data basis for model training.

[0079] After obtaining the sample image detection results, the difference between the sample image detection results and the tray region identifier can be calculated. For example, the tray position identifier is compared with the chip tray detection box output by the model, and the tray gap identifier is compared with the abnormal gap detection box. A loss function is constructed based on the above differences, and the initial model is trained based on the constructed loss function until the model training end condition is met, and the chip tray detection model is obtained.

[0080] The model parameters of the chip tray detection model are stored in the pre-training weight file, so that the pre-training weight file can be called by the microcomputer in the tape automated bonding machine to perform tray number detection and gap anomaly detection on the chip tray to be detected.

[0081] In summary, the model training method of the present disclosure, on the one hand, the first channel convolution feature extracted is fused with the original input feature through the residual separable channel convolution, which can solve the problem of information loss between channels caused by the classic deep separable convolution module. On the other hand, the maximum average attention mechanism can combine the significant information and the average information of the chip tray, more accurately allocate the weight, and improve the detection ability of the gap or groove. On the other hand, the enhanced receptive field module is added in the model, which can combine the global background information, edge information and significant information in the image, effectively distinguish the tray itself and the abnormal gap. On the other hand, through the above-mentioned model training method, the network model for chip tray quantity detection and abnormal gap detection is trained.

[0082] In addition, the present disclosure also provides a chip tray detection method. The chip tray detection method of the present disclosure can be implemented by a server, or the method of the present disclosure can be implemented by a terminal device. The terminal described in the present disclosure can include mobile terminals such as mobile phones, tablet computers, notebook computers, palmtop computers, personal digital assistants (PDA), and fixed terminals such as desktop computers. Figure 6 The schematic diagram of the chip tray detection method flow according to some embodiments of the present disclosure is schematically shown. Referring to Figure 6 The chip tray detection method can include the following steps:

[0083] Step S610, obtaining a to-be-detected image corresponding to a to-be-detected chip tray;

[0084] Step S620, obtaining a pre-trained chip tray detection model, wherein the chip tray detection model is trained based on the model training method;

[0085] Step S630, performing tray detection processing on the to-be-detected image by the chip tray detection model to obtain a model prediction result;

[0086] Step S640, performing quantity detection processing and gap detection processing on the model prediction result respectively to obtain a tray quantity detection result and a tray gap detection result;

[0087] Step S650, determining a tray detection result based on the tray quantity detection result and the tray gap detection result.

[0088] According to the chip tray detection method in the example embodiment, on the one hand, the number of stacked layers of the chip tray can be accurately counted through AI vision. On the other hand, through the gap detection process, the gap of the tray stack is identified, and the sensitivity can reach 0.1 mm level, which can intercept the risk of chip solder ball being crushed. On the other hand, through automatic visual detection, the detection time can be greatly reduced, the detection efficiency can be improved, and the production line beat delay can be avoided.

[0089] In the following, the chip tray detection method in the example embodiment will be further described.

[0090] The delivery process of the tray can include steps such as laser marking (Laser Marking), baking (Baking), automated visual inspection (Automated Visual Inspection, AVI), bundling (Bundler), packaging (Packing), etc. The present disclosure focuses on the chip tray bundling process after the AVI site, and proposes an automatic detection scheme combining artificial intelligence (Artificial Intelligence, AI) visual recognition and Internet of Things (Internet of Things, IoT) technology. Through high-precision image acquisition and real-time analysis of the chip tray, the tray quantity verification, 0.1 mm level gap identification and manufacturing execution system (Manufacturing Execution System, MES) closed-loop control are realized.

[0091] In an example embodiment of the present disclosure, for step S610, the image to be detected corresponding to the chip tray to be detected is obtained, including: obtaining the product batch number collected by the image collection device, performing batch number matching processing on the product batch number to obtain a batch number matching result; when the batch number matching result is matching success, collecting an initial image through the image collection device; performing template matching processing on the initial image to obtain a template matching result; when the template matching result is that the initial image is a chip tray image, performing image correction processing on the initial image to obtain a corrected image; performing region of interest cropping processing on the corrected image to obtain the image to be detected.

[0092] The image acquisition device can be a device for acquiring images of the chip tray. The product batch number can be a product batch number corresponding to the chip tray to be detected before the bundling process. The batch number matching result can be a result obtained by performing consistency matching on the product batch number of the chip tray to be detected and the reference product batch number obtained from the MES. The initial image can be an image of multiple views of the chip tray to be detected, such as front and rear views, acquired by the image acquisition device. The template matching result can be a detection result of detecting whether the initial image is a chip tray image. The chip tray image can be an image containing the chip tray. The image correction process can be a process of correcting the initial image to a front view image. The corrected image can be a front view image of the chip tray to be detected image obtained after the image correction process. The image to be detected can be an image obtained by performing region of interest cropping on the corrected image.

[0093] Reference Figure 7 , Figure 7 A hardware installation schematic diagram of a bundling machine for a chip tray detection method according to an example embodiment of the present disclosure is shown. Figure 7 In the bundling machine 700, image acquisition devices are installed on the top and both sides, such as high-resolution industrial cameras or high-resolution cameras, including a first camera 710 installed on the top, a second camera 720 and a third camera 730 installed on both sides, for real-time acquisition of images of product batch numbers and multiple views of chip trays before bundling, completion of stereo calibration and image correction, and ensuring coverage of the entire area of the tray, such as Figure 7 The image acquisition device can acquire front and rear view images of the chip tray to be detected. In addition, the specific installation position of the image acquisition device can be adjusted according to the structure of the bundling machine and the placement position of the chip tray to realize image acquisition.

[0094] In addition, an AI detection result display panel 740 is also installed in the bundling machine, and a mini computer is installed inside the bundling machine. The mini computer is connected to the circuit and sensor of the bundling machine through a relay to trigger the bundling, and controls the operation of the bundling machine. Figure 8 , Figure 8 A structure schematic diagram of a control system for a chip tray detection method according to an example embodiment of the present disclosure is shown. Figure 8 The edge computing unit of the mini computer 810 integrates a trigger signal linked with the out-band sensor 820 of the bundling machine to ensure that image acquisition and bundling actions are completed in sequence. The out-band sensor 820 is installed at the in-band port of the bundling machine 700, and the mini computer 810 can transmit signals to the out-band sensor 820 through a relay 830. The mini computer 810 can also interact between a server 840 and an MES system 850.

[0095] After the hardware installation is completed, the above hardware device can be used to detect the chip tray. Reference Figure 9 , Figure 9 A schematic diagram of chip tray placement in a taping machine according to an example embodiment of the present disclosure is shown. A chip tray 910 is placed in a taping machine 700, and then the chip tray quantity detection and tray gap detection are performed using the chip tray detection method of the present disclosure to obtain the corresponding tray detection results.

[0096] Specifically, reference Figure 10 , Figure 10 A system workflow diagram of the chip tray detection method according to an example embodiment of the present disclosure is shown. After the program initialization in the host computer is completed and the taping machine is placed by the object to be detected, the image acquisition device (such as the first camera 710) in the taping machine 700 will acquire the image of the object to be detected that has been placed, and obtain the product batch number of the object to be detected.

[0097] After the image acquisition device acquires the product batch number, it obtains the reference product batch number from the MES database, which stores the product batch numbers of the chip products that need to be banded out of the warehouse in the current period. The product batch number is matched with the reference product batch number, and the correctness of the product batch number acquired by the image acquisition device is determined to obtain the batch number matching result. If the physical disk identifier (PDID) is not queried from the MES, it is considered that the batch number matching result is a matching failure, and no subsequent operation will be performed. When the batch number matching result is a matching success, the initial image is acquired by the image acquisition device (such as the second camera 720 and the third camera 730), which can be the front and rear views of the chip tray image.

[0098] The acquired initial image is preprocessed, which specifically includes the following steps: template matching processing is performed on the initial image to determine whether the object to be detected placed in the taping machine is a chip tray (target Tray), and a template matching result is obtained. The template matching is used to detect whether the target Tray exists, and if it exists, the Scale-invariant feature transform (SIFT) algorithm is used to extract the four corners of the target Tray; if it does not exist, the subsequent process is not continued, and the template matching operation can realize the tray target detection.

[0099] When the template matching result is that the initial image is a chip tray image, the front view and the rear view of the chip tray are subjected to image correction processing to obtain a corrected image, and the problem of improper placement of the chip tray is eliminated. For example, the image is corrected by perspective transformation, and the homography matrix is calculated based on the detected corner points to correct the image to a front view.

[0100] Then, the corrected image is subjected to ROI clipping processing to clip the to-be-detected subject tray, prevent environmental factors such as reflection from affecting, and take the obtained ROI image as the to-be-detected image, that is, as the input of the chip tray detection model. By performing the above preprocessing operation on the collected initial image, the perspective distortion caused by non-standard placement of the tray can be eliminated, and the ROI image with front and rear views is obtained, which is beneficial to the model to perform quantity and gap detection processing.

[0101] In an exemplary embodiment of the present disclosure, for step S630, the tray detection processing of the to-be-detected image is performed by the chip tray detection model to obtain a model prediction result, including: performing residual separable channel convolution processing on the original model input feature of the to-be-detected image by the backbone network to obtain first image channel convolution features; generating second image channel convolution features based on the original model input feature and the first image channel convolution features; performing maximum average attention feature extraction on the second image channel convolution features to obtain first intermediate features; performing enhanced receptive field feature extraction on the first intermediate features to obtain second intermediate features; performing feature extraction processing on the first intermediate features and the second intermediate features by the neck network to obtain neck output image features; inputting the neck output image features into the detection head, and outputting the model prediction result by the detection head.

[0102] The original model input feature can be an image feature obtained by feature extraction on the to-be-detected image after inputting the to-be-detected image into the chip tray detection model. The model prediction result can be an image prediction result output by the chip tray detection model, and the model prediction result can include multiple detection boxes, including a tray position detection box and a tray gap detection box.

[0103] After obtaining the to-be-detected image, the to-be-detected image is input into the chip tray detection model, which can include a backbone network, a neck network, and a detection head. The backbone network includes an RSC Layer, and the RSC Layer includes an RSC-Conv module and an MA-CA module. The original model input feature of the to-be-detected image is subjected to residual separable channel convolution processing by the RSC-Conv module in the RSC Layer to obtain first image channel convolution features; and the original model input feature and the first image channel convolution features are subjected to feature fusion processing to generate second image channel convolution features.

[0104] Then, the second image channel convolution feature is subjected to maximum average attention feature extraction by the MA-CA module to obtain a first intermediate feature; and the first intermediate feature is input into the ERF-SPPF module, and the first intermediate feature is subjected to enhanced receptive field feature extraction by the ERF-SPPF module to obtain a second intermediate feature. The first intermediate feature and the second intermediate feature are both taken as the output of the backbone network, and the first intermediate feature and the second intermediate feature are taken as the input of the neck network, and the first intermediate feature and the second intermediate feature are subjected to feature extraction processing by the neck network to obtain a neck output image feature; and the neck output image feature is input into the detection head, and a model prediction result is output by the detection head. The model prediction result can include a tray bounding box and a gap bounding box in the image, which are taken as the data basis for subsequent quantity detection and gap detection of the chip tray.

[0105] The processing process of the chip tray detection model outputting the model prediction result in the embodiment is the same as the processing process of the initial model extracting features from the sample detection image to obtain a sample image detection result, and the present disclosure will not repeat it here.

[0106] For step S640, the model prediction result is subjected to quantity detection processing and gap detection processing respectively to obtain a tray quantity detection result and a tray gap detection result.

[0107] After obtaining the model prediction result, the model prediction result is subjected to quantity detection processing and gap detection processing respectively. Referring to Figure 11 , Figure 11 A schematic diagram of tray quantity detection and gap detection according to an example embodiment of the present disclosure is shown. Quantity detection includes: by target detection on the pick-and-place groove, counting the number of pick-and-place groove targets, and verifying and comparing the counting result with the information read from the MES, obtaining the quantity counting result of the chip tray.

[0108] In the embodiment, the image acquisition device obtains two views of the chip tray to be detected as the detection image, and the pick-and-place groove of the chip to be detected includes one groove on each of the left and right sides in the front and rear views of the tray, which facilitates taking or placing the tray. Figure 11 (a) in (a) and (b) in (b) of Figure 11 The front view and the rear view of the chip tray to be detected are shown in (a) and (b) of the model prediction result, i.e. the schematic diagram of the tray detection box obtained after target detection. After target detection on the front and rear views of the pick-and-place groove, the prediction result of the tray detection box is obtained, and the number of targets of the pick-and-place groove is counted according to the model prediction result. After obtaining the prediction result of the left and right grooves in the front and rear views, i.e. the quantity counting result of the four detection boxes, the final target quantity is determined according to the quantity counting result of the four detection boxes. For example, the mode of the four quantity counting results is taken as the final quantity result, i.e. the target quantity.

[0109] In addition, the target detection of the abnormal gap is used to detect whether the chip tray to be detected has an abnormal gap, to determine whether the chip has jumped, and if there is an abnormal gap, the abnormal gap is marked as a tray gap detection result. For example, Figure 11 (c) in FIG. 1 and Figure 11 (d) in FIG. 1 respectively show the schematic diagrams of two cases of no abnormal gap and abnormal gap of the chip tray.

[0110] In an exemplary embodiment of the present disclosure, when the tray detection result is that the tray state is normal, the chip tray to be detected is subjected to a banding operation; when the tray detection result is that the tray state is abnormal, the banding machine operation is interrupted, the chip tray to be detected is determined as an abnormal chip tray, and an alarm signal is issued; and the abnormal chip tray is pushed to a product management system.

[0111] The tray state normal means that the tray quantity of the chip tray is consistent with the tray shipment quantity in the MES system and there is no tray gap. The abnormal chip tray can be a chip tray having at least one abnormal state such as inconsistent tray quantity and tray gap. The product management system can be a control system for bundling processing or abnormal processing of the detected chip tray.

[0112] After obtaining the tray detection result, the batch information is interactively verified with the MES system through the IoT gateway, specifically including preset parameter verification such as 10 trays + 1 top cover / bundle quantity comparison; then the gap detection result is compared with the standard threshold value, and the comparison result is stored in the MES system in real time. If the tray state is normal, the banding work can be performed. If the tray state is abnormal, such as inconsistent information verification or detection abnormality (quantity inconsistency or tray stacking gap exceeding the standard), an alarm is immediately reminded and the sensor is interrupted to send an opening signal to interrupt the banding machine work, the batch is marked as a problem product, and is pushed to the MES production management system. Through the above steps, the present disclosure realizes an automatic detection system combining AI visual recognition and IoT technology, realizes tray quantity verification, 0.1 millimeter level gap identification and MES system closed loop control through high precision image acquisition and real time analysis.

[0113] It should be noted that the terms "first", "second", etc. used in the present disclosure are only used to distinguish different channel convolution features, initial intermediate features, convolution features and maximum pooling features, and should not impose any limitation on the present disclosure.

[0114] To sum up, the chip tray detection method of the present disclosure, on the one hand, can realize accurate counting through AI vision accurate identification of the stacking number of chip trays. On the other hand, through the gap detection process, the stacking gap of the tray is identified, and the sensitivity can reach the level of 0.1 mm, which can intercept the risk of chip solder ball being pressed. On the other hand, through automatic visual detection, the detection time can be greatly reduced from the original 30s / time to within 1s, and the detection efficiency can be improved by 99% to avoid production line beat delay. On the other hand, the DRAM post-process can realize full automation closed-loop management and control, reduce manual intervention, and reduce labor cost.

[0115] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired result. In addition or alternatively, some steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps, etc.

[0116] Next, the present disclosure also provides a model training device. Referring to Figure 12 The model training device 1200 can include a sample image acquisition module 1210, a channel convolution feature determination module 1220, a first intermediate feature extraction module 1230, a second intermediate feature extraction module 1240, a sample detection result determination module 1250, and a model training module 1260.

[0117] Specifically, the sample image acquisition module 1210 is configured to acquire a pre-constructed initial model and a sample detection image, the sample detection image including a tray area identifier, the tray area identifier including a tray position identifier and a tray gap identifier; the channel convolution feature determination module 1220 is configured to generate a second channel convolution feature based on the original input feature and the first channel convolution feature of the sample detection image from the initial model, the first channel convolution feature being obtained by performing residual separable channel convolution processing on the original input feature; the first intermediate feature extraction module 1230 is configured to perform maximum average attention feature extraction on the second channel convolution feature to obtain a first initial intermediate feature; the second intermediate feature extraction module 1240 is configured to perform enhanced receptive field feature extraction on the first initial intermediate feature to obtain a second initial intermediate feature; the sample detection result determination module 1250 is configured to perform feature extraction processing on the first initial intermediate feature and the second initial intermediate feature to obtain a sample image detection result, the sample image detection result including a tray quantity detection result and a tray gap detection result; and the model training module 1260 is configured to train the initial model based on the difference between the sample image detection result and the tray area identifier to obtain a chip tray detection model.

[0118] In an example implementation of the present disclosure, the initial model comprises a backbone network, the channel convolution feature determination module 1220 comprises a channel convolution feature determination unit configured to: perform initial feature extraction processing on the sample detection image by the backbone network to obtain original input features; perform residual separable channel convolution processing on the original input features of the sample detection image to obtain first channel convolution features; and perform feature fusion processing on the original input features and the first channel convolution features to obtain second channel convolution features.

[0119] In an example implementation of the present disclosure, the channel convolution feature determination unit comprises a channel convolution feature determination subunit configured to: perform grouped convolution processing on the original input features based on a channel dimension to obtain initial convolution features, the initial convolution features lacking inter-channel continuous information; perform feature fusion processing on the initial convolution features and the original input features to obtain first fusion features, the first fusion features comprising inter-channel continuous information; and perform point-by-point convolution processing on the first fusion features to obtain the first channel convolution features.

[0120] In an example implementation of the present disclosure, the first intermediate feature extraction module 1230 comprises a first intermediate feature extraction unit configured to: perform convolution processing on the second channel convolution features to obtain first convolution features; perform average pooling processing and maximum pooling processing on height dimension features and width dimension features of the first convolution features, respectively, to obtain average pooling features and maximum pooling features; obtain a pre-constructed weight configuration function, determine feature weights corresponding to each of the average pooling features and the maximum pooling features based on the weight configuration function; and obtain first initial intermediate features according to the feature weights and the first convolution features.

[0121] In an example implementation of the present disclosure, the first intermediate feature extraction unit comprises a feature extraction subunit configured to: perform average pooling processing and maximum pooling processing on the height dimension features to obtain average height pooling features and maximum height pooling features; perform average pooling processing and maximum pooling processing on the width dimension features to obtain average width pooling features and maximum width pooling features; perform feature fusion processing on the average height pooling features and the average width pooling features to obtain initial average pooling features; perform feature fusion processing on the maximum height pooling features and the maximum width pooling features to obtain initial maximum pooling features; perform average pooling processing on the initial average pooling features based on the width dimension to obtain the average pooling features; and perform maximum pooling processing on the initial maximum pooling features based on the height dimension to obtain the maximum pooling features.

[0122] In an example embodiment of the present disclosure, the second intermediate feature extraction module 1240 includes a second intermediate feature extraction unit configured to: perform maximum pooling processing and average pooling processing on the first initial intermediate feature respectively to obtain a first maximum pooled feature and a first average pooled feature; perform maximum pooling processing on the first initial intermediate feature based on a preconfigured maximum pooling layer number to obtain a second maximum pooled feature; and perform feature fusion processing on the first maximum pooled feature, the first average pooled feature, and the second maximum pooled feature to obtain a second initial intermediate feature.

[0123] In an example embodiment of the present disclosure, the initial model includes a neck network and a detection head, the neck network includes a residual separable channel extraction layer, and the sample detection result determination module 1250 includes a sample detection result determination unit configured to: perform feature fusion processing on the first initial intermediate feature and the second initial intermediate feature to obtain a second fusion feature; perform residual separable channel convolution processing and maximum average attention feature extraction on the second fusion feature by the residual separable channel extraction layer to obtain a neck output feature; and input the neck output feature into the detection head to output a sample image detection result by the detection head.

[0124] In addition, the present disclosure also provides a chip tray detection device. Referring to Figure 13 The chip tray detection device can include an image acquisition module 1310, a model acquisition module 1320, a model prediction module 1330, a quantity and gap detection module 1340, and a detection result determination module 1350.

[0125] Specifically, the image acquisition module 1310 is configured to acquire a to-be-detected image corresponding to a to-be-detected chip tray; the model acquisition module 1320 is configured to acquire a pre-trained chip tray detection model, which is trained based on a model training method; the model prediction module 1330 is configured to perform tray detection processing on the to-be-detected image by the chip tray detection model to obtain a model prediction result; the quantity and gap detection module 1340 is configured to perform quantity detection processing and gap detection processing on the model prediction result respectively to obtain a tray quantity detection result and a tray gap detection result; and the detection result determination module 1350 is configured to determine a tray detection result based on the tray quantity detection result and the tray gap detection result.

[0126] In an example embodiment of the present disclosure, the image acquisition module 1310 comprises an image acquisition unit configured to: acquire a product batch number captured by an image capturing device, perform batch number matching processing on the product batch number to obtain a batch number matching result; when the batch number matching result is a matching success, capture an initial image by the image capturing device; perform template matching processing on the initial image to obtain a template matching result; when the template matching result is that the initial image is a chip tray image, perform image correction processing on the initial image to obtain a corrected image; and perform region of interest cropping processing on the corrected image to obtain a to-be-detected image.

[0127] In an example embodiment of the present disclosure, the chip tray detection model comprises a backbone network, a neck network and a detection head; the model prediction module 1330 comprises a model prediction unit configured to: perform residual separable channel convolution processing on original model input features of the to-be-detected image by the backbone network to obtain first image channel convolution features; generate second image channel convolution features based on the original model input features and the first image channel convolution features; perform maximum average attention feature extraction on the second image channel convolution features to obtain first intermediate features; perform enhanced receptive field feature extraction on the first intermediate features to obtain second intermediate features; perform feature extraction processing on the first intermediate features and the second intermediate features by the neck network to obtain neck output image features; and input the neck output image features to the detection head, and output a model prediction result by the detection head.

[0128] In an example embodiment of the present disclosure, the chip tray detection device 1300 further comprises a chip tray processing module configured to: when the tray detection result is that the tray state is normal, perform a banding operation on the to-be-detected chip tray; when the tray detection result is that the tray state is abnormal, interrupt the operation of the banding machine, determine the to-be-detected chip tray as an abnormal chip tray, and issue an alarm signal; and push the abnormal chip tray to a product management system.

[0129] The specific details of the virtual modules of the model training device and the chip tray detection device described above have been described in detail in the corresponding model training method and chip tray detection method, and thus will not be described here.

[0130] It should be noted that although several modules or units of the model training device and the chip tray detection device are mentioned in the foregoing detailed description, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units.

[0131] In addition, in the exemplary embodiments of the present disclosure, an electronic device capable of implementing the above-mentioned model training method or chip tray detection method is also provided.

[0132] Those skilled in the art can understand that various aspects of the present application can be implemented as a system, a method or a program product. Therefore, various aspects of the present application can be embodied as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" here.

[0133] As shown in Figure 14 The components of electronic device 1400 can include, but are not limited to, at least one processing unit 1410 as described above, at least one storage unit 1420 as described above, a bus 1430 that connects different system components (including the storage unit 1420 and the processing unit 1410), and a display unit 1440.

[0134] The storage unit stores program codes, which can be executed by the processing unit 1410, so that the processing unit 1410 performs the steps described in the above "exemplary method" section according to various exemplary embodiments of the present disclosure.

[0135] The storage unit 1420 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 1421 and / or a cache memory unit 1422, and can further include a read-only memory (ROM) 1423.

[0136] The storage unit 1420 can also include program / utility 1424 having a set of program modules 1425, including but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof can include implementation of a network environment.

[0137] The bus 1430 can represent one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit or a local bus using any of a variety of bus structures.

[0138] The electronic device 1400 can also communicate with one or more external devices 1470 such as a keyboard or pointing device, a Bluetooth device, or a database, and / or one or more devices that enable a user to interact with the electronic device 1400 and / or one or more devices (e.g., a router, a modem, a server, etc.) that enable the electronic device 1400 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface(s) 1450. Still yet, the electronic device 1400 can communicate with one or more networks (such as one or more local area networks (LANs), wide area networks (WANs), and / or the Internet) through network adapter 1460. As depicted, network adapter 1460 communicates with the other components of the electronic device 1400 via bus 1430. It should be appreciated that although not shown, other hardware and / or software components could be used in conjunction with the electronic device 1400. These include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0139] From the above description of the embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0140] In the example embodiments of the present disclosure, a computer-readable storage medium is also provided, which stores a program product capable of implementing the above-mentioned method of the present disclosure. In some possible embodiments, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program codes for causing a terminal device to perform the steps according to various example embodiments of the present disclosure described in the above-mentioned "example method" section of the present disclosure when the program product is run on the terminal device.

[0141] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the methods according to the example embodiments of the present disclosure, and are not for limiting purposes. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0142] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

[0143] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A model training method, characterized in that, include: Obtain a pre-built initial model and sample detection images, wherein the sample detection images include a tray area identifier, and the tray area identifier includes a tray position identifier and a tray gap identifier; The initial model generates a second channel convolution feature based on the original input features of the sample detection image and the first channel convolution feature. The first channel convolution feature is obtained by performing residual separable channel convolution processing on the original input features. Maximum average attention feature extraction is performed on the convolutional features of the second channel to obtain the first initial intermediate features; Enhanced receptive field feature extraction is performed on the first initial intermediate feature to obtain the second initial intermediate feature; Feature extraction processing is performed on the first initial intermediate feature and the second initial intermediate feature to obtain sample image detection results, which include tray quantity detection results and tray gap detection results. Based on the difference between the sample image detection result and the tray area identifier, the initial model is trained to obtain the chip tray detection model.

2. The method according to claim 1, characterized in that, The initial model includes a backbone network. The generation of second-channel convolutional features by the initial model based on the original input features of the sample detection image and the first-channel convolutional features includes: The original input features are obtained by performing initial feature extraction on the sample detection image through the backbone network. The original input features of the sample detection image are subjected to residual separable channel convolution processing to obtain the first channel convolution feature; The original input features are fused with the first channel convolutional features to obtain the second channel convolutional features.

3. The method according to claim 2, characterized in that, The residual separable channel convolution processing of the original input features of the sample detection image to obtain the first channel convolution feature includes: The original input features are grouped and convolved based on the channel dimension to obtain initial convolutional features, which lack continuous information between channels. The initial convolutional features are fused with the original input features to obtain a first fused feature, which includes inter-channel continuous information. The first fused feature is subjected to point-by-point convolution processing to obtain the first channel convolution feature.

4. The method according to claim 1, characterized in that, The step of extracting the first initial intermediate features by performing maximum average attention feature extraction on the second channel convolutional features includes: The second channel convolutional feature is convolved to obtain the first convolutional feature; Average pooling and max pooling are performed on the height and width dimensions of the first convolutional feature, respectively, to obtain average pooling features and max pooling features. Obtain a pre-constructed weight configuration function, and determine the feature weights corresponding to each feature in the average pooling feature and the max pooling feature based on the weight configuration function; The first initial intermediate feature is obtained based on the feature weights and the first convolutional feature.

5. The method according to claim 4, characterized in that, The step of performing average pooling and max pooling on the height and width dimensions of the first convolutional feature, respectively, to obtain average pooling features and max pooling features includes: The height dimension features are subjected to average pooling and max pooling respectively to obtain average height pooling features and max height pooling features. The width dimension features are subjected to average pooling and max pooling respectively to obtain average width pooling features and max width pooling features. The average height pooling feature and the average width pooling feature are fused to obtain the initial average pooling feature. The maximum height pooling feature and the maximum width pooling feature are fused together to obtain the initial maximum pooling feature. The initial average pooling feature is subjected to average pooling based on the width dimension to obtain the average pooling feature. Based on the height dimension, the initial max pooling feature is subjected to max pooling to obtain the max pooling feature.

6. The method according to claim 1, characterized in that, The step of enhancing the receptive field feature extraction of the first initial intermediate feature to obtain the second initial intermediate feature includes: The first initial intermediate features are subjected to max pooling and average pooling respectively to obtain the first max pooling feature and the first average pooling feature. Based on the pre-configured number of max pooling layers, the first initial intermediate feature is subjected to max pooling to obtain the second max pooled feature. The first max pooling feature, the first average pooling feature, and the second max pooling feature are fused to obtain the second initial intermediate feature.

7. The method according to claim 1, characterized in that, The initial model includes a neck network and a detection head. The neck network includes a residual separable channel extraction layer. The feature extraction process on the first initial intermediate features and the second initial intermediate features to obtain the sample image detection result includes: The first initial intermediate feature and the second initial intermediate feature are subjected to feature fusion processing to obtain the second fused feature; The residual separable channel extraction layer performs residual separable channel convolution processing and maximum average attention feature extraction on the second fused feature to obtain the neck output feature; The neck output features are input to the detection head, and the detection head outputs the sample image detection result.

8. A method for detecting chip trays, characterized in that, include: Obtain the image of the chip tray to be inspected; A pre-trained chip tray detection model is obtained, wherein the chip tray detection model is trained based on the model training method described in any one of claims 1-7; The chip tray detection model performs tray detection processing on the image to be detected to obtain the model prediction result; The model prediction results are processed for quantity detection and gap detection respectively to obtain the pallet quantity detection results and pallet gap detection results; Based on the pallet quantity detection results and pallet gap detection results, the pallet detection results are determined.

9. The method according to claim 8, characterized in that, The step of acquiring the image to be detected corresponding to the chip tray to be detected includes: Obtain the product batch number captured by the image acquisition device, perform batch number matching processing on the product batch number, and obtain the batch number matching result; When the batch number matching result is a successful match, the initial image is acquired through the image acquisition device; The initial image is subjected to template matching processing to obtain the template matching result; When the template matching result indicates that the initial image is a chip tray image, the initial image is subjected to image correction processing to obtain a corrected image; The region of interest is cropped from the corrected image to obtain the image to be detected.

10. The method according to claim 8, characterized in that, The chip tray detection model includes a backbone network, a neck network, and a detection head; the process of performing tray detection processing on the image to be detected by the chip tray detection model to obtain the model prediction result includes: The original model input features of the image to be detected are subjected to residual separable channel convolution processing through the backbone network to obtain the first image channel convolution features; Based on the original model input features and the first image channel convolution features, a second image channel convolution feature is generated. Maximum average attention feature extraction is performed on the convolutional features of the second image channel to obtain the first intermediate feature; The first intermediate feature is subjected to enhanced receptive field feature extraction to obtain the second intermediate feature; The neck network performs feature extraction processing on the first intermediate feature and the second intermediate feature to obtain the neck output image features; The neck output image features are input into the detection head, and the detection head outputs the model prediction result.

11. The method according to any one of claims 8-10, characterized in that, The method further includes: When the tray detection result indicates that the tray is in normal condition, the tray for the chip to be tested is subjected to a tape-applying operation. When the tray detection result indicates that the tray is in an abnormal state, the tape-making machine operation is interrupted, the chip tray to be tested is identified as an abnormal chip tray, and an alarm signal is issued. The abnormal chip tray is pushed to the product management system.

12. A model training device, characterized in that, include: The sample image acquisition module is used to acquire a pre-constructed initial model and sample detection images. The sample detection images include a tray area identifier, which includes a tray position identifier and a tray gap identifier. The channel convolution feature determination module is used to generate a second channel convolution feature from the initial model based on the original input features of the sample detection image and the first channel convolution feature. The first channel convolution feature is obtained by performing residual separable channel convolution processing on the original input features. The first intermediate feature extraction module is used to perform maximum average attention feature extraction on the second channel convolutional features to obtain the first initial intermediate features; The second intermediate feature extraction module is used to perform enhanced receptive field feature extraction on the first initial intermediate feature to obtain the second initial intermediate feature; The sample detection result determination module is used to perform feature extraction processing on the first initial intermediate feature and the second initial intermediate feature to obtain the sample image detection result, which includes the tray quantity detection result and the tray gap detection result. The model training module is used to train the initial model based on the difference between the sample image detection result and the tray area identifier to obtain the chip tray detection model.

13. A chip tray detection device, characterized in that, include: The image acquisition module is used to acquire the image of the chip tray to be inspected. The model acquisition module is used to acquire a pre-trained chip tray detection model, wherein the chip tray detection model is trained based on the model training method described in any one of claims 1-7; The model prediction module is used to perform tray detection processing on the image to be detected by the chip tray detection model to obtain the model prediction result; The quantity and gap detection module is used to perform quantity detection processing and gap detection processing on the model prediction results respectively to obtain the pallet quantity detection result and the pallet gap detection result. The detection result determination module is used to determine the pallet detection result based on the pallet quantity detection result and the pallet gap detection result.

14. An electronic device, characterized in that, include: processor; as well as A memory storing computer-readable instructions, which, when executed by the processor, implement the model training method as described in any one of claims 1 to 7, or the chip tray detection method as described in any one of claims 8 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the model training method as described in any one of claims 1 to 7, or the chip tray detection method as described in any one of claims 8 to 11.

Images