Material detection method and device and electronic equipment

Abstract

The invention provides a material detection method and device and electronic equipment, and belongs to the field of data processing, and the method comprises the steps: obtaining a material image, and inputting the material image into a target detection model; carrying out convolution on the material image through a CBH module, and extracting an initial feature map; performing deep convolution and point-by-point convolution on the received feature map through a channel separable convolution layer, and performing batch normalization and nonlinear transformation on the convoluted feature map through an NAM attention sub-module; and a prediction result is generated through processing of the neck network and the head network in sequence, so that the number of the automobile devices in the material image can be counted. Therefore, the target detection model has a backbone network which is simpler, less in calculation amount and higher in accuracy, so that the calculation amount when the target detection model detects the material image is greatly reduced, the detection speed is improved, and meanwhile, the target detection model has higher accuracy. And furthermore, the detection management cost can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a material detection method, device and electronic equipment. Background Art

[0002] In the production plants of the automotive electronics industry, due to poor management or the ever-changing market supply and demand, components, materials and finished products often have inventory backlogs. These stagnant inventories not only occupy a large amount of working capital, but also require additional storage and management resources, which seriously affects the company's benefits. For a long time, how to quickly and accurately count and manage these stagnant inventories has become one of the key issues to improve operational efficiency and decision-making quality.

[0003] Existing automotive component counting technology solutions include manual counting, barcode / QR code scanning, radio frequency identification, and automated warehouse management systems. Manual counting is inefficient and prone to human error. Barcode / QR code scanning requires close-range scanning, which is still limited in efficiency when processing large quantities of materials, and the scanning accuracy is low. Radio frequency identification is expensive. Automated warehouse management systems are expensive.

[0004] Therefore, there is an urgent need for a detection method with fast detection speed and high accuracy. Summary of the invention

[0005] In view of this, an object of the present invention is to provide a material detection method, device and electronic equipment, which can detect automobile components with higher accuracy and faster detection speed, and reduce detection management costs.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the present invention provides a material detection method, the method comprising:

[0008] Acquire a material image and input the material image into a target detection model; wherein the target detection model includes a backbone network, a neck network, and a head network, the backbone network includes a CBH module and a plurality of feature extraction modules connected in sequence, and the feature extraction module includes a channel-separable convolutional layer and a NAM attention submodule;

[0009] The material image is convolved through the CBH module, and the initial feature map is extracted; the received feature map is deep-convolved and point-by-point convolution is performed through the channel-separable convolution layer, and the convolved feature map is batch-normalized and nonlinearly transformed through the NAM attention submodule; the feature map output by the backbone network is sequentially connected, upsampled, and feature-extracted through the neck network; the dimension of the feature map output by the neck network is adjusted through the head network, and a prediction result is generated;

[0010] According to the prediction result, the quantity of the target material in the material image is counted.

[0011] Optionally, the method for acquiring the target detection model includes:

[0012] Label each image sample, and divide the labeled image samples into training set, validation set and test set;

[0013] Constructing a network structure of the model to obtain a neural network model; wherein the neural network model includes a backbone network, a neck network and a head network, the backbone network includes a CBH module and a plurality of feature extraction modules connected in sequence, and the feature extraction module includes a channel-separable convolutional layer and a NAM attention submodule connected in sequence;

[0014] Based on the training set and the validation set, the neural network model is iteratively trained using the Focal-EIoU function to obtain multiple candidate neural network models;

[0015] The test set is used to test each candidate neural network model, and based on the test results, the optimal neural network model is selected as the target detection model.

[0016] Optionally, the step of iteratively training the neural network model based on the training set and the validation set using a Focal-EIoU loss function to obtain a plurality of candidate neural network models includes:

[0017] Extracting a plurality of image samples from the training set and splicing the plurality of image samples to obtain a training image;

[0018] Inputting the training image into the latest neural network model;

[0019] The training image is convolved by the CBH module to extract and output an initial feature map; the received feature map is convolved by the channel-separable convolution layer, and the convolved feature map is batch normalized and nonlinearly transformed by the NAM attention submodule; the feature map output by the backbone network is sequentially connected, upsampled, and feature extracted by the neck network; the dimension of the feature map output by the neck network is adjusted by the head network, and a prediction result is generated;

[0020] Based on the prediction result, a loss value is obtained using a Focal-EIoU loss function, and according to the loss value, parameters of the neural network model are adjusted to obtain a new neural network model;

[0021] Extracting an image sample from the verification set, and inputting the image sample into the intermediate model, and obtaining the latest verification value of the neural network model according to the prediction result of the intermediate model on the image sample;

[0022] Update the number of iterations, and determine whether the number of iterations reaches a preset target number;

[0023] If so, the neural network model with the best verification value and the current latest neural network model are taken as candidate results;

[0024] If not, the process returns to the step of extracting a plurality of image samples from the training set and splicing the plurality of image samples to obtain a training image.

[0025] Optionally, the NAM attention submodule includes a batch normalization layer and a Sigmoid activation function;

[0026] The NAM attention submodule includes a batch normalization layer and a Sigmoid activation function;

[0027] The step of performing batch normalization and nonlinear transformation on the convolved feature map by the NAM attention submodule comprises:

[0028] The convolutional feature map is normalized by the batch normalization layer to extract the scaling factor of each channel;

[0029] Combining the scaling factors, calculating the weight of each channel;

[0030] Each scaling factor is multiplied by the weight of the corresponding channel, and a nonlinear transformation is performed through the Sigmoid activation function to obtain an output feature map of the backbone network.

[0031] Optionally, the step of calculating the weight of each channel in combination with each scaling factor includes:

[0032] For each channel, a ratio of the scaling factor of the channel to the sum of the scaling factors is calculated to obtain a weight of the channel.

[0033] Optionally, the prediction result includes a prediction bounding box and a category confidence;

[0034] The step of counting the number of target objects in the material image according to the prediction result comprises:

[0035] For each predicted bounding box, determining the category of the predicted bounding box according to the category confidence corresponding to the predicted bounding box;

[0036] The number of predicted bounding boxes whose categories are consistent with the category of the target material is counted.

[0037] Optionally, the step of performing depthwise convolution and pointwise convolution on the received feature map through the channel-separable convolution layer comprises:

[0038] Through the channel-separable convolution layer, each input channel of the received feature map is convolved separately, and each feature obtained by the separate convolution is mapped to the same output channel to obtain a new feature map.

[0039] Optionally, the CBH module includes a convolutional layer, a batch normalization layer and an activation function connected in sequence.

[0040] In a second aspect, the present invention provides a material detection device, including an image input module, a model processing module and a statistical module;

[0041] The image input module is used to obtain a material image and input the material image into a target detection model; wherein the target detection model includes a backbone network, a neck network and a head network, the backbone network includes a CBH module and a plurality of feature extraction modules connected in sequence, and the feature extraction module includes a channel separable convolutional layer and a NAM attention submodule;

[0042] The model processing module is used to convolve the material image through the CBH module and extract the initial feature map; perform depth convolution and point-by-point convolution on the received feature map through the channel separable convolution layer, and perform batch normalization and nonlinear transformation on the convolved feature map through the NAM attention submodule; sequentially connect, upsample and extract features from the feature map output by the backbone network through the neck network; adjust the dimension of the feature map output by the neck network through the head network, and generate a prediction result;

[0043] The statistical module is used to count the number of target materials in the material image according to the prediction result.

[0044] In a third aspect, the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the material detection method as described in the first aspect.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the material detection method as described in the first aspect.

[0046] In a fifth aspect, the present invention provides a computer program product having a computer-readable storage medium, on which instructions are embedded. When the instructions are implemented by a computing unit, the instructions are implemented, and the computing unit is configured to implement the material detection method as described in the first aspect.

[0047] The material detection method, device and electronic device provided by the embodiment of the present invention include: obtaining a material image and inputting the material image into a target detection model; convolving the material image through a CBH module and extracting an initial feature map; performing deep convolution and point-by-point convolution on the received feature map through a channel-separable convolution layer, and performing batch normalization and nonlinear transformation on the convolved feature map through a NAM attention submodule; sequentially connecting, upsampling and feature extracting the feature map output by the backbone network through a neck network; adjusting the dimension of the feature map output by the neck network through a head network and generating a prediction result; and counting the number of automotive components in the material image according to the prediction result. In this way, the above-mentioned target detection model has a simpler, less computationally intensive and more accurate backbone network, that is, a lighter backbone network, thereby greatly reducing the computational complexity of the target detection model when detecting the material image, improving the detection speed, and having higher accuracy. Furthermore, the detection management cost can be reduced.

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0050] Figure 1A simplified structural diagram of a material detection system provided in an embodiment of the present invention is shown.

[0051] Figure 2 A block diagram of an electronic device provided by an embodiment of the present invention is shown.

[0052] Figure 3 One of the flow charts of the material detection method provided in the embodiment of the present invention is shown.

[0053] Figure 4 A schematic diagram of the structure of a feature extraction module provided by an embodiment of the present invention is shown.

[0054] Figure 5 Shows Figure 3 Schematic diagram of the process flow of some sub-steps of step 13.

[0055] Figure 6 A schematic diagram of the structure of the NAM attention submodule provided in an embodiment of the present invention is shown.

[0056] Figure 7 A schematic diagram of the structure of a target detection model provided by an embodiment of the present invention is shown.

[0057] Figure 8 The second flowchart of the material detection method provided in the embodiment of the present invention is shown.

[0058] Fig. 9 Shows Figure 8 Schematic diagram of the process of some sub-steps of step 25.

[0059] Fig.10 A block diagram of a material detection device provided by an embodiment of the present invention is shown.

[0060] Explanation of the accompanying drawings: 10 - material detection system; 110 - camera; 120 - conveyor belt; 130 - detection equipment; 20 - electronic equipment; 210 - memory; 220 - processor; 230 - communication module; 30 - material detection device; 310 - image input module; 320 - model processing module; 330 - statistical module. DETAILED DESCRIPTION

[0061] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0062] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0063] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0064] The material detection method provided by the embodiment of the present invention can be applied to Figure 1 In the material detection system 10 shown, the material detection system 10 can be a subsystem of an automated warehouse management system for materials such as automotive components, fruits, and vegetables. The material detection system 10 includes a camera 110, a conveyor belt 120, and a detection device 130. The detection device 130 can be connected to the camera through any connection method such as the Internet, WiFi, and Bluetooth.

[0065] The conveyor belt 120 is used to transport materials.

[0066] The camera 110 is used to photograph the materials on the conveyor belt 120 to obtain material images.

[0067] The detection device 130 is used to obtain the material image captured by the camera 110 in real time and implement the material detection method provided in the embodiment of the present invention.

[0068] The detection device 130 may be, but is not limited to, a computer device, a programmable logic controller, a single-chip microcomputer, a field programmable gate array, etc.

[0069] Please refer to Figure 2 , is a block diagram of an electronic device 20, which may be Figure 1The detection device 130 in the material detection system 10 shown. The electronic device 20 includes a memory 210, a processor 220 and a communication module 230. The memory 210, the processor 220 and the communication module 230 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0070] The memory 210 is used to store programs or data and can be, but not limited to, a random access memory, a read-only memory, a programmable read-only memory, an erasable read-only memory, an electrically erasable read-only memory, and the like.

[0071] The processor 220 is used to read / write data or programs stored in the memory 210 and execute corresponding functions. For example, Figure 1 In the material detection system 10 shown, the processor 220 of the detection device 130 reads the computer program stored in the memory 210 to implement the material detection method provided in the embodiment of the present invention.

[0072] The communication module 230 is used to establish a communication connection between the electronic device 20 and other communication terminals, and to send and receive data. Figure 1 In the material detection system 10 shown, the detection device 130 can send and receive data with the camera 110.

[0073] It should be understood that Figure 2 The structure shown is only a schematic diagram of the structure of the electronic device 20. The electronic device 20 may also include Figure 2 More or fewer components as shown, or with Figure 2 Different configurations shown. Figure 2 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0074] In order to improve the current problems of slow counting speed, low accuracy, low efficiency, or high cost of materials such as automotive components, the present invention provides a material detection method, referring to Figure 3 , including steps 11 to 17. So that: Figure 1 In the material detection system 10 shown, the detection device 130 can Figure 2 The structure shown implements step 11 to step 17 when the processor 220 executes the computer program stored in the memory 210.

[0075] Step 11: Obtain a material image and input the material image into the target detection model.

[0076] Among them, the target detection model includes a backbone network, a neck network and a head network. The backbone network includes a CBH module and multiple feature extraction modules connected in sequence. The feature extraction module includes a channel-separable convolutional layer and a NAM attention sub-module.

[0077] The neck network and the head network of the target detection model can be respectively consistent with the neck network and the head network of the yolov5 model, that is, the target detection model can be a model obtained by training after replacing the backbone network of the yolov5 model with a backbone network including a CBH module and multiple feature extraction modules connected in sequence.

[0078] Step 13, convolve the material image through the CBH module and extract the initial feature map; perform depth convolution and point-by-point convolution on the received feature map through the channel-separable convolution layer, and perform batch normalization and nonlinear transformation on the convolved feature map through the NAM attention submodule.

[0079] Step 15, the feature map output by the backbone network is sequentially connected, upsampled, and feature extracted through the neck network; the dimension of the feature map output by the neck network is adjusted through the head network, and a prediction result is generated.

[0080] Step 17: Count the number of target materials in the material image based on the prediction results.

[0081] The target material can be any material that needs to be counted, such as automotive components, fruits, etc. The NAM attention submodule is a channel attention submodule based on the NAM attention mechanism.

[0082] For example, in combination Figure 1 In the material detection system 10 shown, when it is necessary to count the quantity of materials that are about to enter or leave the warehouse in the automated warehouse management system, the camera 110 shoots the conveyor belt 120 to obtain a video stream or a material image. When it is a video stream, the detection device 130 periodically intercepts the material image, and when the material image is received, the detection device 130 directly receives it.

[0083] The detection device 130 inputs the material image acquired in real time into the target detection model: the material image is convolved through the CBH module, and the initial feature map is extracted, and the feature map is input into the channel-separable convolution layer of the first feature extraction module; the feature map is deep-convolved and point-wise-convolved through the channel-separable convolution layer, and the convolved feature map is input into the NAM attention submodule of the first feature extraction module; the convolved feature map is batch-normalized and nonlinearly transformed through the NAM attention submodule, and the feature map obtained is then input into the second feature extraction module; after the same processing as the first feature extraction module, the feature map extracted by the second feature extraction module is input into the third feature extraction module, and the above process is repeated until the last feature extraction module, and the feature map output by the last feature extraction module (i.e., the feature map output by the backbone network) is input into the neck network; the feature map output by the backbone network is sequentially connected, upsampled, and feature-extracted through the neck network, and the current feature map obtained is input into the head network; the dimension of the feature map output by the neck network is adjusted through the head network, and a prediction result is generated. The prediction results include multiple material bounding boxes in the material image and the material category of each material bounding box.

[0084] Therefore, the detection device 130 counts the number of material boundary boxes whose material categories are the target categories in the material image according to the prediction results, and thus the number of the target materials can be obtained.

[0085] The depth convolution operation is to apply a K×K convolution kernel to each input channel separately, that is, the depth convolution does not mix the features of different channels, but only performs convolution operations independently on each channel. Therefore, when the input feature map has M channels, the output feature map has N channels, and the height and width of the input feature map are H and W respectively, the computational complexity of the depth convolution is H×W×K 2 ×M.

[0086] The point-by-point convolution is a 1×1 convolution kernel, which is used to map the features generated by the depth convolution to the output channel. At this point, each output pixel combines the information of all input channels. Therefore, the point-by-point convolution is H×W×N×M.

[0087] Therefore, the total computational effort of the channel-separable convolutional layer is: FLOPS = H × W × K 2 ×M+H×W×N×M.

[0088] The traditional YOLOv5 model backbone network includes a 5-layer downsampling module (CONV), a 4-layer feature extraction module C3 and an SPPF module connected in sequence. When the input feature map has M channels and the output feature map has N channels, the computational complexity of the feature extraction module C3 is: FLOPS = H × W × K 2 ×M×N.

[0089] Compared with the computational complexity of the feature extraction module C3 in the traditional yolov5 model, the computational complexity of the feature extraction module of the target detection model in the embodiment of the present invention is reduced by at least: When K=3 (usually K is 3 or 5), and usually N is greater than K 2 The number of computations is much larger, so the amount of calculation is reduced by at least 8-9 times. As a result, the detection time of the model can be shortened and the detection speed can be improved.

[0090] In addition, the backbone network of the target detection model provided by the embodiment of the present invention is a layer of CBH modules and multiple feature extraction modules. The feature extraction module also includes a NAM attention submodule, that is, an attention mechanism is introduced, which greatly improves the detection accuracy.

[0091] In order to enable the model to be trained at a deeper level, reduce the gradient vanishing problem, ensure the efficiency and accuracy of the model in a resource-constrained environment, and further improve the accuracy and efficiency of material detection. The CBH module includes a convolutional layer, a batch normalization layer, and an activation function connected in sequence. The convolutional layer uses a 3X3 convolution kernel to extract features, the batch normalization layer is used to accelerate the training speed of the neural network and improve the stability of the model, and the activation function is used to effectively improve the nonlinear expression ability of the model.

[0092] The activation function can be flexibly selected, for example, it can be a Sigmoid function, a ReLU function, or any other activation function.

[0093] In order to effectively improve the nonlinear expression ability of the model, the activation function of the CBH module is the Hard-Swish function. At this time, the calculation formula of the CBH module can be expressed as: HS(x)=x*ReLU6(x+3) / 6, ReLU6() represents the activation function, x represents the input material image, and HS(x) represents the output.

[0094] The feature map output by the CBH module enters the first channel-separable convolution layer for depth convolution and point-by-point convolution. In step 13, the process of depth convolution and point-by-point convolution of the received feature map by the channel-separable convolution layer is as follows: through the channel-separable convolution layer, each input channel of the received feature map is convolved separately, and each feature obtained by the separate convolution is mapped to the same output channel to obtain a new feature map. In this way, a K×K convolution kernel is applied to each input channel of the feature map separately (i.e., depth convolution operation), and the feature generated by the depth convolution is mapped to the output channel (i.e., point-by-point convolution).

[0095] In this way, while ensuring the extraction of key features, the computational complexity and the number of model parameters are significantly reduced, thereby improving the detection speed.

[0096] The feature map obtained by the channel-separable convolution layer is input to the NAM attention submodule to perform attention aggregation on the input feature map. The setting of the NAM attention submodule for attention aggregation can be flexibly selected, for example, it can be a self-attention network, a multi-head attention network, or any other attention network, and its implementation method is not limited.

[0097] In order to further improve the overall performance and efficiency of the target detection model without significantly increasing the computational burden of the model, the NAM attention submodule includes a batch normalization layer and a sigmoid activation function. At this point, the structure of the feature extraction module can be as follows: Figure 4 As shown, it includes 3 channel-separable convolutional layers (i.e., CONV1, CONV2, and CONV3) and a NAM attention submodule (i.e., NAMC).

[0098] On this basis, refer to Figure 5 , the process of performing batch normalization and nonlinear transformation on the convolutional feature map through the NAM attention submodule in step 13 includes steps 131 to 135.

[0099] Step 131, normalize the convolutional feature map through a batch normalization layer to extract the scaling factor of each channel.

[0100] It should be understood that the feature map after the convolution is the feature map output by the channel-separable convolution layer. The scaling factor is the standard deviation in the batch normalization process.

[0101] Here, the processing of the batch normalization layer can be expressed by the following formula: X represents the feature map after convolution, μ B represents the mean of the mini-batch B, σ B represents the standard deviation of the mini-batch B, γ and β represent the affine transformation parameters (i.e., scaling and offset, the final parameter values ​​are determined after training), and ∈ is a constant.

[0102] Step 133, combining each scaling factor to calculate the weight of each channel.

[0103] Step 135, multiply each scaling factor by the weight of the corresponding channel, and perform nonlinear transformation through the Sigmoid activation function to obtain the output feature map of the backbone network.

[0104] After the above steps 131 to 135, the output feature map of the backbone network can be expressed as: c =SIGMOD(W i (Norm(X))), SIGMOD() represents the activation function, Norm(X) represents the output of the batch normalization layer, Wi Represents the weight of the i-th channel.

[0105] The weight may be a preset value or calculated according to various scaling factors. The above method is only an example and its implementation method is not limited.

[0106] In order to enhance the model's focus on key features and improve detection accuracy and performance, in step 133, for each channel, the ratio of the scaling factor of the channel to the sum of all scaling factors is calculated to obtain the weight of the channel.

[0107] Among them, the weight calculation formula of each channel can be: r i represents the scaling factor of the ith channel, W i represents the weight of the i-th channel.

[0108] Taking the feature map with 4 channels as an example, the processing of the NAM attention submodule can be as follows: Figure 6 shown.

[0109] Through the above steps 131 to 135, each channel of the input feature map is first normalized to make it more uniform in the numerical range, thereby improving the model's ability to focus on important channels. Then, in the processing of the Sigmoid activation function, the scaling factor in batch normalization is used as the key metric (i.e., the key indicator for measuring the degree of variation of each channel), and the weight is dynamically adjusted according to the importance of each channel in the input feature map to adjust the information content of the feature map. Thus, it is ensured that the information of the channel in the feature map is effectively weighted and adjusted to enhance the model's attention to key features while maintaining the richness of the original information.

[0110] In this way, the feature extraction module can effectively distinguish and emphasize the feature channels that are critical to the model performance, thereby improving the overall performance and efficiency of the model without significantly increasing the computational burden.

[0111] The output feature map of the backbone network obtained through the above process first enters the neck network for connection, upsampling, feature extraction and other operations, and the head network adjusts the dimension of the feature map output by the neck network and generates a prediction result.

[0112] Among them, the neck network and the head network can be the same as the neck network and the head network of any existing YOLO model, or the same as the neck network and the head network of other target detection models, and the specific structure is not limited.

[0113] When the neck network and the head network are the same as the neck network and the head network of the YOLOv5 model, respectively, the network structure of the target detection model can be as follows Figure 7 The backbone network includes CBH modules and 11 feature extraction modules (i.e., MobileNAMC layers) connected in sequence.

[0114] The neck network is a PAN structure, and based on FPN (feature pyramid), a path enhancement part is added, including the sequentially connected Conv layer (i.e., convolution layer), Upsample layer (i.e., upsampling layer), Concat layer (i.e., connection layer), C3 feature extraction module, Conv layer, Upsample layer, Concat layer (i.e., connection layer), C3 feature extraction module, Conv layer, Concat layer, C3 feature extraction module, Conv layer, Concat layer, C3 feature extraction module. The Conv layer performs convolution on the input feature map, the Conact layer is used to connect the feature map, the Upsample layer uses bilinear interpolation to upsample the input feature map, and the C3 feature extraction module is used for feature extraction.

[0115] The head network includes multiple Conv2d modules, which are 1×1 two-dimensional convolutions used to change the dimension of the feature map output by the neck network to output the prediction results.

[0116] In order to make the target detection model in the above content have better detection accuracy and detection speed, before executing the above step 11, a process of obtaining the target detection model can also be included. This process can be obtained by training the constructed initial neural network model using any training method (such as supervised training, unsupervised training, etc.).

[0117] In order to improve the prediction performance of the target detection model, an iterative training method combined with a loss function is used to train the model to obtain the target detection model. The loss function can be a CIoU loss function, a mean square error loss function, or any other loss function, and the choice is not limited.

[0118] In order to solve the problem that the regression of the predicted bounding box cannot be continuously optimized during the training process, and to improve the situation that the number of high-quality (i.e. low-error) predicted bounding boxes is far less than the low-quality predicted bounding boxes with large errors, the EIoU loss function (i.e. EIoU-Loss) and the Focal loss function (i.e. Focal-Loss) are used together in the iterative training of the target detection model, with EIoU-Loss as the loss function for bounding box detection and Focal Loss as the loss function for bounding box classification. Figure 8 The process of training the target detection model includes steps 21 to 27.

[0119] Step 21, annotate each image sample, and divide each annotated image sample into a training set, a validation set, and a test set.

[0120] For example, the Labelme tool is used to annotate the dataset of image samples, and the generated annotation file is converted into the YOLO format and divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0121] Step 23, construct the network structure of the model to obtain the neural network model.

[0122] The initial neural network model can be obtained by replacing the backbone network of the yolov5 model with a backbone network composed of a CBH module and multiple feature extraction modules. The neural network model includes a backbone network, a neck network, and a head network. The backbone network includes a CBH module and multiple feature extraction modules connected in sequence. The feature extraction module includes a channel-separable convolutional layer and a NAM attention submodule connected in sequence. At this time, the structure of the neural network model can be as follows: Figure 7 shown.

[0123] Step 25, based on the training set and the validation set, the neural network model is iteratively trained using the Focal-EIoU function to obtain multiple candidate neural network models.

[0124] Here, it should be noted that the network structures of the multiple candidate neural network models are the same, but the model parameters between each two are different.

[0125] Step 27, using the test set to test each candidate neural network model, and based on the test results, select the optimal neural network model as the target detection model.

[0126] In step 25, the training set may be used for iteration to obtain multiple models after parameter adjustment, and then the models in the validation set may be used for verification. Alternatively, the training set may be used for iterative parameter adjustment once, and then the validation set may be used for verification once, and the candidate neural network model may be obtained by looping this process. The above method is only an example, and the iterative training method is not limited.

[0127] For example, in the material images of automotive components, the number of automotive components in the image is usually much less than the background area (negative samples), which will cause serious imbalance between positive and negative samples. Therefore, in order to improve this imbalance, the model pays more attention to negative samples and ignores positive samples during training, and at the same time simplifies the testing process. Fig. 9 , step 25 may include the following steps 251 to 256.

[0128] Step 251: extract multiple image samples from the training set, and splice the multiple image samples to obtain a training image.

[0129] For example, eight image samples are extracted and concatenated into one training image.

[0130] Step 252, input the training image into the latest neural network model, and use the neural network model to generate prediction results for the training image.

[0131] The process of generating the prediction result of the training image is the same as step 13 above: the training image is convolved through the CBH module to extract and output the initial feature map; the received feature map is convolved through the channel-separable convolutional layer, and the convolved feature map is batch normalized and nonlinearly transformed through the NAM attention submodule; the feature map output by the backbone network is sequentially connected, upsampled and feature extracted through the neck network; the dimension of the feature map output by the neck network is adjusted through the head network, and the prediction result is generated.

[0132] Step 253, based on the prediction result, the Focal-EIoU loss function is used to obtain a loss value, and according to the loss value, the parameters of the neural network model are adjusted to obtain a new neural network model.

[0133] Here, the prediction result includes multiple prediction boxes, and each prediction box has a corresponding real box in the label.

[0134] Among them, the Focal-EIoU loss function can be expressed as: L FL (p t )=-α t ·(1-p t ) γ ·log(p t ), L FL (p t ) represents the final loss value, α t Represents the factor that controls the weight of positive and negative samples, p t It represents the model's predicted probability of the correct category of the prediction box, (1-p t ) is the modulation factor. When the sample is easy to separate (for example, the probability of occurrence of negative samples is close to 0), the loss weight will decrease.

[0135] IoU represents the ratio of the intersection area and the union area of ​​the predicted box and the real box, w = kw gt ,h=kh gt , w c and h c Respectively represent the width and height of the minimum bounding box covering the predicted box and the real box, w and h represent the width and height of the predicted box, respectively. GT and h GT Represent the width and height of the real box, B represents the predicted box, and BGT represents the true box, and ρ represents the Euclidean distance between the center points of the predicted box and the true box.

[0136] Any known method can be used to adjust parameters, such as regularization, discarding neurons, lowering the learning rate, etc. when the loss value is too large, which will not be described in detail here.

[0137] Step 254, extracting image samples from the verification set, and inputting the image samples into the intermediate model, and obtaining the verification value of the current latest neural network model based on the prediction result of the intermediate model for the image samples.

[0138] The verification value may be a loss value, or an intersection-over-union ratio between the true box and the predicted box, or a correct rate, etc., which is not limited here.

[0139] Step 255, update the number of iterations, and determine whether the number of iterations reaches the preset target number. If yes, execute step 256, if not, return to execute step 251.

[0140] Step 256, taking the neural network model with the best verification value and the current latest neural network model as candidate results.

[0141] Here, using the final latest neural network model and the neural network model with the final verification value as candidate models helps to reduce the amount of testing in step 27 and improve the speed of model acquisition.

[0142] In step 27, the accuracy and calculation speed of each model can be obtained, and the accuracy and calculation speed can be weighted and summed to obtain the score of each model, and the model with the highest score can be selected as the final target detection model. Alternatively, the model with the highest accuracy or the fastest speed can be selected as the final target detection model. The above method is only an example, and its implementation method is not limited.

[0143] In traditional training, CIoU Loss is used as the loss function for bounding box detection, and BCE Loss is used as the loss function for bounding box classification. When CIoU Loss predicts bounding boxes and regresses, once the true box and the predicted box are in linear proportion in terms of aspect ratio, the length and width of the predicted box cannot increase or decrease at the same time, resulting in the inability to continuously optimize the regression of the predicted box. When BCE Loss is used for classification loss calculation, the loss of negative samples will dominate, causing the model to focus more on the background and ignore the detection of the target material.

[0144] In the above steps 21 to 27, EIoU-Loss directly introduces the difference in width and height to calculate the loss, reducing complex nonlinear calculations, so it is faster and more efficient when training on the target material data set. In addition, the first two parts of EIoU-Loss continue the method in CIoU Loss. The loss of width and height minimizes the difference in width and height between the predicted box and the real box, and the phenomenon of unsustainable optimization of the predicted bounding box will not occur. Focal Loss reduces the loss contribution of easy-to-classify negative samples by adding a modulation factor, so that the model pays more attention to difficult-to-classify positive samples, and the loss weight of the target material (positive sample) is improved, so that the model pays more attention to a small number of positive samples.

[0145] Therefore, through the above training, a target detection model with higher detection accuracy, efficiency and faster detection speed is obtained.

[0146] The above target detection model can be used to obtain the prediction result of the material image, and the prediction result includes the prediction bounding box and the category confidence. After obtaining the prediction result, in step 17: for each prediction bounding box, according to the category confidence corresponding to the prediction bounding box, determine the category of the prediction bounding box; and count the number of prediction bounding boxes whose categories are consistent with the category of the target material.

[0147] Based on the same concept of the above material detection method, refer to Fig.10 The embodiment of the present invention further provides a material detection device 30, which can be applied to Figure 1 The detection device 130 of the material detection system 10 shown includes an image input module 310 , a model processing module 320 and a statistical module 330 .

[0148] The image input module 310 is used to obtain the material image and input the material image into the target detection model. The target detection model includes a backbone network, a neck network and a head network. The backbone network includes a CBH module and multiple feature extraction modules connected in sequence. The feature extraction module includes a channel separable convolution layer and a NAM attention submodule.

[0149] The model processing module 320 is used to convolve the material image through the CBH module and extract the initial feature map; perform depth convolution and point-by-point convolution on the received feature map through the channel-separable convolution layer, and perform batch normalization and nonlinear transformation on the convolved feature map through the NAM attention submodule; sequentially connect, upsample and extract features from the feature map output by the backbone network through the neck network; adjust the dimension of the feature map output by the neck network through the head network, and generate a prediction result.

[0150] The statistics module 330 is used to count the number of target materials in the material image according to the prediction results.

[0151] Furthermore, it also includes a model acquisition module, which is used to execute the above steps 21 to 27 and related sub-steps. The specific implementation and effects are described above.

[0152] The material detection device 30, through the synergy of the image input module 310, the model processing module 320 and the statistical module 330, uses a target detection model with a simpler, less computationally intensive and more accurate backbone network, i.e., a lighter backbone network, thereby greatly reducing the computational complexity of the target detection model when detecting material images, improving the detection speed, and having higher accuracy. In turn, it is possible to reduce the detection management cost.

[0153] For the specific implementation and effect of the material detection device 30, please refer to the description of the implementation of the material detection method above. For example, for the specific implementation and effect of the image input module 310, please refer to the description of the relevant content of step 11 above, for the specific implementation and effect of the model processing module 320, please refer to the description of the relevant content of step 13 and step 15 above, and for the specific implementation and effect of the statistical module 330, please refer to the description of the relevant content of step 17 above, which will not be repeated here.

[0154] In addition, each module of the material detection device 30 can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor 220 in the electronic device 20 in the form of hardware, or can be stored in the memory 210 of the electronic device 20 in the form of software, so that the processor 220 can call and execute the operations corresponding to each module to implement the material detection method provided above.

[0155] The embodiment of the present invention also provides an electronic device 20, including a processor 220 and a memory 210, wherein the memory 210 stores a computer program that can be executed by the processor 220, and the processor 220 can execute the computer program to implement the material detection method proposed in the embodiment of the present invention.

[0156] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by the processor 220, the material detection method proposed in the embodiment of the present invention is implemented.

[0157] The material detection method, device, electronic device, and storage medium provided by the embodiments of the present invention have at least the following beneficial effects:

[0158] (1) The backbone network of the target detection model consists of a CBH module and a feature extraction module. Through this backbone network design, the model significantly reduces the number of parameters and computation while maintaining detection accuracy;

[0159] (2) The bounding box loss function uses EIoU Loss, which further reduces the computational overhead and solves the problem that CIoU Loss cannot continuously optimize the bounding box during training;

[0160] (3) To cope with the imbalance of positive and negative samples in model training, the present invention introduces Focal-Loss in the bounding box regression process, which further improves the detection accuracy of automotive components;

[0161] (4) A system equipped with the above object detection model can automatically identify and count different types of materials (such as automotive components), eliminating the need to scan physical labels one by one. This can not only significantly improve the counting speed, especially when processing a large number of components, but also reduce human errors.

[0162] (5) It has strong environmental adaptability and can operate in complex production and storage environments, such as places with high temperature, high humidity or uneven lighting;

[0163] (6) It can realize the automation and intelligent counting of materials such as automotive components, which helps to improve the management efficiency of the entire production and supply chain.

[0164] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0165] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0166] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory 210, a random access memory 210, a magnetic disk or an optical disk.

[0167] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A material detection method, characterized in that: The method comprises: Acquire a material image and input the material image into a target detection model; wherein the target detection model includes a backbone network, a neck network, and a head network, the backbone network includes a CBH module and a plurality of feature extraction modules connected in sequence, and the feature extraction module includes a channel-separable convolutional layer and a NAM attention submodule; The material image is convolved through the CBH module, and the initial feature map is extracted; the received feature map is deep-convolved and point-by-point convolution is performed through the channel-separable convolution layer, and the convolved feature map is batch-normalized and nonlinearly transformed through the NAM attention submodule; the feature map output by the backbone network is sequentially connected, upsampled, and feature-extracted through the neck network; the dimension of the feature map output by the neck network is adjusted through the head network, and a prediction result is generated; According to the prediction result, the quantity of the target material in the material image is counted.

2. The material detection method according to claim 1, characterized in that: The method for acquiring the target detection model includes: Label each image sample, and divide the labeled image samples into training set, validation set and test set; Constructing a network structure of the model to obtain a neural network model; wherein the neural network model includes a backbone network, a neck network and a head network, the backbone network includes a CBH module and a plurality of feature extraction modules connected in sequence, and the feature extraction module includes a channel-separable convolutional layer and a NAM attention submodule connected in sequence; Based on the training set and the validation set, the neural network model is iteratively trained using the Focal-EIoU function to obtain multiple candidate neural network models; The test set is used to test each candidate neural network model, and based on the test results, the optimal neural network model is selected as the target detection model.

3. The material detection method according to claim 2, characterized in that: The step of iteratively training the neural network model based on the training set and the validation set using the Focal-EIoU loss function to obtain multiple candidate neural network models includes: Extracting a plurality of image samples from the training set and splicing the plurality of image samples to obtain a training image; Inputting the training image into the latest neural network model; The training image is convolved by the CBH module to extract and output an initial feature map; the received feature map is convolved by the channel-separable convolution layer, and the convolved feature map is batch normalized and nonlinearly transformed by the NAM attention submodule; the feature map output by the backbone network is sequentially connected, upsampled, and feature extracted by the neck network; the dimension of the feature map output by the neck network is adjusted by the head network, and a prediction result is generated; Based on the prediction result, a loss value is obtained using a Focal-EIoU loss function, and according to the loss value, parameters of the neural network model are adjusted to obtain a new neural network model; Extracting an image sample from the verification set, and inputting the image sample into the intermediate model, and obtaining the latest verification value of the neural network model according to the prediction result of the intermediate model on the image sample; Update the number of iterations, and determine whether the number of iterations reaches a preset target number; If so, the neural network model with the best verification value and the current latest neural network model are taken as candidate results; If not, the process returns to the step of extracting a plurality of image samples from the training set and splicing the plurality of image samples to obtain a training image.

4. The material detection method according to claim 1, characterized in that: The NAM attention submodule includes a batch normalization layer and a Sigmoid activation function; The step of performing batch normalization and nonlinear transformation on the convolved feature map by the NAM attention submodule comprises: The convolutional feature map is normalized by the batch normalization layer to extract the scaling factor of each channel; Combining the scaling factors, calculating the weight of each channel; Each scaling factor is multiplied by the weight of the corresponding channel, and a nonlinear transformation is performed through the Sigmoid activation function to obtain an output feature map of the backbone network.

5. The material detection method according to claim 4, characterized in that: The step of calculating the weight of each channel in combination with each scaling factor comprises: For each channel, a ratio of the scaling factor of the channel to the sum of the scaling factors is calculated to obtain a weight of the channel.

6. The material detection method according to any one of claims 1 to 5, characterized in that: The prediction result includes a prediction bounding box and a category confidence; The step of counting the number of target objects in the material image according to the prediction result comprises: For each predicted bounding box, determining the category of the predicted bounding box according to the category confidence corresponding to the predicted bounding box; The number of predicted bounding boxes whose categories are consistent with the category of the target material is counted.

7. The material detection method according to any one of claims 1 to 5, characterized in that: The step of performing depth convolution and point-by-point convolution on the received feature map through the channel-separable convolution layer comprises: Through the channel-separable convolution layer, each input channel of the received feature map is convolved separately, and each feature obtained by the separate convolution is mapped to the same output channel to obtain a new feature map.

8. The material detection method according to any one of claims 1 to 5, characterized in that: The CBH module includes a convolutional layer, a batch normalization layer and an activation function connected in sequence.

9. A material detection device, characterized in that: It includes image input module, model processing module and statistical module; The image input module is used to obtain a material image and input the material image into a target detection model; wherein the target detection model includes a backbone network, a neck network and a head network, the backbone network includes a CBH module and a plurality of feature extraction modules connected in sequence, and the feature extraction module includes a channel separable convolutional layer and a NAM attention submodule; The model processing module is used to convolve the material image through the CBH module and extract the initial feature map; perform depth convolution and point-by-point convolution on the received feature map through the channel separable convolution layer, and perform batch normalization and nonlinear transformation on the convolved feature map through the NAM attention submodule; sequentially connect, upsample and extract features from the feature map output by the backbone network through the neck network; adjust the dimension of the feature map output by the neck network through the head network, and generate a prediction result; The statistical module is used to count the number of target materials in the material image according to the prediction result.

10. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the material detection method according to any one of claims 1 to 8.

Images