A product intelligent identification method and device

Abstract

The present application provides a method and device for intelligent product recognition, which can be used in the field of image recognition or other fields. The method includes: inputting a product image into a shallow feature extraction model and a deep feature extraction model respectively to obtain a shallow feature image and a deep feature image corresponding to the product image, and inputting the pre-trained channel attention model respectively to obtain a corresponding weighted shallow feature image and a weighted deep feature image; fusing the shallow feature image and the weighted shallow feature image to obtain a first feature image; fusing the deep feature image and the weighted deep feature image to obtain a second feature image; fusing the first feature image and the second feature image to obtain a third feature image, and identifying the category of the product in the product image according to the third feature image. The device is used to execute the method. The method and device for intelligent product recognition provided by the present application realize fast and accurate intelligent recognition of product categories, and have low requirements for GPU.

Description

Technical Field

[0001] The present application relates to the field of image recognition, and in particular to a method and device for intelligent product recognition. Background Art

[0002] In recent years, with the widespread adoption of machine learning algorithms, the demand for AI recognition technology has become increasingly diverse. Automatically identifying product types and quality requires comprehensive, end-to-end technical and equipment support. Traditionally, when classifying and pricing products like plants, people relied primarily on experience and knowledge from books to distinguish the product's color, type, and quality grade, thereby guiding the selling price. However, due to the limited level of human cognition and the lack of uniformity in everyone's judgment standards, it is difficult to obtain scientific price guidance. Currently, many product recognition apps and devices have emerged, using specific evaluation criteria to make product identification more scientific and labor-saving. However, ensuring the scientific nature of product recognition requires not only dedicated hardware equipment, but also the computational accuracy and sophistication of the product recognition algorithm.

[0003] Product image classification algorithms have long been a hot topic in deep learning. They not only meet the needs of everyday product identification but also reduce the workload for professionals. Traditional product identification relies primarily on three features: color, shape, and texture. This method requires manual classification of these features and suffers from low accuracy.

[0004] Existing technologies often use convolutional neural networks for product recognition, but accuracy is low due to interference from factors such as shooting angle and product orientation. Existing methods also often lose a significant amount of positional information during feature extraction, resulting in reduced recognition accuracy for images after rotational transformations. Increasing the number of neural network layers to improve recognition accuracy can lead to overfitting and gradient explosion. While using dropout and batch normalization to address these issues can also lead to a surge in computational complexity, potentially causing GPU memory overflow. Summary of the Invention

[0005] In response to the problems in the prior art, the embodiments of the present application provide a product intelligent identification method and device, which can at least partially solve the problems in the prior art.

[0006] In a first aspect, the present application provides a product intelligent identification method, comprising:

[0007] Input the product image into the shallow feature extraction model and the deep feature extraction model respectively to obtain the shallow feature image and the deep feature image;

[0008] Inputting the shallow feature image and the deep feature image into a pre-trained channel attention model respectively to obtain corresponding weighted shallow feature image and weighted deep feature image;

[0009] fusing the shallow feature image and the weighted shallow feature image to obtain a first feature image; the shallow feature image, the weighted shallow feature image and the first feature image have the same size and number of channels;

[0010] fusing the deep feature image and the weighted deep feature image to obtain a second feature image; the deep feature image, the weighted deep feature image and the second feature image have the same size and number of channels;

[0011] The first feature image and the second feature image are fused to obtain a third feature image, and the category of the product in the product image is identified according to the third feature image.

[0012] Among them, the shallow feature extraction model includes m shallow convolution modules connected in sequence, and the product picture is input into the shallow feature extraction model and the deep feature extraction model respectively to obtain a shallow feature image and a deep feature image correspondingly, including: inputting the product picture into the m shallow convolution modules connected in sequence, performing splicing and segmentation operations, and outputting the shallow feature image; among the m shallow convolution modules, the input of the nth shallow convolution module is the output of the n-1th shallow convolution module, wherein n is greater than or equal to 2 and less than or equal to m.

[0013] The shallow feature extraction model includes: a first shallow convolution module, a second shallow convolution module, a third shallow convolution module, a fourth shallow convolution module, and a fifth shallow convolution module; the product image is input into the shallow feature extraction model and the deep feature extraction model respectively to obtain a shallow feature image and a deep feature image, including:

[0014] Inputting the product image into the first shallow convolution module to obtain multiple first shallow intermediate images;

[0015] Inputting the plurality of first shallow intermediate images into the second shallow convolution module respectively to obtain a plurality of second shallow intermediate images;

[0016] splicing the plurality of second shallow intermediate images and inputting the convolutional layers into the third shallow convolutional module to obtain a third shallow intermediate image;

[0017] Inputting a plurality of third shallow layer segmented images obtained by segmenting the third shallow layer intermediate image into the fourth shallow layer convolution module to obtain a plurality of fourth shallow layer intermediate images;

[0018] Inputting each fourth shallow intermediate image into the fifth shallow convolution module to obtain a plurality of fifth shallow intermediate images;

[0019] The fifth shallow intermediate images are spliced together to obtain the shallow feature image.

[0020] The step of inputting the product image into the deep feature extraction model to obtain the deep feature image includes:

[0021] Inputting the product image into a backbone network to obtain a first deep intermediate image and a second deep intermediate image; the backbone network includes a first preset number of first deep convolution modules; the second deep intermediate image is obtained by passing the first deep intermediate image through a second preset number of first deep convolution modules;

[0022] splicing the first deep intermediate image and the second deep intermediate image to obtain a third deep intermediate image;

[0023] The third deep intermediate image is input into a deep feature extraction module to obtain the deep feature image.

[0024] Among them, the step of sequentially inputting the third deep intermediate image into the deep feature extraction module to obtain the deep feature image includes: sequentially inputting the third deep intermediate image into the second deep convolution module, the convolution layer and the maximum pooling layer to obtain the deep feature image.

[0025] The step of inputting the shallow feature image and the deep feature image into a pre-trained channel attention model to obtain corresponding weighted shallow feature images and weighted deep feature images comprises:

[0026] Inputting the shallow feature image into a pre-trained channel attention model to obtain a corresponding weighted shallow feature image;

[0027] The deep feature image is input into a pre-trained channel attention model to obtain a corresponding weighted deep feature image.

[0028] The channel attention model includes: a first weight feature acquisition module and a second weight feature acquisition module; the inputting of the shallow feature image into the pre-trained channel attention model to obtain the corresponding weighted shallow feature image includes:

[0029] Inputting the shallow feature image into the first weight feature acquisition module to obtain a first shallow weight feature;

[0030] Inputting the shallow feature image into the second weight feature acquisition module to obtain a second shallow weight feature;

[0031] activating a third shallow weight feature obtained by fusing the first shallow weight feature and the second shallow weight feature to obtain a third shallow weight; the dimension of the third shallow weight is equal to the number of channels of the shallow feature image;

[0032] The third shallow layer weight is multiplied by each channel of the shallow layer feature image to obtain the weighted shallow layer feature image.

[0033] The channel attention model includes: a first weight feature acquisition module and a second weight feature acquisition module; the inputting the deep feature image into the pre-trained channel attention model to obtain the corresponding weighted deep feature image includes:

[0034] Inputting the deep feature image into the first weight feature acquisition module to obtain a first deep weight feature;

[0035] Inputting the deep feature image into the second weight feature acquisition module to obtain a second deep weight feature;

[0036] activating a third deep weight feature obtained by fusing the first deep weight feature and the second deep weight feature to obtain a third deep weight; the dimension of the third deep weight is equal to the number of channels of the deep feature image;

[0037] The third deep weight is multiplied by each channel of the deep feature image to obtain the weighted deep feature image.

[0038] In a second aspect, the present application provides a product intelligent identification device, comprising:

[0039] The shallow and deep feature image extraction units are used to input the product image into the shallow feature extraction model and the deep feature extraction model respectively to obtain the shallow feature image and the deep feature image respectively;

[0040] A shallow and deep weighted image acquisition unit, configured to input the shallow feature image and the deep feature image into a pre-trained channel attention model to obtain corresponding weighted shallow feature images and weighted deep feature images;

[0041] A first feature image acquisition unit is configured to fuse the shallow feature image and the weighted shallow feature image to obtain a first feature image; the shallow feature image, the weighted shallow feature image, and the first feature image have the same size and number of channels;

[0042] A second feature image acquisition unit is configured to fuse the deep feature image and the weighted deep feature image to obtain a second feature image; the deep feature image, the weighted deep feature image, and the second feature image have the same size and number of channels;

[0043] A product classification recognition unit is configured to fuse the first feature image and the second feature image to obtain a third feature image, and recognize the category of the product in the product image according to the third feature image.

[0044] In a third aspect, the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any one of the above embodiments when executing the computer program.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any of the above embodiments is implemented.

[0046] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the method described in any of the above embodiments.

[0047] The product intelligent recognition method and device provided by the present application respectively inputs the product image into a shallow feature extraction model and a deep feature extraction model to obtain a shallow feature image and a deep feature image, and respectively inputs the pre-trained channel attention model to obtain a corresponding weighted shallow feature image and a weighted deep feature image, fuses the shallow feature image and the weighted shallow feature image to obtain a first feature image, fuses the deep feature image and the weighted deep feature image to obtain a second feature image, and fuses the first feature image and the second feature image to obtain a third feature image. The classification of the product in the product image is identified based on the third feature image, thereby realizing intelligent recognition of the product. While obtaining the deep features of the image, a simple network model is used to obtain shallow features and fuse them with the deep features, fully retaining the detailed features and avoiding the occurrence of overfitting. The channel self-attention mechanism is incorporated to allocate different attention to different channels, and the target information is further highlighted based on the weighted image, thereby improving the accuracy of the model in recognizing products. The product image is enhanced before recognition, and the position features of the image are retained during the feature extraction process, further ensuring the accuracy of product recognition and preventing factors such as shooting angle from interfering with the model. The model has fewer parameters and has lower performance requirements for the GPU. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 A schematic diagram of the overall structure of a product intelligent recognition model provided in some embodiments of the present application;

[0050] Figure 2 A schematic diagram of the structure of the self-channel attention model (CSA) provided in some embodiments of the present application;

[0051] Figure 3 A flowchart of a product intelligent identification method provided in some embodiments of the present application;

[0052] Figure 4 A schematic diagram of the structure of a shallow feature extraction model provided in some embodiments of the present application;

[0053] Figure 5 A flowchart of a product intelligent identification method provided in some embodiments of the present application;

[0054] Figure 6 A schematic diagram of the structure of a deep feature extraction model provided in some embodiments of the present application;

[0055] Figure 7 A flowchart of a product intelligent identification method provided in some embodiments of the present application;

[0056] Figure 8 A flowchart of a product intelligent identification method provided in some embodiments of the present application;

[0057] Figure 9 A flowchart of a product intelligent identification method provided in some embodiments of the present application;

[0058] Figure 10 A flowchart of a product intelligent identification method provided in some embodiments of the present application;

[0059] Figure 11 A schematic diagram of the concat operation provided in some embodiments of the present application;

[0060] Figure 12 A schematic diagram of the add fusion operation provided in some embodiments of the present application;

[0061] Figure 13 A schematic diagram of image labels in a product image dataset provided by some embodiments of the present application;

[0062] Figure 14 A flowchart of a product intelligent identification method provided in some embodiments of the present application;

[0063] Figure 15 A schematic diagram of the structure of a product intelligent identification device provided in some embodiments of the present application;

[0064] Figure 16A schematic diagram of the structure of the shallow and deep feature image extraction units provided in some embodiments of the present application;

[0065] Figure 17 A schematic diagram of the structure of the shallow and deep feature image extraction units provided in some embodiments of the present application;

[0066] Figure 18 A schematic structural diagram of a shallow layer and deep layer weighted image acquisition unit provided in some embodiments of the present application;

[0067] Figure 19 A schematic diagram of the structure of a weighted shallow feature image acquisition module provided in some embodiments of the present application;

[0068] Figure 20 A schematic diagram of the structure of a weighted deep feature image acquisition module provided in some embodiments of the present application;

[0069] Figure 21 A schematic diagram of the physical structure of an electronic device provided in some embodiments of the present invention. DETAILED DESCRIPTION

[0070] To make the purpose, technical solutions, and advantages of the embodiments of the present application more clearly understood, the embodiments of the present application are further described in detail below with reference to the accompanying drawings. The illustrative embodiments of the present application and their descriptions are used to explain the present application but are not intended to limit the present application. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of the present application may be combined with each other in any manner.

[0071] The impressive performance of deep learning has made deep learning-based object recognition methods a mainstream trend. The accuracy of deep feature maps is a key factor in determining the algorithm's detection results. Convolutional Neural Networks (CNNs) are a representative deep learning algorithm and a primary means of extracting deep features. In recent years, with the continuous advancement of the numerical computing power of electronic devices, the representation learning performance of CNNs has also continued to improve, enabling better fitting of large data sets and widespread application in fields such as computer vision. With the development of deep learning, many classic CNNs have emerged, including AlexNet, DarkNet, VGG, GoogleNet, ResNet, and DenseNet. In theory, building deeper CNNs is an effective means of extracting richer deep features. However, experimentally, increasing the depth of CNNs beyond a certain level often leads to gradient vanishing and degradation in backpropagation. In other words, training accuracy does not increase continuously with network depth, but instead gradually levels off after reaching a certain limit, and the training error actually increases. Therefore, appropriate methods should be adopted to avoid overfitting. Furthermore, it is important to prevent the computational overhead caused by overfitting from significantly increasing GPU memory. In addition, when performing product identification, it is also necessary to improve the recognition accuracy while ensuring the recognition speed.

[0072] To solve the above problems, this paper selects a convolutional neural network with a relatively simple structure as the basic network and improves it. It proposes a product intelligent recognition model with multiple branches. It adds multiple branches on the basis of the convolutional neural network and introduces the channel self-attention model (CSA) to highlight the target information and suppress the background, effectively improving the recognition accuracy while ensuring the recognition speed.

[0073] Figure 1 This is a schematic diagram of the overall structure of the product intelligent recognition model provided by some embodiments of the present application, such as Figure 1 As shown in the figure, the intelligent product recognition model consists of two different branches: the upper branch is the shallow feature extraction model, and the lower branch is the deep feature extraction model. The feature images output by the shallow and deep feature extraction models have the same size and number of channels. The intelligent product recognition model inputs the feature images extracted by the two branches into a self-channel attention model (CSA) and fuses the feature images output by the two CSA models to further identify the product type.

[0074] The attention mechanism (AM) is a resource allocation mechanism that determines the proportion of computing resources based on the amount of information contained in the input signal. It can be understood as reallocating originally evenly distributed resources based on the importance of the attention target, allocating more resources to important objects and less to unimportant or less-important objects. With the development of neural networks, the attention mechanism has become a key component of this field, widely used in fields such as natural language processing, statistical learning, and speech recognition. The attention mechanism involved in computer vision research can be intuitively explained by the human visual system. When perceiving an object, human vision generally does not focus on the entire scene from beginning to end. Instead, it observes specific parts based on needs and ignores irrelevant information. Furthermore, when people notice that certain areas of a scene frequently contain objects of interest, their brains learn this information, and when similar scenes appear in the future, they naturally focus on these areas. Using the attention mechanism in a multi-directional object detection network framework can identify correlations between features and allocate more weight to important features, thereby focusing on target information, suppressing background information, and avoiding interference from irrelevant factors. Common attention mechanisms include channel-wise, spatial, high-order, and self-attention.

[0075] In image processing, the concept of channel is crucial. Color images use channels to store their color elements. In RGB mode, color images consist of three channels: red, green, and blue. Using convolutional neural networks for feature extraction can obtain a fusion of spatial and channel information within the local receptive region of an image, describing the image using a hierarchical model with a global receptive region. Each layer of a convolutional network contains several convolution kernels, each corresponding to a feature channel. After convolution of an RGB image, a multi-dimensional channel is generated based on the number of convolution kernels. The features in each channel are formed by combining multiple features from the previous layer. The importance of a channel is reflected by the strength of its pixel values. If equal importance is assigned to each channel feature, the target information cannot be highlighted, and different types of information cannot be flexibly processed. This results in a lack of discriminative learning capabilities. Using a channel attention mechanism can highlight valid information and avoid this problem.

[0076] Object recognition addresses the problem of what is being detected, while channel attention can be understood as what the neural network should look at—that is, what the target is. Channel attention (CA) essentially adds a weight prediction branch to a convolutional neural network, ultimately resulting in a model that automatically captures the importance of features within different channels. CA models the dependencies between channels, adaptively adjusts the feature response values of each channel, and assigns different weights to each channel, allowing the network to selectively highlight important features and suppress useless ones. This paper improves on channel attention by designing a channel self-attention model (CSA).

[0077] The self-channel attention model (CSA) is the first model structure created in this application. Figure 2 Schematic diagram of the structure of the self-channel attention model (CSA) provided by some embodiments of the present application. Figure 2 As shown in the figure, the self-channel attention model (CSA) has two weight extraction branches. In each branch, average pooling (AVG-pooling) and maximum pooling (Max-pooling) are used to compress the original image, thereby extracting two different weights. The self-channel attention model fuses the two different weights and assigns the fused weights to the original feature image to obtain a weighted feature image. In addition, based on the weighted feature image, the self-channel attention model further fuses the weighted feature image with the original feature image to obtain a new feature image. The original feature image includes a shallow feature image and a deep feature image. CSA can be connected to any layer of a convolutional neural network, regardless of the network structure, and has strong scalability and versatility.

[0078] After completing the attention allocation and fusion, the product intelligent recognition model provided in this application uses the Add operation to connect the feature maps of the same size in the two branch networks, further integrating the shallow detail features of the shallow feature extraction model branch and the deep semantic features of the deep feature extraction model branch, thereby improving the detection accuracy.

[0079] The product intelligent recognition model provided in this application uses two different methods to perform feature fusion. The first feature fusion is to directly perform dimension splicing using the Concat method within the deep feature extraction model, while fully retaining the shallow original features, fusing deep features and expanding the channel dimension to prevent information distortion. The second feature fusion is to add the feature images of the same scale obtained by inputting the shallow feature image and the deep feature image into the self-channel attention model (CSA) respectively, thereby increasing the amount of information in each dimension of the feature map without changing the output dimension. Finally, three fully connected layers are used to operate on the fused feature map. The output of the last fully connected layer is the prediction result of the product classification.

[0080] The following describes the specific implementation process of the product intelligent identification method provided by the embodiment of the present invention by taking the server as an example of the execution entity.

[0081] Figure 3 This is a flowchart of a product intelligent identification method provided in one embodiment of the present application, such as Figure 3 As shown, the product intelligent identification method provided by this application includes:

[0082] S301: Input the product image into the shallow feature extraction model and the deep feature extraction model respectively to obtain a shallow feature image and a deep feature image;

[0083] S302: Inputting the shallow feature image and the deep feature image into the pre-trained channel attention model respectively to obtain the corresponding weighted shallow feature image and weighted deep feature image;

[0084] S303: Fusing the shallow feature image and the weighted shallow feature image to obtain a first feature image;

[0085] S304: Fusing the deep feature image and the weighted deep feature image to obtain a second feature image;

[0086] S305: Fusing the first feature image and the second feature image to obtain a third feature image, and identifying the category of the product in the product image according to the third feature image.

[0087] The product intelligent recognition method provided by the present application obtains shallow feature images and deep feature images respectively by inputting product images into a shallow feature extraction model and a deep feature extraction model, and obtains corresponding weighted shallow feature images and weighted deep feature images respectively by inputting them into a pre-trained channel attention model, fuses the shallow feature images with the weighted shallow feature images to obtain a first feature image, fuses the deep feature images with the weighted deep feature images to obtain a second feature image, and fuses the first feature image with the second feature image to obtain a third feature image. The product classification in the product image is identified based on the third feature image, thereby realizing intelligent recognition of the product. The product intelligent recognition method and device provided by the present application obtain deep features and shallow features of the image respectively during the recognition process, incorporates a channel self-attention mechanism, allocates different attention to different channels, and further highlights target information based on the weighted image, thereby improving the accuracy of the model in recognizing products. The product image is enhanced before recognition, and the position features of the image are retained during the feature extraction process, further ensuring the accuracy of product recognition and preventing factors such as shooting angle from interfering with the model. The model has fewer parameters and has lower performance requirements for the GPU.

[0088] Each step is explained in detail below.

[0089] S301: Input the product image into the shallow feature extraction model and the deep feature extraction model respectively to obtain a shallow feature image and a deep feature image;

[0090] Specifically, the structures of the shallow feature extraction model and the deep feature extraction model are as follows: Figure 1 As shown in the figure, the size, number and step size of the convolution kernels in each convolution layer in the shallow feature extraction model and the deep feature extraction model can be reasonably set according to the actual situation, so that the extracted shallow feature image and deep feature image have the same size and number of channels.

[0091] Since the product images collected during product recognition often have different sizes, the product images can be randomly cropped according to the preset size, that is, images of the preset size can be randomly cropped from the product images, thereby ensuring that the size of the input image is the same as the input size required by the shallow feature extraction model and the deep feature extraction model.

[0092] In some embodiments, the product image can be randomly cropped multiple times to obtain multiple input images, which are respectively input into the shallow feature extraction model and the deep feature extraction model for subsequent feature extraction and recognition, thereby further improving the accuracy of product recognition.

[0093] In some embodiments, the shallow feature extraction model includes m shallow convolution modules connected sequentially. In this case, S301 includes: inputting the product image into the m shallow convolution modules connected sequentially, performing splicing and segmentation operations, and outputting a shallow feature image; among the m shallow convolution modules, the input of the nth shallow convolution module is the output of the n-1th shallow convolution module, where n is greater than or equal to 2 and less than or equal to m.

[0094] Specifically, the shallow feature extraction model includes a first shallow convolution module, a second shallow convolution module, ..., and an mth shallow convolution module, which are connected in sequence. The product image is input into the first shallow convolution module, and the output image of the first shallow convolution module is input into the second shallow convolution module. The output image of the second shallow convolution module is input into the third shallow convolution module, ..., until the mth shallow convolution module. The output image of the mth shallow convolution module is the shallow feature image. If the previous convolution module only outputs one feature image, the image output by the previous convolution module can be segmented according to actual conditions such as hardware conditions before inputting it into the current convolution module. If the previous convolution module outputs multiple feature images, the multiple feature images can be spliced according to actual needs.

[0095] Each convolution module contains a convolution layer. In addition, pooling layers, normalization layers, activation functions, etc. can be set in each convolution module according to actual needs, and this application does not limit this. The shallow feature extraction model is used to retain shallow detail features and position features while extracting image features. Therefore, there should not be too many shallow convolution modules, and the number of shallow convolution modules in the shallow feature extraction model should be limited, for example, m≤7, but this application is not limited to this.

[0096] Figure 4 This is a schematic diagram of the structure of the shallow feature extraction model provided by some embodiments of the present application. Based on the above implementations, further, as Figure 4 As shown, the shallow feature extraction model includes: a first shallow convolution module 410, a second shallow convolution module 420, a third shallow convolution module 430, a fourth shallow convolution module 440 and a fifth shallow convolution module 450. Figure 5 This is a flowchart of extracting shallow feature images provided by some embodiments of the present application. Based on the above embodiments, further, as Figure 5 As shown, S301 includes:

[0097] S501: Input the product image into the first shallow convolution module to obtain multiple first shallow intermediate images;

[0098] Specifically, if Figure 4As shown, the first shallow convolution module 410 includes a convolution layer (Conv) and a maximum pooling layer (Maxpooling). After the product image is input into the convolution layer, the image obtained is segmented according to the channel to obtain multiple feature images, which are respectively input into the maximum pooling layer to obtain multiple first shallow intermediate images.

[0099] For example, the size of the input product image is 227x227x3, where 227x227 represents the length and width, and 3 represents the number of channels. Using 96 11x11x3 convolution kernels for convolution, with an expansion of 0 and a stride of 4, the width of the output image can be calculated using the following formula:

[0100]

[0101] Among them, w is the width of the output image, w input is the width of the input image, w kernel is the convolution kernel width, padding is the number of expansions, and stride is the step size of the convolution. The number of convolution kernels is the number of channels in the output image, so the size of the output image is 55×55×96.

[0102] The output image is segmented according to the number of channels, usually divided equally into two, resulting in two feature images of size 55×55×48. Maximum pooling is performed on each of these images to obtain two first shallow intermediate images. However, this application does not impose a limit on the specific number of segmentations.

[0103] In one embodiment, ReLU can also be used as an activation function to activate the feature image output by the convolution layer, and a local response normalization layer LRN is set to perform local response normalization on the activated image. When the local response normalization layer LRN is used for normalization, the normalized parameters can be expressed by the following formula:

[0104]

[0105] in, is the pixel value of the pixel with coordinates (x, y) in the ith channel before normalization, is the pixel value of the pixel with coordinates (x, y) in the i-th channel after normalization, N is the number of channels of the feature image output by the convolution layer, k, α, β, and n are hyperparameters that can be set according to experience, for example, k = 2, n = 5, α = 0.0001, β = 0.75, but this application is not limited to this.

[0106] Compared to activation functions like sigmoid, ReLu sets the output of some neurons to 0, improving network sparsity and reducing parameter interdependence, thereby alleviating overfitting. Furthermore, ReLu's functional form is simpler, significantly reducing the computational effort required for activation compared to sigmoid. Using LRN creates a competitive mechanism between pixels in adjacent channels, causing values with relatively large responses to become relatively larger, suppressing other neurons with smaller responses and enhancing the model's generalization capabilities.

[0107] S502: Inputting the plurality of first shallow intermediate images into the second shallow convolution module respectively to obtain a plurality of second shallow intermediate images;

[0108] Specifically, if Figure 4 As shown, the second shallow convolution module 420 also includes a convolution layer and a maximum pooling layer. The second shallow convolution module 420 can be deployed on multiple GPUs. The multiple first shallow intermediate images obtained in S501 are input to the second shallow convolution modules in different GPUs for feature extraction, thereby reducing the requirements for a single GPU and preventing GPU memory overflow. After the extraction is completed, multiple second shallow intermediate images are obtained.

[0109] In one embodiment, ReLU can also be set as an activation function after the convolution layer for activation, and an LRN layer can be used for normalization.

[0110] S503: splicing the multiple second shallow intermediate images and inputting the convolution module into the third shallow intermediate image to obtain a third shallow intermediate image;

[0111] Specifically, splitting the feature image for training may cause the correlation between some channels to be lost. Therefore, multiple second shallow intermediate images are reassembled into a feature image and input into the third shallow convolution module 430 for feature extraction to obtain the third shallow intermediate image. In addition, after the feature extraction is completed, ReLU can be used as the activation function to activate the extraction result. Figure 4 As shown, the third shallow convolution module 430 only includes one convolution layer.

[0112] S504: Inputting a plurality of third shallow layer segmented images obtained by segmenting the third shallow layer intermediate image into a fourth shallow layer convolution module to obtain a plurality of fourth shallow layer intermediate images;

[0113] Specifically, if Figure 4As shown, the fourth shallow convolution module 440 includes only one convolution layer and is deployed on multiple GPUs. The obtained third shallow intermediate image is re-segmented to obtain multiple third shallow segmented images. Each of the multiple third shallow segmented images is input into the fourth shallow convolution module in a different GPU for feature extraction, thereby obtaining multiple fourth shallow intermediate images. Similarly, ReLU can be used to activate the extracted image.

[0114] S505: Inputting each fourth shallow intermediate image into a fifth shallow convolution module to obtain a plurality of fifth shallow intermediate images;

[0115] Specifically, if Figure 4 As shown, the fifth shallow convolution module 450 also includes only one convolution layer. The fifth shallow convolution module 450 is deployed on multiple GPUs together with each fourth shallow convolution module 440. After extracting the fourth shallow intermediate image, the fourth shallow intermediate image is input into the fifth shallow convolution module 450 to obtain the fifth shallow intermediate image.

[0116] S506: Splicing the fifth shallow intermediate images to obtain a shallow feature image.

[0117] Specifically, the extracted fifth shallow intermediate images are spliced through concat. Before and after splicing, the size of the image at each layer remains unchanged, and the number of channels is equal to the sum of the number of channels of the five shallow intermediate images.

[0118] The product intelligent recognition method provided by the present application obtains multiple first shallow intermediate images by inputting the product image into the first shallow convolution module, and obtains multiple second shallow intermediate images by inputting them into the second shallow convolution module respectively. The multiple second shallow intermediate images are spliced and inputted into the third shallow convolution module to obtain the third shallow intermediate image. The third shallow segmented images are segmented to obtain multiple third shallow segmented images, which are inputted into the fourth shallow convolution module to obtain multiple fourth shallow intermediate images, which are inputted into the fifth shallow convolution module to obtain multiple fifth shallow intermediate images. The shallow feature images are obtained after splicing, thereby realizing the extraction of shallow feature images. The shallow feature extraction model structure is relatively simple, and in the feature extraction process, the extraction process is dispersed to multiple GPUs, which reduces the requirements for GPU performance and reduces the risk of GPU memory overflow.

[0119] In addition, although this application takes splitting the feature image and dividing it into different GPUs for feature extraction as the preferred embodiment, when the GPU memory is sufficient, the image can be not split and feature extraction can always be completed on one GPU. This application does not impose any restrictions on this.

[0120] In some embodiments, as Figure 6As shown, the deep feature extraction model includes: a backbone network 610 and a deep feature extraction module 620. In some embodiments, the deep feature extraction module 620 includes a deep convolution module, a convolution layer (Conv) and a maximum pooling layer (Maxpooling) connected in sequence. Among them, the deep convolution module, namely the Darknetconv2d_BN_Leaky module, is referred to as the DBL module. Figure 1 As shown in Figure 1, each DBL module consists of a convolutional layer, a normalization layer (BatchNormalization, BN) and a ReLU activation function connected in sequence. The first deep convolution module and the second deep convolution module are both DBL modules.

[0121] Figure 7 This is a flowchart of extracting weighted shallow feature images provided by some embodiments of the present application. Based on the above embodiments, further, as shown in FIG. Figure 7 As shown, S301 also includes:

[0122] S701: Input the product image into the backbone network to obtain a first deep intermediate image and a second deep intermediate image;

[0123] Specifically, backbone network 610 consists of DBL modules and maximum pooling layers, including a first preset number of DBL modules. The maximum pooling layer is only provided after the first DBL module. No pooling layers are provided in other DBL modules or after the DBL modules. Instead, the pooling layer operation is replaced by setting a step size for the convolutional layers in the DBL modules. In addition, a batch normalization layer is used in the DBL module to replace the dropout layer. The batch normalization layer normalizes the model, thereby preventing overfitting and accelerating the convergence of the model.

[0124] The product image is input into the backbone network 610, and after feature extraction is performed by a first preset number of DBL modules, a first deep intermediate image is obtained. Feature extraction is continued on the first deep intermediate image, and after passing through a second preset number of DBL modules, a second deep intermediate image is obtained.

[0125] For example, the first preset number is 25 and the second preset number is 18. A product image is input into the backbone network 610, passes through the first DBL module, undergoes a max pooling operation, and further performs feature extraction through seven DBL modules to obtain a first deep intermediate image. Feature extraction continues on the first deep intermediate image, and after passing through 18 DBL modules, a second deep intermediate image is obtained.

[0126] However, the present application is not limited to this. The number of DBL modules can be appropriately adjusted according to actual needs, but it should be ensured that the deep feature image extracted by the deep feature extraction model and the shallow feature image extracted by the shallow feature extraction model have the same size and number of channels.

[0127] S702: stitching the first deep intermediate image and the second deep intermediate image to obtain a third deep intermediate image;

[0128] Specifically, the first deep intermediate image and the second deep intermediate image are spliced through a concat operation to obtain a third deep intermediate image, and the number of channels of the third deep intermediate image is equal to the sum of the number of channels of the first deep intermediate image and the second deep intermediate image.

[0129] S703: Input the third deep intermediate image into the deep feature extraction module to obtain a deep feature image.

[0130] Specifically, the deep feature extraction module is used to perform further feature extraction on the third deep intermediate image obtained after fusion to obtain the final deep feature image.

[0131] In one embodiment, the deep feature extraction module includes a second deep convolution module, a convolution layer, and a maximum pooling layer connected in sequence. In this case, S703 includes:

[0132] The third deep intermediate image is sequentially input into the second deep convolution module, convolution layer and maximum pooling layer to obtain the deep feature image.

[0133] Specifically, when the second deep convolution module, convolution layer and maximum pooling layer are used to extract features from the third deep intermediate image, the third deep intermediate image is divided into multiple grids, and the position, type and other features in each grid are extracted separately. Before extraction, the size of each grid can be normalized. The number of the second deep convolution modules and the number of convolution layers in the deep feature extraction module can be adjusted according to actual needs, for example, Figure 6 As shown, the number of the second deep convolution modules is 6, and the third deep intermediate image passes through 6 sequentially connected second deep convolution modules and is input into the convolution layer and the maximum pooling layer, but this application is not limited to this.

[0134] In one embodiment, the number of channels of the third deep intermediate image can be set according to the number of types of products to be predicted, with each type corresponding to a separate channel.

[0135] In another embodiment, the number of channels in the third deep intermediate image can be set based on how the product's location is represented. For example, the product's location can be represented by an outer frame that encloses the product. The parameters of each outer frame include its width, height, the horizontal coordinate of its center point, the vertical coordinate of its center point, and the confidence level that the product to be detected is within the outer frame. A channel can be set for each feature corresponding to each parameter.

[0136] By clustering the bounding boxes of each product image in historical product images, we can derive several optimal prior bounding box sizes and set a set of parameters for each prior box. Since each box corresponds to five parameters, the number of channels for position information is five times the number of prior bounding box types.

[0137] In addition, different channels can be set at the same time to represent the types and characteristics corresponding to the positions, or other appropriate methods can be selected to determine the number of channels according to actual conditions. This application does not limit the specific setting method of the number of channels.

[0138] The product intelligent recognition method provided by the present application obtains a first deep intermediate image and a second deep intermediate image by inputting a product image into a backbone network, splicing the first deep intermediate image and the second deep intermediate image to obtain a third deep intermediate image, and sequentially inputting the third deep intermediate image into a deep convolution module, a convolution layer, and a maximum pooling layer to obtain a deep feature image, thereby realizing the extraction of the deep feature image. During the image extraction process, the gradient original features in the first deep intermediate image and the deep features in the second deep intermediate image are spliced, fully retaining the shallow detail features and position features, avoiding the loss of position features, and thus improving the accuracy of subsequent recognition.

[0139] S302: Inputting the shallow feature image and the deep feature image into the pre-trained channel attention model respectively to obtain the corresponding weighted shallow feature image and weighted deep feature image;

[0140] Specifically, if Figure 1As shown, a self-channel attention model (CSA) can be set after the shallow feature extraction model and the deep feature extraction model, and the channel attention model (CA) is included in the self-channel attention model (CSA). The channel attention model provided in this application adds a branch on the basis of the traditional channel attention model, and uses average pooling (AVG-pooling) and maximum pooling (Max-pooling) to compress the original image, thereby obtaining the weight of each channel. The fully connected layer (Fully Connected Layer, FC) in the self-channel attention model (CSA) of the shallow feature extraction model and the self-channel attention model (CSA) connected to the deep feature extraction model can have the same parameters or different parameters, and this application does not impose any restrictions on this.

[0141] In some embodiments, as Figure 8 As shown, S302 includes:

[0142] S801: Input the shallow feature image into the pre-trained channel attention model to obtain the corresponding weighted shallow feature image;

[0143] Specifically, the channel attention model allocates attention to each channel in the shallow feature image, extracts the weight of each channel in the shallow feature image according to the shallow feature image, multiplies the weight of each channel with each pixel in the corresponding channel layer, and obtains a weighted shallow feature image.

[0144] In some embodiments, the channel attention model includes: a first weight acquisition module and a second weight acquisition module. Figure 9 This is a flow chart of obtaining a weighted shallow feature image provided by an embodiment of the present application. Based on the above embodiments, Figure 9 As shown, S801 includes:

[0145] S901: Inputting the shallow feature image into the first weight feature acquisition module to obtain the first shallow weight feature;

[0146] Specifically, if Figure 2 As shown in the figure, the first weighted feature acquisition module consists of an average pooling layer, a fully connected layer, a ReLU activation function, and a fully connected layer connected in sequence. The average pooling layer compresses the image, reducing its width and height to 1×1 while maintaining the number of channels. The remaining eigenvalues correspond one-to-one to each channel. Two fully connected layers and the ReLU function are used to excite the eigenvalues compressed by the average pooling layer, generating the first shallow weighted features. By using average pooling to compress an image, more attention is paid to its global features, fully preserving its location information.

[0147] In one embodiment, if the shallow feature image has C channels, the fully connected layer after the average pooling layer can have C×S neurons, and another fully connected layer not connected to the average pooling layer can have C neurons. Where S is a scaling parameter, which is a constant greater than 0 and less than or equal to 1. By setting the scaling parameter S, the parameters in the fully connected layer connected to the average pooling layer can be reduced, thereby further reducing the amount of computation.

[0148] S902: Inputting the shallow feature image into a second weight feature acquisition module to obtain a second shallow weight feature;

[0149] For details, please refer to Figure 2 The second weighted feature acquisition module consists of a maximum pooling layer, a fully connected layer, a ReLU activation function, and a fully connected layer connected in sequence. The maximum pooling layer is used to compress the image, also compressing the image width and height to 1×1. The number of channels remains unchanged, so that the remaining eigenvalues correspond one-to-one to each channel. Two fully connected layers and the ReLU function are used to excite the eigenvalues compressed by the average pooling layer to obtain the second shallow weighted feature. By using maximum pooling to compress the image, the characteristic texture information of the image can be extracted, the difference between the features of each pixel can be increased, and useless information can be reduced.

[0150] In one embodiment, similarly, if the shallow feature image has C channels, the fully connected layer after the average pooling layer can have C×S neurons, and another fully connected layer not connected to the average pooling layer can have C neurons. Where S is a scaling parameter, which is a constant greater than 0 and less than or equal to 1. By setting the scaling parameter S, the parameters in the fully connected layer connected to the average pooling layer can be reduced, thereby further reducing the amount of computation.

[0151] S903: Activate the third shallow weight feature obtained by fusing the first shallow weight feature and the second shallow weight feature to obtain a third shallow weight;

[0152] Specifically, the weight features of the corresponding channels in the first and second shallow weight features are added together to obtain a third shallow weight feature. The third shallow weight feature is activated using a sigmoid function, and the weight values of each weight are mapped to a range between 0 and 1 to obtain a third shallow weight. The dimension of the third shallow weight is equal to the number of channels in the shallow feature image.

[0153] S904: Multiply the third shallow layer weight by each channel of the shallow layer feature image to obtain a weighted shallow layer feature image.

[0154] Specifically, the weight value of each dimension in the third shallow weight is multiplied by the pixel value of each pixel in the corresponding channel layer in the shallow feature image to complete the weight distribution and obtain the weighted shallow feature image.

[0155] The product intelligent recognition method provided by the present application obtains the first shallow weight feature by inputting the shallow feature image into the first weight feature acquisition module, obtains the second shallow weight feature by inputting the shallow feature image into the second weight feature acquisition module, activates the third shallow weight feature obtained by fusing the first shallow weight feature and the second shallow weight feature, obtains the third shallow weight, and multiplies the third shallow weight with each channel of the shallow feature image to obtain a weighted shallow feature image, thereby realizing the redistribution of attention to each channel so that the weight of the channel containing the target is larger. The weight vector is used to perform a multiplication operation with the original feature map, and the distribution of weights allows the model to pay more attention to the channel features with the largest amount of information, while suppressing those unimportant channel features. In the multi-directional target detection algorithm, more attention can be paid to target information and background information can be suppressed.

[0156] S802: Input the deep feature image into a pre-trained channel attention model to obtain the corresponding weighted deep feature image.

[0157] Specifically, the channel attention model allocates attention to each channel in the deep feature image, extracts the weight of each channel in the deep feature image according to the deep feature image, multiplies the weight of each channel with each pixel in the corresponding channel layer, and obtains a weighted deep feature image.

[0158] In some embodiments, the channel attention model includes: a first weight feature acquisition module and a second weight feature acquisition module. Figure 10 This is a flowchart of obtaining a weighted deep feature image provided by some embodiments of the present application. Based on the above embodiments, further, as shown in FIG. Figure 10 As shown, S802 includes:

[0159] S1001: Inputting the deep feature image into a first weight feature acquisition module to obtain a first deep weight feature;

[0160] Specifically, if Figure 2 As shown in the figure, the first weighted feature acquisition module consists of an average pooling layer, a fully connected layer, a ReLU activation function, and a fully connected layer connected in sequence. The average pooling layer compresses the image, reducing its width and height to 1×1 while maintaining the number of channels. The remaining eigenvalues correspond one-to-one to each channel. Two fully connected layers and the ReLU function are used to excite the eigenvalues compressed by the average pooling layer to obtain the first deep weighted features. By using average pooling to compress an image, more attention is paid to its global features, fully preserving its positional information.

[0161] In one embodiment, if the deep feature image has C channels, the fully connected layer after the average pooling layer can have C×S neurons, and another fully connected layer not connected to the average pooling layer can have C neurons. S is a scaling parameter, which is a constant greater than 0 and less than or equal to 1. By setting the scaling parameter S, the parameters in the fully connected layer connected to the average pooling layer can be reduced, thereby further reducing the amount of computation.

[0162] S1002: Inputting the deep feature image into a second weight feature acquisition module to obtain a second deep weight feature;

[0163] For details, please refer to Figure 2 The second weighted feature acquisition module consists of a maximum pooling layer, a fully connected layer, a ReLU activation function, and a fully connected layer connected in sequence. The maximum pooling layer is used to compress the image, also compressing the image width and height to 1×1. The number of channels remains unchanged, so that the remaining eigenvalues correspond one-to-one to each channel. Two fully connected layers and the ReLU function are used to excite the eigenvalues compressed by the average pooling layer to obtain the second deep weighted feature. By using maximum pooling to compress the image, the characteristic texture information of the image can be extracted, the difference between the features of each pixel can be increased, and useless information can be reduced.

[0164] In one embodiment, similarly, if the deep feature image has C channels, the fully connected layer after the average pooling layer can have C×S neurons, and another fully connected layer not connected to the average pooling layer can have C neurons. Where S is a scaling parameter, which is a constant greater than 0 and less than or equal to 1. By setting the scaling parameter S, the parameters in the fully connected layer connected to the average pooling layer can be reduced, thereby further reducing the amount of computation.

[0165] S1003: activating a third deep weight feature obtained by fusing the first deep weight feature and the second deep weight feature to obtain a third deep weight;

[0166] Specifically, the weight features of corresponding channels in the first deep weight feature and the second deep weight feature are added together to obtain a third deep weight feature. The third deep weight feature is activated using a sigmoid function, and the weight values of each weight are mapped to a range between 0 and 1 to obtain a third deep weight. The dimension of the third deep weight is equal to the number of channels in the deep feature image.

[0167] S1004: Multiply the third deep weight by each channel of the deep feature image to obtain a weighted deep feature image.

[0168] Specifically, the weight value of each dimension in the third deep weight is multiplied by the pixel value of each pixel in the corresponding channel layer in the deep feature image to complete the weight distribution and obtain the weighted deep feature image.

[0169] The product intelligent recognition method provided by the present application obtains the first deep weight feature by inputting the deep feature image into the first weight feature acquisition module, obtains the second deep weight feature by inputting the deep feature image into the second weight feature acquisition module, activates the third deep weight feature obtained by fusing the first deep weight feature and the second deep weight feature, obtains the third deep weight, and multiplies the third deep weight with each channel of the deep feature image to obtain a weighted deep feature image, thereby realizing the redistribution of attention to each channel so that the weight of the channel containing the target is larger. The weight vector is used to perform a multiplication operation with the original feature map, and the weight distribution allows the model to pay more attention to the channel features with the largest amount of information, while suppressing those unimportant channel features. In the multi-directional target detection algorithm, more attention can be paid to target information and background information can be suppressed.

[0170] S303: Fusing the shallow feature image and the weighted shallow feature image to obtain a first feature image;

[0171] Specifically, the self-channel attention model (CSA) uses the add operation to fuse the shallow feature image and the weighted shallow feature image obtained by the channel attention model (CA) to obtain a first feature image. In the first feature image, the pixel value of each channel layer is equal to the sum of the pixel values of the corresponding position in the shallow feature image and the weighted shallow feature image. The shallow feature image, the weighted shallow feature image, and the first feature image have the same size and number of channels. This operation can increase the difference in feature information at each position in the output feature map, further highlighting the target features.

[0172] Figure 11 This is a schematic diagram of the concat operation provided by some embodiments of the present application. Figure 11 As shown in Figure 1, when concatenating feature images using concat, the feature images a and b to be joined only need to have the same width and height in each channel; they do not need to have the same number of channels. When concatenating in the channel dimension, the number of channels in the resulting image c equals the sum of the number of channels in the feature images a and b to be joined. Concat fully preserves the original information of each feature map.

[0173] Figure 12 This is a schematic diagram of the add fusion operation provided in some embodiments of the present application. Figure 12As shown in the figure, when images are fused using the add method, the feature images a and b to be joined must not only have the same width and height in each channel, but also the same number of channels. During fusion, the pixel values of pixels in feature images a and b that have the same channel and the same horizontal and vertical coordinates are summed to produce feature image c. Feature images a, b, and c all have the same width, height, and number of channels. Using the add method ensures that features are fully fused.

[0174] This algorithm combines Concat connection and adds Add connection to realize multiple fusion methods, fully combining the advantages of each, so as to obtain a more accurate feature map.

[0175] S304: fusing the deep feature image and the weighted deep feature image to obtain a second feature image; the deep feature image, the weighted deep feature image and the second feature image have the same size and number of channels;

[0176] Specifically, the self-channel attention model (CSA) uses the add operation to fuse the deep feature image and the weighted deep feature image obtained by the channel attention model (CA) to obtain a second feature image. In the second feature image, the pixel value of each channel layer is equal to the sum of the pixel values of the corresponding position in the corresponding channel layer in the deep feature image and the weighted deep feature image. The deep feature image, the weighted deep feature image and the second feature image have the same size and number of channels. This operation can increase the difference in feature information at each position of the output feature map, further highlighting the target features.

[0177] S305: Fusing the first feature image and the second feature image to obtain a third feature image, and identifying the category of the product in the product image according to the third feature image.

[0178] Specifically, the add operation is used to fuse the first feature image and the second feature image to obtain the third feature image. The size and number of channels of the first feature image and the second feature image should be exactly the same. As shown in formula (1), the number, size, expansion number and step size of the convolution kernel in each convolution layer can be adjusted to adjust the size of the output feature image. By reasonably setting the parameters of each convolution layer in the two branches, it is ensured that the output shallow feature image and the deep feature image have the same size and number of channels, so that the size and number of channels of the first feature image and the second feature image should be exactly the same.

[0179] After obtaining the third feature image, it is fed into a preset number of fully connected layers to identify the product category. The output layer uses a softmax activation function, with the output value representing the probability that the product in the image belongs to the corresponding category. The number of neurons in the output layer can be adjusted based on the number of product categories to be identified.

[0180] For example, Figure 13 This is a schematic diagram of image labels in a product image dataset provided by some embodiments of the present application. Based on the above embodiments, further, as Figure 13 As shown, the classification labels of product images are multi-level classification labels, including classification of product quality grade, variety, shape, color, etc.

[0181] In one embodiment, each neuron in the output layer represents mutually exclusive product categories. Taking flowers as an example, the product categories can include pink double-petaled kalanchoe, purple double-petaled kalanchoe, orange-red large-flowered cyclamen, etc., but the present application is not limited to this. At this time, the number of neurons in the output layer is the number of mutually exclusive product categories. The data output by each neuron represents the probability that the product image belongs to the category represented by the neuron, and the category with the highest probability is selected as the category to which the product in the product image belongs. This representation method can intuitively represent the category of the product without the need for subsequent processing, but it is necessary to traverse all specific categories. The total number of neurons is equal to the product of the number of categories of the product in terms of type, shape, and color, and more neurons need to be set in the output layer.

[0182] In another embodiment, the categories represented by each neuron in the output layer are not completely mutually exclusive, but rather have certain subordinate relationships. Again, taking flowers as an example, each neuron could represent, for example, Kalanchoe, Cyclamen, Chinese Rose, Phalaenopsis, Double Kalanchoe, Single Kalanchoe, pink, light pink, soft pink, purple, red, yellow, etc. The number of neurons in the output layer is equal to the sum of the product categories, product colors, and shape types within each product category. The data output by each neuron represents the probability that the product image belongs to the corresponding category.

[0183] The probability that a product belongs to a specific category is equal to the product of the probabilities of the product type, shape, and color corresponding to that category. For example, the probability that a product belongs to a pink double-petaled kalanchoe is equal to the product of the probabilities output by the neuron representing the kalanchoe category, the neuron representing the double-petaled kalanchoe category, and the neuron representing the pink category. When identifying product categories, the probability of the product belonging to each specific category is calculated by multiplying the probabilities of the categories corresponding to each neuron's output. The category with the highest probability is then selected as the category of the product in the product image.

[0184] Since there is a lot of repetition in the colors of products under different types and shapes, the number of output neurons of this representation method is only at the sum level rather than the product level, which greatly reduces the number of output neurons.

[0185] In addition, multiple product intelligent recognition models can also be established to identify the quality grade, type, shape, color, etc. of the product respectively. This application does not limit the specific representation form of the output layer neurons of the product intelligent recognition model.

[0186] The product intelligent recognition method provided by the present application is to input the product image into a shallow feature extraction model and a deep feature extraction model respectively to obtain a shallow feature image and a deep feature image, and input them into a pre-trained channel attention model to obtain a corresponding weighted shallow feature image and a weighted deep feature image, fuse the shallow feature image and the weighted shallow feature image to obtain a first feature image, fuse the deep feature image and the weighted deep feature image to obtain a second feature image, and fuse the first feature image and the second feature image to obtain a third feature image, and identify the category of the product in the product image according to the third feature image, thereby realizing intelligent recognition of the product. While obtaining the deep features of the image, a simple network model is used to obtain shallow features and fuse them with the deep features, fully retaining the detailed features and avoiding the occurrence of overfitting; the channel self-attention mechanism is incorporated to allocate different attention to different channels, and the target information is further highlighted based on the weighted image, thereby improving the accuracy of the model in recognizing products; the product image is enhanced before recognition, and the position features of the image are retained during the feature extraction process, further ensuring the accuracy of product recognition and preventing factors such as shooting angle from interfering with the model; the model has fewer parameters and has lower performance requirements for the GPU.

[0187] Before using the product intelligent recognition model provided in this application to identify product images, the established product intelligent recognition model should also be trained. Figure 14 This is a flowchart of a product intelligent identification method provided in some embodiments of the present application. Based on the above embodiments, further, as Figure 14 As shown, the product intelligent identification method provided by this application also includes:

[0188] S1401: Obtain a product image annotation dataset, and divide the product image annotation dataset into a training set and a test set;

[0189] Specifically, the product image annotation dataset contains product images of various categories and the specific categories corresponding to each product image. The images in the product image annotation dataset are collected by staff through on-site photography and other methods. Since the differences in angles, positions, lighting, backgrounds and other factors when staff collect the dataset will cause certain interference to feature extraction, it is necessary to use denoising, sharpening, cropping and other methods to pre-process the images, standardize and unify their backgrounds, and avoid the influence of irrelevant factors. In addition, since product images need to be collected, pre-processed and annotated manually, the number of images in the product image annotation dataset is relatively limited. In order to avoid overfitting, data enhancement can be completed by rotating, mirroring, translating and other operations on the images to increase the number of images in the dataset, thereby improving the recognition accuracy of the trained model.

[0190] In machine learning algorithms, we typically divide the original dataset into training and test sets. The training set is used to calculate gradients, update weights, and train the neural network model; the test set is used to test the model's final performance and evaluate its accuracy, error, and other factors.

[0191] There are three common methods for partitioning a dataset: holdout, cross-validation, and bootstrap. The holdout method typically uses stratified sampling to sample data, partitioning the dataset according to a specific ratio. This method is commonly used in tasks such as decision trees, naive Bayes classifiers, linear regression, and logistic regression. Cross-validation divides the dataset into k equal parts. The first part is used as the test set, the remaining k-1 parts are used as the training set, the second part is used as the test set, and the remaining k-1 parts are used as the training set. This cycle is repeated, and the results are weighted averaged to form the final evaluation result. This method can reduce the chance of a single random partition, enhance the model's generalization ability, and improve data utilization efficiency. The bootstrap method uses repeated sampling with replacement to sample data. Each time, a sample is taken from the dataset as an element in the training set, and then replaced. This process is repeated m times, resulting in a training set of size m, in which some samples appear repeatedly and others do not. The samples that did not appear are used as the test set, and the remaining samples are used as the training set. Bootstrapping is useful when the dataset is small and it is difficult to effectively divide the training and test sets. In addition, bootstrapping can generate multiple different training sets from the initial dataset, which is of great benefit to methods such as ensemble learning.

[0192] Since this application has performed data augmentation and the data set has a large number of samples, for simplicity of operation, the holdout method can be used to divide the samples, forming a training set and a data set in a ratio of 7:3. Although this application uses the holdout method as a preferred embodiment, other appropriate methods can also be used to divide the training set and test set according to actual conditions.

[0193] S1402: Based on the training set and the test set, the product intelligent recognition model is trained using the stochastic gradient descent method.

[0194] Specifically, the product intelligent recognition model includes a shallow feature extraction model, a deep feature extraction model, a self-channel attention model (CSA), and a fully connected layer. An appropriate loss function can be selected and trained using stochastic gradient descent to obtain the final product intelligent recognition model. The loss function can be mean squared error (MSE), cross entropy, softmax loss function, etc., which is not limited in this application.

[0195] The product intelligent recognition method provided in this application obtains a product image annotation dataset, divides the product image annotation dataset into a training set and a test set, and uses the stochastic gradient descent method to train the product intelligent recognition model based on the training set and the test set, thereby realizing the determination of model parameters and ensuring the accuracy of model prediction.

[0196] When using the product intelligent identification method provided in this application to identify products, you can first attach an RFID tag to the test product or print a purchase order; after the labeled product is read by a card issuing machine, place it on a shooting table or scan the product QR code on the purchase order with a barcode scanner. After the label serial number is entered into the system, click the "Start Identification" button to start taking pictures and save the photos into the corresponding document; thereafter, use this preset threshold, for example, when it is 80%, it is judged as qualified, otherwise it is judged as unqualified; when it is judged to be qualified, click the "Pass" button to achieve product identification and grading.

[0197] Based on the same inventive concept, the embodiments of the present application also provide a product intelligent identification device, which can be used to implement the method described in the above embodiments, as described in the following embodiments. Since the principle of solving the problem by the product intelligent identification device is similar to that of the product intelligent identification method, the implementation of the product intelligent identification device can refer to the implementation of the method based on software performance benchmark determination, and the repetitions will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0198] Figure 15 This is a schematic diagram of the structure of the product intelligent identification device provided in some embodiments of the present application, such as Figure 15As shown, the product intelligent identification device provided by the present application includes: a shallow and deep feature image extraction unit 1510, a shallow and deep weighted image acquisition unit 1520, a first feature image acquisition unit 1530, a second feature image acquisition unit 1540 and a product classification identification unit 1550;

[0199] The shallow and deep feature image extraction unit 1510 is used to input the product image into the shallow feature extraction model and the deep feature extraction model respectively to obtain the shallow feature image and the deep feature image;

[0200] Specifically, the structures of the shallow feature extraction model and the deep feature extraction model are as follows: Figure 1 As shown in the figure, the size, number and step size of the convolution kernels in each convolution layer in the shallow feature extraction model and the deep feature extraction model can be reasonably set according to the actual situation, so that the extracted shallow feature image and deep feature image have the same size and number of channels.

[0201] Since the product images collected during product recognition often have different sizes, the shallow and deep feature image extraction unit 1510 can randomly crop the product images according to the preset size, that is, randomly crop an image of a preset size from the product image, thereby ensuring that the size of the input image is the same as the input size required by the shallow feature extraction model and the deep feature extraction model.

[0202] In some embodiments, the shallow and deep feature image extraction unit 1510 can randomly crop the product image multiple times to obtain multiple input images, which are respectively input into the shallow feature extraction model and the deep feature extraction model for subsequent feature extraction and recognition, thereby further improving the accuracy of product recognition.

[0203] In some embodiments, the shallow feature extraction model includes m shallow convolution modules connected sequentially, and the product picture is input into the shallow feature extraction model and the deep feature extraction model respectively to obtain a shallow feature image and a deep feature image correspondingly, including: inputting the product picture into the m shallow convolution modules connected sequentially, performing splicing and segmentation operations, and outputting a shallow feature image; among the m shallow convolution modules, the input of the nth shallow convolution module is the output of the n-1th shallow convolution module, where n is greater than or equal to 2 and less than or equal to m.

[0204] Specifically, the shallow feature extraction model includes a first shallow convolution module, a second shallow convolution module, ..., and an mth shallow convolution module, which are connected in sequence. The product image is input into the first shallow convolution module, and the output image of the first shallow convolution module is input into the second shallow convolution module. The output image of the second shallow convolution module is input into the third shallow convolution module, ..., until the mth shallow convolution module. The output image of the mth shallow convolution module is the shallow feature image. If the previous convolution module only outputs one feature image, the image output by the previous convolution module can be segmented according to actual conditions such as hardware conditions before inputting it into the current convolution module. If the previous convolution module outputs multiple feature images, the multiple feature images can be spliced according to actual needs.

[0205] Each convolution module contains a convolution layer. In addition, pooling layers, normalization layers, activation functions, etc. can be set in each convolution module according to actual needs, and this application does not limit this. The shallow feature extraction model is used to retain shallow detail features and position features while extracting image features. Therefore, there should not be too many shallow convolution modules, and the number of shallow convolution modules in the shallow feature extraction model should be limited, for example, m≤7, but this application is not limited to this.

[0206] The shallow and deep weighted image acquisition unit 1520 is used to input the shallow feature image and the deep feature image into the pre-trained channel attention model to obtain the corresponding weighted shallow feature image and weighted deep feature image;

[0207] Specifically, if Figure 1 As shown, a self-channel attention model (CSA) can be set after the shallow feature extraction model and the deep feature extraction model, and the channel attention model (CA) is included in the self-channel attention model (CSA). The channel attention model provided in this application adds a branch on the basis of the traditional channel attention model, and uses average pooling (AVG-pooling) and maximum pooling (Max-pooling) to compress the original image, thereby obtaining the weight of each channel. The fully connected layer (Fully Connected Layer, FC) in the self-channel attention model (CSA) of the shallow feature extraction model and the self-channel attention model (CSA) connected to the deep feature extraction model can have the same parameters or different parameters, and this application does not impose any restrictions on this.

[0208] The first feature image acquisition unit 1530 is used to fuse the shallow feature image and the weighted shallow feature image to obtain the first feature image; the shallow feature image, the weighted shallow feature image and the first feature image have the same size and number of channels;

[0209] Specifically, the first feature image acquisition unit 1530 adopts a self-channel attention model (CSA). Based on the shallow feature image and the weighted shallow feature image obtained by the channel attention model (CA), the shallow feature image and the weighted shallow feature image are fused using an add operation to obtain a first feature image. In the first feature image, the pixel value of each pixel in each channel layer is equal to the sum of the pixel values of the corresponding position pixels in the shallow feature image and the weighted shallow feature image. The shallow feature image, the weighted shallow feature image, and the first feature image have the same size and number of channels. This operation can increase the difference in feature information at each position of the output feature map, further highlighting the target features.

[0210] The second feature image acquisition unit 1540 is used to fuse the deep feature image and the weighted deep feature image to obtain a second feature image; the deep feature image, the weighted deep feature image and the second feature image have the same size and number of channels;

[0211] Specifically, the second feature image acquisition unit 1540 adopts a self-channel attention model (CSA). Based on the deep feature image and the weighted deep feature image obtained by the channel attention model (CA), the deep feature image and the weighted deep feature image are fused using an add operation to obtain a second feature image. In the second feature image, the pixel value of each channel layer is equal to the sum of the pixel values of the corresponding position pixels in the corresponding channel layer in the deep feature image and the weighted deep feature image. The deep feature image, the weighted deep feature image and the second feature image have the same size and number of channels. This operation can increase the difference in feature information at each position of the output feature map, further highlighting the target features.

[0212] The product category recognition unit 1550 is configured to fuse the first feature image and the second feature image to obtain a third feature image, and recognize the category of the product in the product image according to the third feature image.

[0213] Specifically, the product classification recognition unit 1550 uses an add operation to fuse the first feature image and the second feature image to obtain a third feature image. The size and number of channels of the first feature image and the second feature image should be exactly the same. As shown in formula (1), the size of the output feature image can be adjusted by adjusting the number, size, expansion number, and step size of the convolution kernels in each convolution layer. By reasonably setting the parameters of each convolution layer in the two branches, it is ensured that the output shallow feature image and the deep feature image have the same size and number of channels, so that the size and number of channels of the first feature image and the second feature image are exactly the same.

[0214] After obtaining the third feature image, it is fed into a preset number of fully connected layers to identify the product category. The output layer uses a softmax activation function, with the output value representing the probability that the product in the image belongs to the corresponding category. The number of neurons in the output layer can be adjusted based on the number of product categories to be identified.

[0215] For example, Figure 14 This is a schematic diagram of image labels in a product image dataset provided by some embodiments of the present application. Based on the above embodiments, further, as Figure 14 As shown, the classification labels of product images are multi-level classification labels, including classification of product quality grade, variety, shape, color, etc.

[0216] In one embodiment, each neuron in the output layer represents mutually exclusive product categories, such as pink double-petaled kalanchoe, purple double-petaled kalanchoe, and orange-red large-flowered cyclamen. The number of neurons in the output layer equals the number of mutually exclusive product categories. The data output by each neuron represents the probability that the product image belongs to the corresponding category, and the category with the highest probability is selected as the category of the product in the product image. This representation method intuitively represents product categories without requiring subsequent processing, but it does require traversing all specific categories. The total number of neurons is equal to the product of the number of categories for type, shape, and color, requiring a large number of neurons in the output layer.

[0217] In another embodiment, the categories represented by each neuron in the output layer are not completely mutually exclusive, but rather have certain subordinate relationships. For example, they can represent Kalanchoe, Cyclamen, Rose, Phalaenopsis, Double Kalanchoe, Single Kalanchoe, pink, light pink, soft pink, purple, red, and yellow, respectively. The number of neurons in the output layer is equal to the sum of the product categories, product color categories, and shape categories within each product category. The data output by each neuron represents the probability that the product image belongs to the corresponding category.

[0218] The probability that a product belongs to a specific category is equal to the product of the probabilities of the product type, shape, and color corresponding to that category. For example, the probability that a product belongs to a pink double-petaled kalanchoe is equal to the product of the probabilities output by the neuron representing the kalanchoe category, the neuron representing the double-petaled kalanchoe category, and the neuron representing the pink category. When identifying product categories, the probability of the product belonging to each specific category is calculated by multiplying the probabilities of the categories corresponding to each neuron's output. The category with the highest probability is then selected as the category of the product in the product image.

[0219] Since there is a lot of repetition in the colors of products under different types and shapes, the number of output neurons of this representation method is only at the sum level rather than the product level, which greatly reduces the number of output neurons.

[0220] In addition, multiple product intelligent recognition models can also be established to identify the quality grade, type, shape, color, etc. of the product respectively. This application does not limit the specific representation form of the output layer neurons of the product intelligent recognition model.

[0221] The product intelligent recognition device provided by the present application realizes intelligent recognition of products through the shallow and deep feature image extraction unit 1510, the shallow and deep weighted image acquisition unit 1520, the first feature image acquisition unit 1530, the second feature image acquisition unit 1540 and the product classification recognition unit 1550. The product intelligent recognition method and device provided by the present application, while acquiring the deep features of the image, uses a simple network model to obtain shallow features and fuses them with the deep features, fully retaining the detailed features and avoiding the occurrence of overfitting; incorporates the channel self-attention mechanism to allocate different attention to different channels, and further highlights the target information based on the weighted image, thereby improving the accuracy of the model in identifying products; performs image enhancement on the product image before recognition, and retains the positional features of the image during the feature extraction process, further ensuring the accuracy of product recognition and preventing factors such as shooting angle from interfering with the model; the model has fewer parameters and has lower performance requirements for the GPU.

[0222] In some embodiments, the shallow feature extraction model includes m shallow convolution modules connected sequentially. At this time, the shallow and deep feature image extraction unit 1510 is specifically used to: input the product image into the m shallow convolution modules connected sequentially, perform splicing and segmentation operations, and output a shallow feature image; among the m shallow convolution modules, the input of the nth shallow convolution module is the output of the n-1th shallow convolution module, where n is greater than or equal to 2 and less than or equal to m.

[0223] Specifically, the shallow feature extraction model includes a first shallow convolution module, a second shallow convolution module, ..., and an mth shallow convolution module, which are connected in sequence. The product image is input into the first shallow convolution module, and the output image of the first shallow convolution module is input into the second shallow convolution module. The output image of the second shallow convolution module is input into the third shallow convolution module, ..., until the mth shallow convolution module. The output image of the mth shallow convolution module is the shallow feature image. If the previous convolution module only outputs one feature image, the image output by the previous convolution module can be segmented according to actual conditions such as hardware conditions before inputting it into the current convolution module. If the previous convolution module outputs multiple feature images, the multiple feature images can be spliced according to actual needs.

[0224] Each convolution module contains a convolution layer. In addition, pooling layers, normalization layers, activation functions, etc. can be set in each convolution module according to actual needs, and this application does not limit this. The shallow feature extraction model is used to retain shallow detail features and position features while extracting image features. Therefore, there should not be too many shallow convolution modules, and the number of shallow convolution modules in the shallow feature extraction model should be limited, for example, m≤7, but this application is not limited to this.

[0225] Figure 16 This is a schematic diagram of the structure of the shallow feature image extraction module provided in some embodiments of the present application. Figure 15 On the basis of the embodiment, further, the shallow feature extraction model includes: a first shallow convolution module, a second shallow convolution module, a third shallow convolution module, a fourth shallow convolution module and a fifth shallow convolution module. At this time, Figure 16 As shown, in the product intelligent recognition device provided by the present application, the shallow and deep feature image extraction unit 1510 includes: a first shallow intermediate image extraction module 1511, a second shallow intermediate image extraction module 1512, a third shallow intermediate image extraction module 1513, a fourth shallow intermediate image extraction module 1514, a fifth shallow intermediate image extraction module 1515 and a shallow feature image acquisition module 1516;

[0226] The first shallow intermediate image extraction module 1511 is used to input the product image into the first shallow convolution module to obtain multiple first shallow intermediate images;

[0227] The second shallow intermediate image extraction module 1512 is used to input the multiple first shallow intermediate images into the second shallow convolution module respectively to obtain multiple second shallow intermediate images;

[0228] The third shallow intermediate image extraction module 1513 is used to stitch multiple second shallow intermediate images and input them into the third shallow convolution module to obtain a third shallow intermediate image;

[0229] The fourth shallow intermediate image extraction module 1514 is used to input a plurality of third shallow segmented images obtained by segmenting the third shallow intermediate image into the fourth shallow convolution module to obtain a plurality of fourth shallow intermediate images;

[0230] The fifth shallow intermediate image extraction module 1515 is used to input each fourth shallow intermediate image into the fifth shallow convolution module to obtain multiple fifth shallow intermediate images;

[0231] The shallow feature image acquisition module 1516 is used to splice the fifth shallow intermediate images to obtain a shallow feature image.

[0232] The product intelligent identification device provided in the present application realizes the extraction of shallow feature images through the first shallow intermediate image extraction module 1511, the second shallow intermediate image extraction module 1512, the third shallow intermediate image extraction module 1513, the fourth shallow intermediate image extraction module 1514, the fifth shallow intermediate image extraction module 1515 and the shallow feature image acquisition module 1516. The shallow feature extraction model structure is relatively simple, and during the feature extraction process, the extraction process is distributed to multiple GPUs, which reduces the requirements for GPU performance and reduces the risk of GPU memory overflow.

[0233] Figure 17 This is a schematic diagram of the structure of the deep feature image extraction module provided in some embodiments of the present application. Figure 15 On the basis of the embodiment, further, as Figure 17 As shown, in the product intelligent recognition device provided by the present application, the shallow and deep feature image extraction unit 1510 includes: a first and second deep intermediate image extraction module 1517, a third deep intermediate image acquisition module 1518 and a deep feature image acquisition module 1519;

[0234] The first and second deep intermediate image extraction module 1517 is used to input the product image into the backbone network to obtain a first deep intermediate image and a second deep intermediate image; the backbone network includes a first preset number of first deep convolution modules; the second deep intermediate image is obtained by passing the first deep intermediate image through a second preset number of first deep convolution modules;

[0235] The third deep intermediate image acquisition module 1518 is configured to stitch the first deep intermediate image and the second deep intermediate image together to obtain a third deep intermediate image.

[0236] The deep feature image acquisition module 1519 is used to input the third deep intermediate image into the deep feature extraction module to obtain a deep feature image.

[0237] In one embodiment, the deep feature extraction module includes a second deep convolution module, a convolution layer, and a maximum pooling layer connected in sequence. In this case, the deep feature image acquisition module 1519 is specifically used to:

[0238] The third deep intermediate image is sequentially input into the second deep convolution module, convolution layer and maximum pooling layer to obtain the deep feature image.

[0239] Specifically, when the second deep convolution module, convolution layer and maximum pooling layer are used to extract features from the third deep intermediate image, the third deep intermediate image is divided into multiple grids, and the position, type and other features in each grid are extracted separately. Before extraction, the size of each grid can be normalized. The number of the second deep convolution modules and the number of convolution layers in the deep feature extraction module can be adjusted according to actual needs, for example, Figure 6 As shown, the number of the second deep convolution modules is 6, and the third deep intermediate image passes through 6 sequentially connected second deep convolution modules and is input into the convolution layer and the maximum pooling layer, but this application is not limited to this.

[0240] In one embodiment, the number of channels of the third deep intermediate image can be set according to the number of types of products to be predicted, with each type corresponding to a separate channel.

[0241] In another embodiment, the number of channels in the third deep intermediate image can be set based on how the product's location is represented. For example, the product's location can be represented by an outer frame that encloses the product. The parameters of each outer frame include its width, height, the horizontal coordinate of its center point, the vertical coordinate of its center point, and the confidence level that the product to be detected is within the outer frame. A channel can be set for each feature corresponding to each parameter.

[0242] By clustering the bounding boxes of each product image in historical product images, we can derive several optimal prior bounding box sizes and set a set of parameters for each prior box. Since each box corresponds to five parameters, the number of channels for position information is five times the number of prior bounding box types.

[0243] In addition, different channels can be set at the same time to represent the types and characteristics corresponding to the positions, or other appropriate methods can be selected to determine the number of channels according to actual conditions. This application does not limit the specific setting method of the number of channels.

[0244] The product intelligent identification device provided in the present application realizes the extraction of deep feature images through the first and second deep intermediate image extraction modules 1517, the third deep intermediate image acquisition module 1518 and the deep feature image acquisition module 1519. During the image extraction process, the gradient original features in the first deep intermediate image and the deep features in the second deep intermediate image are spliced, which fully retains the shallow detail features and position features, avoids the loss of position features, and thus improves the accuracy of subsequent recognition.

[0245] Figure 18 This is a schematic diagram of the structure of the shallow and deep weighted image acquisition unit provided in some embodiments of the present application. Based on the above embodiments, further, as Figure 18As shown, in the product intelligent identification device provided by the present application, the shallow and deep weighted image acquisition unit 1520 includes: a weighted shallow feature image acquisition module 1521 and a weighted deep feature image acquisition module 1522;

[0246] The weighted shallow feature image acquisition module 1521 is used to input the shallow feature image into the pre-trained channel attention model to obtain the corresponding weighted shallow feature image;

[0247] The weighted deep feature image acquisition module 1522 is used to input the deep feature image into the pre-trained channel attention model to obtain the corresponding weighted deep feature image.

[0248] The product intelligent identification device provided in this application realizes the separate extraction of weighted shallow feature images and weighted deep feature images through the weighted shallow feature image acquisition module 1521 and the weighted deep feature image acquisition module 1522, providing a basis for subsequent further feature extraction, feature fusion and product identification.

[0249] Figure 19 is a structural diagram of a weighted shallow feature image acquisition module provided in some embodiments of the present application. Figure 18 On the basis of the embodiment, further, the channel attention model includes: a first weight feature acquisition module and a second weight feature acquisition module. At this time, Figure 19 As shown, in the product intelligent recognition device provided by the present application, the weighted shallow feature image acquisition module 1521 includes: a first shallow weight feature extraction submodule 1521a, a second shallow weight feature extraction submodule 1521b, a third shallow weight acquisition submodule 1521c and a weighted shallow feature image acquisition submodule 1521d;

[0250] The first shallow weight feature extraction submodule 1521a is used to input the shallow feature image into the first weight feature acquisition module to obtain the first shallow weight feature;

[0251] The second shallow weight feature extraction submodule 1521b is used to input the shallow feature image into the second weight feature acquisition module to obtain the second shallow weight feature;

[0252] The third shallow weight acquisition submodule 1521c is used to activate the third shallow weight feature obtained by fusing the first shallow weight feature and the second shallow weight feature to obtain a third shallow weight; the dimension of the third shallow weight is equal to the number of channels of the shallow feature image;

[0253] The weighted shallow feature image acquisition submodule 1521d is used to multiply the third shallow weight by each channel of the shallow feature image to obtain a weighted shallow feature image.

[0254] The product intelligent recognition device provided in this application realizes the redistribution of attention to each channel through the first shallow weight feature extraction submodule 1521a, the second shallow weight feature extraction submodule 1521b, the third shallow weight acquisition submodule 1521c and the weighted shallow feature image acquisition submodule 1521d, so that the channel weight containing the target is larger. The weight vector is multiplied with the original feature map, and the weight distribution allows the model to pay more attention to the channel features with the largest amount of information and suppress those unimportant channel features. In the multi-directional target detection algorithm, more attention can be paid to the target information and background information can be suppressed.

[0255] Figure 20 This is a schematic diagram of the structure of the weighted deep feature image acquisition module provided in some embodiments of the present application. Figure 18 On the basis of the embodiment, further, the channel attention model includes: a first weight feature acquisition module and a second weight feature acquisition module. At this time, Figure 20 As shown, in the product intelligent identification device provided by the present application, the weighted deep feature image acquisition module 1522 includes: a first deep weight feature extraction submodule 1522a, a second deep weight feature extraction submodule 1522b, a third deep weight acquisition submodule 1522c and a weighted deep feature image acquisition submodule 1522d;

[0256] The first deep weight feature extraction submodule 1522a is used to input the deep feature image into the first weight acquisition module to obtain the first deep weight feature;

[0257] The second deep weight feature extraction submodule 1522b is used to input the deep feature image into the second weight acquisition module to obtain the second deep weight feature;

[0258] The third deep weight acquisition submodule 1522c is used to activate the third deep weight feature obtained by fusing the first deep weight feature and the second deep weight feature to obtain a third weight feature; the dimension of the third deep weight is equal to the number of channels of the deep feature image;

[0259] The weighted deep feature image acquisition submodule 1522d is used to multiply the third deep weight by each channel of the deep feature image to obtain a weighted deep feature image.

[0260] The product intelligent identification device provided in this application realizes the redistribution of attention to each channel through the first deep weight feature extraction submodule 1522a, the second deep weight feature extraction submodule 1522b, the third deep weight acquisition submodule 1522c and the weighted deep feature image acquisition submodule 1522d, so that the weight of the channel containing the target is larger. The weight vector and the original feature map are multiplied and the weight distribution allows the model to pay more attention to the channel features with the largest amount of information and suppress those unimportant channel features. In the multi-directional target detection algorithm, more attention can be paid to the target information and background information can be suppressed.

[0261] Figure 21 is a schematic diagram of the physical structure of an electronic device provided by some embodiments of the present invention, such as Figure 21 As shown, the electronic device may include: a processor (processor) 2101, a communication interface (Communications Interface) 2102, a memory (memory) 2103 and a communication bus 2104, wherein the processor 2101, the communication interface 2102, and the memory 2103 communicate with each other through the communication bus 2104. The processor 2101 can call the logic instructions in the memory 2103 to execute the following method: input the product picture into the shallow feature extraction model and the deep feature extraction model respectively to obtain the shallow feature image and the deep feature image correspondingly; input the shallow feature image and the deep feature image into the pre-trained channel attention model respectively to obtain the corresponding weighted shallow feature image and weighted deep feature image; fuse the shallow feature image and the weighted shallow feature image to obtain the first feature image; the shallow feature image, the weighted shallow feature image and the first feature image have the same size and number of channels; fuse the deep feature image and the weighted deep feature image to obtain the second feature image; the deep feature image, the weighted deep feature image and the second feature image have the same size and number of channels; fuse the first feature image and the second feature image to obtain the third feature image, and identify the classification of the product in the product picture according to the third feature image.

[0262] In addition, the logic instructions in the above-mentioned memory 2103 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0263] This embodiment discloses a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: inputting a product picture into a shallow feature extraction model and a deep feature extraction model respectively to obtain a shallow feature image and a deep feature image correspondingly; inputting the shallow feature image and the deep feature image into a pre-trained channel attention model respectively to obtain a corresponding weighted shallow feature image and a weighted deep feature image; fusing the shallow feature image and the weighted shallow feature image to obtain a first feature image; the shallow feature image, the weighted shallow feature image and the first feature image have the same size and number of channels; fusing the deep feature image and the weighted deep feature image to obtain a second feature image; the deep feature image, the weighted deep feature image and the second feature image have the same size and number of channels; fusing the first feature image and the second feature image to obtain a third feature image, and identifying the classification of the product in the product picture according to the third feature image.

[0264] This embodiment provides a computer-readable storage medium, which stores a computer program, and the computer program enables the computer to execute the methods provided by the above-mentioned method embodiments, for example, including: inputting the product picture into a shallow feature extraction model and a deep feature extraction model respectively to obtain a shallow feature image and a deep feature image correspondingly; inputting the shallow feature image and the deep feature image into a pre-trained channel attention model respectively to obtain a corresponding weighted shallow feature image and a weighted deep feature image; fusing the shallow feature image and the weighted shallow feature image to obtain a first feature image; the shallow feature image, the weighted shallow feature image and the first feature image have the same size and number of channels; fusing the deep feature image and the weighted deep feature image to obtain a second feature image; the deep feature image, the weighted deep feature image and the second feature image have the same size and number of channels; fusing the first feature image and the second feature image to obtain a third feature image, and identifying the classification of the product in the product picture according to the third feature image.

[0265] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0266] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0267] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0268] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0269] Throughout this specification, reference to terms such as "one embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0270] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A product intelligent identification method, characterized in that: include: Input the product image into the shallow feature extraction model and the deep feature extraction model respectively to obtain the shallow feature image and the deep feature image; Inputting the shallow feature image and the deep feature image into a pre-trained channel attention model respectively to obtain corresponding weighted shallow feature image and weighted deep feature image; fusing the shallow feature image and the weighted shallow feature image to obtain a first feature image; the shallow feature image, the weighted shallow feature image and the first feature image have the same size and number of channels; fusing the deep feature image and the weighted deep feature image to obtain a second feature image; the deep feature image, the weighted deep feature image and the second feature image have the same size and number of channels; fusing the first feature image and the second feature image to obtain a third feature image, and identifying the category of the product in the product image according to the third feature image; The shallow feature extraction model includes m shallow convolution modules connected in sequence. The process of inputting the product image into the shallow feature extraction model and the deep feature extraction model to obtain a shallow feature image and a deep feature image respectively includes: inputting the product image into the m shallow convolution modules connected in sequence, performing splicing and segmentation operations, and outputting the shallow feature image; among the m shallow convolution modules, the input of the nth shallow convolution module is the output of the n-1th shallow convolution module, where n is greater than or equal to 2 and less than or equal to m; The step of inputting the product image into the shallow feature extraction model and the deep feature extraction model to obtain the shallow feature image and the deep feature image respectively further includes: Inputting the product image into a backbone network to obtain a first deep intermediate image and a second deep intermediate image; the backbone network includes a first preset number of first deep convolution modules; the second deep intermediate image is obtained by passing the first deep intermediate image through a second preset number of first deep convolution modules; splicing the first deep intermediate image and the second deep intermediate image to obtain a third deep intermediate image; The third deep intermediate image is sequentially input into the second deep convolution module, the convolution layer and the maximum pooling layer to obtain the deep feature image.

2. The product intelligent identification method according to claim 1, characterized in that: The shallow feature extraction model includes: a first shallow convolution module, a second shallow convolution module, a third shallow convolution module, a fourth shallow convolution module, and a fifth shallow convolution module; the product image is input into the shallow feature extraction model and the deep feature extraction model respectively to obtain a shallow feature image and a deep feature image, including: Inputting the product image into the first shallow convolution module to obtain multiple first shallow intermediate images; Inputting the plurality of first shallow intermediate images into the second shallow convolution module respectively to obtain a plurality of second shallow intermediate images; splicing the plurality of second shallow intermediate images and inputting the convolutional layers into the third shallow convolutional module to obtain a third shallow intermediate image; Inputting a plurality of third shallow layer segmented images obtained by segmenting the third shallow layer intermediate image into the fourth shallow layer convolution module to obtain a plurality of fourth shallow layer intermediate images; Inputting each fourth shallow intermediate image into the fifth shallow convolution module to obtain a plurality of fifth shallow intermediate images; The fifth shallow intermediate images are spliced together to obtain the shallow feature image.

3. The product intelligent identification method according to claim 1, characterized in that: The step of inputting the shallow feature image and the deep feature image into a pre-trained channel attention model to obtain corresponding weighted shallow feature images and weighted deep feature images includes: Inputting the shallow feature image into a pre-trained channel attention model to obtain a corresponding weighted shallow feature image; The deep feature image is input into a pre-trained channel attention model to obtain a corresponding weighted deep feature image.

4. The product intelligent identification method according to claim 3, characterized in that: The channel attention model includes: a first weight feature acquisition module and a second weight feature acquisition module; the inputting the shallow feature image into the pre-trained channel attention model to obtain the corresponding weighted shallow feature image includes: Inputting the shallow feature image into the first weight feature acquisition module to obtain a first shallow weight feature; Inputting the shallow feature image into the second weight feature acquisition module to obtain a second shallow weight feature; activating a third shallow weight feature obtained by fusing the first shallow weight feature and the second shallow weight feature to obtain a third shallow weight; the dimension of the third shallow weight is equal to the number of channels of the shallow feature image; The third shallow layer weight is multiplied by each channel of the shallow layer feature image to obtain the weighted shallow layer feature image.

5. The product intelligent identification method according to claim 3, characterized in that: The channel attention model includes: a first weight feature acquisition module and a second weight feature acquisition module; the inputting the deep feature image into the pre-trained channel attention model to obtain the corresponding weighted deep feature image includes: Inputting the deep feature image into the first weight feature acquisition module to obtain a first deep weight feature; Inputting the deep feature image into the second weight feature acquisition module to obtain a second deep weight feature; activating a third deep weight feature obtained by fusing the first deep weight feature and the second deep weight feature to obtain a third deep weight; the dimension of the third deep weight is equal to the number of channels of the deep feature image; The third deep weight is multiplied by each channel of the deep feature image to obtain the weighted deep feature image.

6. A product intelligent identification device, characterized in that: include: The shallow and deep feature image extraction units are used to input the product image into the shallow feature extraction model and the deep feature extraction model respectively to obtain the shallow feature image and the deep feature image respectively; A shallow and deep weighted image acquisition unit, configured to input the shallow feature image and the deep feature image into a pre-trained channel attention model to obtain corresponding weighted shallow feature images and weighted deep feature images; A first feature image acquisition unit is configured to fuse the shallow feature image and the weighted shallow feature image to obtain a first feature image; the shallow feature image, the weighted shallow feature image, and the first feature image have the same size and number of channels; A second feature image acquisition unit is configured to fuse the deep feature image and the weighted deep feature image to obtain a second feature image; the deep feature image, the weighted deep feature image, and the second feature image have the same size and number of channels; a product classification recognition unit, configured to fuse the first feature image and the second feature image to obtain a third feature image, and recognize the category of the product in the product image according to the third feature image; The shallow feature extraction model includes m shallow convolution modules connected in sequence, and the shallow and deep feature image extraction unit is specifically used to: input the product image into the m shallow convolution modules connected in sequence, perform splicing and segmentation operations, and output the shallow feature image; among the m shallow convolution modules, the input of the nth shallow convolution module is the output of the n-1th shallow convolution module, where n is greater than or equal to 2 and less than or equal to m; The shallow and deep feature image extraction units include: A first and second deep intermediate image extraction module is configured to input the product image into a backbone network to obtain a first deep intermediate image and a second deep intermediate image; the backbone network includes a first preset number of first deep convolution modules; the second deep intermediate image is obtained by passing the first deep intermediate image through a second preset number of first deep convolution modules; a third deep intermediate image acquisition module, configured to stitch the first deep intermediate image and the second deep intermediate image together to obtain a third deep intermediate image; The deep feature image acquisition module is used to input the third deep intermediate image into the second deep convolution module, the convolution layer and the maximum pooling layer in sequence to obtain the deep feature image.

7. The product intelligent identification device according to claim 6, characterized in that: The shallow feature extraction model includes: a first shallow convolution module, a second shallow convolution module, a third shallow convolution module, a fourth shallow convolution module and a fifth shallow convolution module; the shallow and deep feature image extraction units include: a first shallow intermediate image extraction module, configured to input the product image into the first shallow convolution module to obtain a plurality of first shallow intermediate images; A second shallow intermediate image extraction module is used to input the multiple first shallow intermediate images into the second shallow convolution module respectively to obtain multiple second shallow intermediate images; a third shallow intermediate image extraction module, configured to concatenate the plurality of second shallow intermediate images and input the concatenated images into the third shallow convolution module to obtain a third shallow intermediate image; a fourth shallow intermediate image extraction module, configured to input a plurality of third shallow segmented images obtained by segmenting the third shallow intermediate image into the fourth shallow convolution module to obtain a plurality of fourth shallow intermediate images; a fifth shallow intermediate image extraction module, configured to input each fourth shallow intermediate image into the fifth shallow convolution module to obtain a plurality of fifth shallow intermediate images; The shallow feature image acquisition module is used to splice the fifth shallow intermediate images to obtain the shallow feature image.

8. The product intelligent identification device according to claim 6, characterized in that: The shallow and deep weighted image acquisition unit includes: A weighted shallow feature image acquisition module is used to input the shallow feature image into a pre-trained channel attention model to obtain a corresponding weighted shallow feature image; The weighted deep feature image acquisition module is used to input the deep feature image into a pre-trained channel attention model to obtain the corresponding weighted deep feature image.

9. The product intelligent identification device according to claim 8, characterized in that: The channel attention model includes: a first weight feature acquisition module and a second weight feature acquisition module; the weighted shallow feature image acquisition module includes: A first shallow weight feature extraction submodule is used to input the shallow feature image into the first weight feature acquisition module to obtain a first shallow weight feature; A second shallow weight feature extraction submodule is used to input the shallow feature image into the second weight feature acquisition module to obtain a second shallow weight feature; A third shallow weight acquisition submodule is used to activate a third shallow weight feature obtained by fusing the first shallow weight feature and the second shallow weight feature to obtain a third shallow weight; the dimension of the third shallow weight is equal to the number of channels of the shallow feature image; The weighted shallow feature image acquisition submodule is used to multiply the third shallow weight by each channel of the shallow feature image to obtain the weighted shallow feature image.

10. The product intelligent identification device according to claim 8, characterized in that: The channel attention model includes: a first weight feature acquisition module and a second weight feature acquisition module; the weighted deep feature image acquisition module includes: A first deep weight feature extraction submodule is configured to input the deep feature image into the first weight feature acquisition module to obtain a first deep weight feature; A second deep weight feature extraction submodule is used to input the deep feature image into the second weight feature acquisition module to obtain a second deep weight feature; A third deep weight acquisition submodule is used to activate a third deep weight feature obtained by fusing the first deep weight feature and the second deep weight feature to obtain a third deep weight; the dimension of the third deep weight is equal to the number of channels of the deep feature image; The weighted deep feature image acquisition submodule is used to multiply the third deep weight by each channel of the deep feature image to obtain the weighted deep feature image.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images