Method and apparatus for identifying goods

Abstract

The application provides a kind of goods identification method and device, the goods identification method includes: obtaining the image of goods to be identified;The first feature map is obtained by extracting the feature of the image of goods to be identified;First feature map is input into preset attention model and is adjusted to obtain the second feature map;Image fusion is carried out based on first feature map and second feature map, and third feature map is obtained;Based on third feature map, goods identification is carried out, and the target goods identification result of the image of goods to be identified is obtained.The application extracts the first feature map of the relatively shallow image of goods to be identified, then adjusts the weight of the first feature map based on the preset attention model to obtain the second feature map, finds useful information, fuses the second feature map and the first feature map, obtains the third feature map and carries out goods identification, and the third feature map has relatively rich feature information, which can improve the accuracy of goods identification.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the field of artificial intelligence, and particularly relates to a goods identification method and device. BACKGROUND

[0002] In order to master the sales situation and display mode of goods, goods placed on shelves are generally checked manually. However, a large number of personnel are required to participate in the checking of a large number of sites, which is costly and time-consuming. With the development of artificial intelligence, there are schemes for goods identification in the prior art, which use a neural network model to identify goods and then classify the goods. However, most of the existing technologies use a conventional convolutional neural network (CNN) to extract features and then classify the features. However, the features extracted in this way are not rich enough, and the accuracy of goods identification is low.

[0003] That is, the accuracy of goods identification in the prior art is low. SUMMARY

[0004] The present application provides a goods identification method and device, which aims to solve the problem of low accuracy of goods identification in the prior art.

[0005] In a first aspect, the present application provides a goods identification method, which comprises:

[0006] obtaining a to-be-identified goods image;

[0007] performing feature extraction on the to-be-identified goods image to obtain a first feature map;

[0008] inputting the first feature map into a preset attention model to adjust the weight and obtain a second feature map;

[0009] performing image fusion based on the first feature map and the second feature map to obtain a third feature map;

[0010] performing goods identification based on the third feature map to obtain a target goods identification result of the to-be-identified goods image.

[0011] Optionally, the performing feature extraction on the to-be-identified goods image to obtain a first feature map comprises:

[0012] obtaining a preset target residual network model, wherein the target residual network model comprises a first residual block and a second residual block, and the second residual block is configured to perform a convolution operation on the output of the first residual block,

[0013] inputting the to-be-identified goods image into the target residual network model to obtain the first feature map output by the first residual block for feature extraction on the to-be-identified goods image;

[0014] The image fusion based on the first feature map and the second feature map obtains a third feature map, including:

[0015] The second residual block performs a convolution operation on the first feature map to obtain a fourth feature map;

[0016] The image fusion based on the fourth feature map and the second feature map obtains the third feature map.

[0017] Optionally, the image fusion based on the fourth feature map and the second feature map obtains the third feature map, including:

[0018] Down-sampling processing is performed on the second feature map to obtain a down-sampling processed second feature map;

[0019] The fourth feature map and the down-sampling processed second feature map are fused to obtain the third feature map.

[0020] Optionally, the product identification based on the third feature map obtains a target product identification result of the to-be-identified product image, including:

[0021] Up-sampling processing is performed on the third feature map to obtain an up-sampling processed third feature map;

[0022] The up-sampling processed third feature map is input into the preset attention model to perform weight adjustment to obtain a fifth feature map;

[0023] Down-sampling processing is performed on the fifth feature map to obtain a down-sampling processed fifth feature map;

[0024] The down-sampling processed fifth feature map and the third feature map are fused to obtain a target feature map;

[0025] The product identification based on the target feature map obtains a target product identification result of the to-be-identified product image.

[0026] Optionally, the product identification based on the target feature map obtains a target product identification result of the to-be-identified product image, including:

[0027] The target feature map is input into a preset attention model to perform weight adjustment to obtain a sixth feature map;

[0028] The sixth feature map and the fourth feature map are fused to obtain a seventh feature map;

[0029] The product identification based on the seventh feature map obtains a target product identification result of the to-be-identified product image.

[0030] Optionally, the obtaining the preset target residual network model comprises:

[0031] obtaining a preset residual network model, wherein the preset residual network model is any one of resnet18, resnet34, resnet50, resnet101, and resnet152;

[0032] weighting the cross-entropy loss, the triplet loss, and the center loss function according to the preset weight coefficient to obtain a total loss function;

[0033] training the preset residual network model based on the total loss function and a preset training set to obtain the target residual network model.

[0034] Optionally, the preset attention model is a Transformer model.

[0035] In a second aspect, the present application provides a goods identification device, the goods identification device comprising:

[0036] an acquisition unit configured to acquire a to-be-identified goods image;

[0037] a feature extraction unit configured to perform feature extraction on the to-be-identified goods image to obtain a first feature map;

[0038] a weight adjustment unit configured to input the first feature map into a preset attention model to perform weight adjustment and obtain a second feature map;

[0039] a fusion unit configured to perform image fusion based on the first feature map and the second feature map to obtain a third feature map;

[0040] an identification unit configured to perform goods identification based on the third feature map to obtain a target goods identification result of the to-be-identified goods image.

[0041] Optionally, the feature extraction unit is configured to:

[0042] obtaining a preset target residual network model, wherein the target residual network model comprises a first residual block and a second residual block, the second residual block is configured to perform convolution operation on the output of the first residual block,

[0043] inputting the to-be-identified goods image into the target residual network model to obtain the first feature map output by the first residual block for feature extraction on the to-be-identified goods image;

[0044] the fusion unit is configured to:

[0045] obtain a fourth feature map by performing a convolution operation on the first feature map according to the second residual block;

[0046] perform image fusion based on the fourth feature map and the second feature map to obtain the third feature map.

[0047] Optionally, the fusion unit is configured to:

[0048] perform down-sampling processing on the second feature map to obtain a second feature map after down-sampling processing;

[0049] fuse the fourth feature map and the second feature map after down-sampling processing to obtain the third feature map.

[0050] Optionally, the identification unit is configured to:

[0051] perform up-sampling processing on the third feature map to obtain a third feature map after up-sampling processing;

[0052] input the third feature map after up-sampling processing into the preset attention model to perform weight adjustment to obtain a fifth feature map;

[0053] perform down-sampling processing on the fifth feature map to obtain a fifth feature map after down-sampling processing;

[0054] fuse the fifth feature map after down-sampling processing and the third feature map to obtain a target feature map;

[0055] perform product identification based on the target feature map to obtain a target product identification result of the to-be-identified product image.

[0056] Optionally, the identification unit is configured to:

[0057] input the target feature map into a preset attention model to perform weight adjustment to obtain a sixth feature map;

[0058] fuse the sixth feature map and the fourth feature map to obtain a seventh feature map;

[0059] perform product identification based on the seventh feature map to obtain a target product identification result of the to-be-identified product image.

[0060] Optionally, the feature extraction unit is configured to:

[0061] obtain a preset residual network model, wherein the preset residual network model is any one of resnet18, resnet34, resnet50, resnet101, and resnet152;

[0062] The cross-entropy loss, the triplet loss, and the center loss function are weighted according to preset weight coefficients to obtain a total loss function.

[0063] The preset residual network model is trained based on the total loss function and a preset training set to obtain the target residual network model.

[0064] Optionally, the preset attention model is a Transformer model.

[0065] In a third aspect, the present application provides a computer device, which comprises:

[0066] one or more processors;

[0067] a memory; and

[0068] one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the item identification method according to any one of the first aspect.

[0069] In a fourth aspect, the present application provides a computer readable storage medium, which stores a plurality of instructions adapted to be loaded by a processor to perform the steps in the item identification method according to any one of the first aspect.

[0070] The present application provides an item identification method and device, which comprises: obtaining an image of an item to be identified; performing feature extraction on the image of the item to be identified to obtain a first feature map; inputting the first feature map into a preset attention model to adjust the weight and obtain a second feature map; performing image fusion based on the first feature map and the second feature map to obtain a third feature map; and performing item identification based on the third feature map to obtain a target item identification result of the image of the item to be identified. In the case of low item identification accuracy in the prior art, the present application creatively proposes an item identification method, which adjusts the weight of the first feature map based on a preset attention model after extracting the relatively shallow first feature map of the image of the item to be identified, obtains a second feature map, finds useful information, fuses the second feature map and the first feature map, obtains a third feature map, and performs item identification, the third feature map has relatively rich feature information, and the accuracy of item identification can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0072] Figure 1 is a scene schematic diagram of a goods identification system provided by an embodiment of the present application;

[0073] Figure 2 is a flow schematic diagram of an embodiment of a goods identification method in the present application;

[0074] Figure 3 is a structure schematic diagram of a target residual network model in an embodiment of a goods identification method in the present application;

[0075] Figure 4 is a structure schematic diagram of an embodiment of a goods identification device provided by the present application;

[0076] Figure 5 is a structure schematic diagram of an embodiment of a computer device provided by the present application. DETAILED DESCRIPTION

[0077] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0078] In the description of the present application, it should be understood that the orientation or positional relationship indicated by the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “upper”, “lower”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inner”, “outer” and the like is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms “first” and “second” are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features limited by “first” and “second” can explicitly or implicitly include one or more features. In the description of the present application, the meaning of “multiple” is two or more, unless otherwise specifically limited.

[0079] In this application, the word "exemplary" is used to mean "serving as an example, instance, or illustration." Any implementation described as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations. The following description is presented to enable any person skilled in the art to make and use the application. In the following description, for purposes of explanation, numerous details are set forth. It is apparent, however, to one skilled in the art, that the present application can be practiced without the use of these specific details. In other instances, well known structures and processes are not elaborated upon in order not to obscure the description of the present application with unnecessary detail. Thus, the present application is not intended to be limited by the embodiments shown, but is to be accorded with the widest scope consistent with the principles and features disclosed.

[0080] The embodiments of the present application provide a goods identification method and device, which are described in detail as follows.

[0081] Please refer to Figure 1 , Figure 1 is a schematic diagram of a scene of a goods identification system provided by the embodiments of the present application. The goods identification system can include a computer device 100, and the computer device 100 is integrated with a goods identification device.

[0082] In the embodiments of the present application, the computer device 100 can be a stand-alone server, or a server network or a server cluster composed of servers. For example, the computer device 100 described in the embodiments of the present application includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets, or a cloud server composed of a plurality of servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0083] In the embodiments of the present application, the computer device 100 described above can be a general-purpose computer device or a special-purpose computer device. In specific implementation, the computer device 100 can be a desktop computer, a laptop computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, an embedded device, etc. The embodiments do not limit the type of the computer device 100.

[0084] Those skilled in the art can understand that the application environment shown in Figure 1 is only one application scenario of the solution of the present application, and does not constitute a limitation on the application scenarios of the solution of the present application. Other application environments can include more or fewer computer devices than those shown in Figure 1 , for example Figure 1Only one computer device is shown in the figure, and it can be understood that the goods identification system can also include one or more other computer devices that can process data, which is not limited here.

[0085] In addition, as Figure 1 shown, the goods identification system can also include a memory 200 for storing data.

[0086] It should be noted that Figure 1 the scene diagram of the goods identification system shown is only an example, and the goods identification system and the scene described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. It can be known by those skilled in the art that, with the evolution of the goods identification system and the appearance of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0087] First, the present application provides a goods identification method, which comprises: acquiring a to-be-identified goods image; performing feature extraction on the to-be-identified goods image to obtain a first feature map; inputting the first feature map into a preset attention model to adjust the weight and obtain a second feature map; performing image fusion based on the first feature map and the second feature map to obtain a third feature map; and performing goods identification based on the third feature map to obtain a target goods identification result of the to-be-identified goods image.

[0088] As Figure 2 shown, Figure 2 is an embodiment flowchart of the goods identification method in the embodiments of the present application, which comprises the following steps S201-S205:

[0089] S201, acquiring a to-be-identified goods image.

[0090] In a specific embodiment, a camera is controlled to take a picture of a to-be-identified goods to obtain a to-be-identified goods image. The to-be-identified goods can be laundry detergent, instant noodles, etc. Of course, the to-be-identified goods image can be pre-stored.

[0091] Generally, goods are placed on shelves, and the shelves have multiple compartments, each of which has goods. The image captured may have multiple goods. In order to identify multiple goods, in another specific embodiment, a current shelf image obtained by a camera taking a picture of a shelf is acquired, goods detection is performed on the current shelf image to obtain multiple goods detection boxes, and images in the multiple goods detection boxes are cropped to obtain multiple to-be-identified goods images. Specifically, a preset target detection network model is used to perform goods detection on the shelf image to obtain multiple goods detection boxes. The target detection network model can be a target detection model such as YOLOV5 or SSD, which can be selected according to specific circumstances.

[0092] Preferably, the target detection network model can be YOLOV5, and the target detection network model is a two-classification model for classifying images into a background category and a product category. Since YOLOV5 is single-stage, fast, and we use a two-classification method for detection, the detection recall rate and accuracy are also high. Specifically, although there are many product categories, such as 100,000 product categories in a certain data set, we regard all these product categories as one category without distinction, plus the background, which is actually a two-classification problem, thereby greatly reducing the difficulty of detection. Since the two-classification method is used, the model can accurately detect the product, thereby laying a good foundation for the subsequent recognition.

[0093] In a specific embodiment, a reference shelf image is obtained, the reference shelf image is an image obtained by a camera historically photographing a shelf, the reference shelf image has the same size as the current shelf image, the reference shelf image is labeled with a plurality of grid frames, the plurality of grid frames can be manually labeled, the reference shelf image is overlapped with the current shelf image, and images in the plurality of grid frames in the current shelf image are cropped to obtain a plurality of to-be-recognized product images. The to-be-recognized product image is obtained by taking the reference shelf image as a template, and for a scene in which the camera and the shelf position are unchanged, the efficiency of obtaining the to-be-recognized product image can be improved.

[0094] S202, feature extraction is performed on the to-be-recognized product image to obtain a first feature map.

[0095] In a specific embodiment, feature extraction is performed on the to-be-recognized product image to obtain a first feature map, which can include: obtaining a preset target residual network model, inputting the to-be-recognized product image into the target residual network model, and obtaining a first feature map output by the first residual block for feature extraction on the to-be-recognized product image.

[0096] The target residual network model includes a first residual block and a second residual block, and the second residual block is configured to perform a convolution operation on the output of the first residual block.

[0097] In a specific embodiment, obtaining a preset target residual network model can include:

[0098] (1) obtaining a preset residual network model, wherein the preset residual network model is any one of resnet18, resnet34, resnet50, resnet101, and resnet152.

[0099] The ResNet network is referenced to the VGG19 network, is modified on the basis, and adds residual units through a short circuit mechanism. The changes mainly reflect that the ResNet directly uses the convolution of stride = 2 to do down sampling, and uses the global average pool layer to replace the full connection layer. An important design principle of the ResNet is that when the feature map size is reduced by half, the number of feature maps is doubled, which maintains the complexity of the network layer. The ResNet increases the short circuit mechanism between every two layers compared with the ordinary network, which forms residual learning, wherein the dashed line indicates that the number of feature maps changes. For the 18-layer and 34-layer ResNet, the residual learning between two layers is performed, when the network is deeper, the residual learning between three layers is performed, the three-layer convolution kernels are 1x1, 3x3 and 1x1, and one of the notable features is that the number of feature maps of the hidden layer is relatively small, and is 1 / 4 of the number of output feature maps.

[0100] The Resnet50 includes conv1, conv2_x, conv3_x, conv4_x and conv5_x. The Resnet50 first performs convolution operation on the input conv1, and then includes four residual blocks (ResidualBlock). The four residual blocks Block are conv2_x, conv3_x, conv4_x and conv5_x, and finally perform full connection operation to facilitate the classification task.

[0101] The first residual block and the second residual block can be any two of the four residual blocks in the Resnet50.

[0102] (2) The cross-entropy loss, the triplet loss and the center loss function are weighted according to the preset weight coefficient to obtain a total loss function.

[0103] The cross-entropy loss L ID satisfies the relationship shown in the following formula (1),

[0104]

[0105] Wherein, qi represents the class of sample i, pi represents the probability that sample i belongs to a certain class, and N represents the number of samples entering the training network at a time.

[0106] In addition, the triplet loss tripletloss can make the distance between the same objects smaller and the distance between different objects farther. The triplet loss tripletloss satisfies the relationship shown in the following formula (2),

[0107] L Triplet =[d p -d n +α] + (2)

[0108] where d p and d n are the feature distances of positive sample pair and negative sample pair, and a is the margin setting for picking out difficult sample pairs, which is generally set to 0.3.

[0109] However, the triplet loss only considers the difference between d p and d n , and ignores the absolute distance between positive and negative sample pairs, such as d p = 0.3 and d n = 0.5, their loss values are 0.3-0.5+0.3 = 0.1, and the loss value of another pair d p = 1.3 and d n = 1.5 is also 0.1, although the loss values are the same, but the latter pair obviously fails to cluster the positive sample pair, and the former fails to separate the negative sample pair. Therefore, we add the center loss function.

[0110] where the center loss satisfies the relationship shown in the following formula (3),

[0111]

[0112] where y j is the label of the jth sample, is the cluster center with the label y j . Therefore, minimizing the center loss will increase the distance between classes and decrease the distance within the class.

[0113] where the preset weight coefficients include the weight coefficient of the cross-entropy loss, the weight coefficient of the triplet loss, and the weight coefficient of the circle loss. The preset weight coefficients can be set according to specific circumstances. Preferably, the weight coefficient of the cross-entropy loss is 1, the weight coefficient of the triplet loss is 1, and the weight coefficient of the circle loss is 0.0005.

[0114] The total loss function satisfies the relationship shown in the following formula (4):

[0115] L = L ID + L Triplet + βL C (4)

[0116] where β = 0.0005.

[0117] (3) training the preset residual network model based on the total loss function and the preset training set to obtain a target residual network model.

[0118] The preset training set includes a plurality of samples and corresponding real labels. Specifically, the plurality of samples in the preset training set are input into the preset residual network model to predict the predicted categories of the plurality of samples, and the loss values of the predicted categories of the plurality of samples and the real labels are calculated according to the total loss function. When the loss value is not less than a predetermined loss, the network weights of the preset residual network model are updated to complete one iteration update. The network weights of the preset residual network model are iteratively updated for multiple times until the loss value is less than the predetermined loss, and the preset residual network model at this time is determined as the target residual network model.

[0119] In one specific embodiment, the first residual block is Block2, that is, conv2_x, and the second residual block is Block5, that is, conv5_x. For example, the first feature map size is 56*56*256.

[0120] S203, input the first feature map into the preset attention model for weight adjustment to obtain a second feature map.

[0121] In one specific embodiment, the preset attention model is a Transformer model. In other embodiments, the preset attention model can be a spatial attention model or a channel attention model. The Transformer model discards the CNN and RNN used in previous deep learning tasks. The model is widely used in the field of NLP, such as machine translation, question and answer system, text summary, and speech recognition. The overall structure of the Transformer is the same as that of the seq2seq model, and the encoer-decoder architecture is also used in the Transformer model. However, the structure is more complex than the Attention. In the paper, the encoder layer is stacked with 6 encoders, and the decoder layer is also stacked with 6 encoders.

[0122] In one specific embodiment, the first feature map of h*w*c is expanded by using the Flatten operation to obtain a vector of h*w*c, because the operation of the Transformer model needs to take a sequence of vectors as input, and then the Transformer model calculates the similarity between all vectors, that is, a (h*w)*(h*w) matrix is obtained. Finally, the matrix and the h*w*c vector are multiplied to obtain a h*w*c second feature map.

[0123] However, the traditional fusion of bottom and high-level information limits the performance improvement because of the lack of semantic information at the bottom. Therefore, we use the Transformer model to fuse the bottom-level information as a guide for high-level semantic information, which can improve the accuracy of the second feature map and improve the accuracy of product identification.

[0124] The Transformer model uses a conventional Softmax layer, and the formula of the conventional Softmax layer is relatively complex, which means that each prediction needs to be calculated based on the entire data set, which will be a huge time overhead and the training speed is slow. In order to improve the training speed, the Transformer model is obtained; the softmax layer of the Transformer model is replaced with a hierarchical softmax layer to obtain a preset attention model. Hierarchical Softmax is a way to reduce the amount of calculation using Huffman tree structure. Huffman tree is a binary tree, which is actually constantly doing binary classification. Replacing the softmax layer of the Transformer model with a hierarchical softmax layer can speed up the training speed of the model.

[0125] S204, image fusion based on the first feature map and the second feature map is obtained The third feature map.

[0126] In one specific embodiment, the first feature map and the second feature map are image fused to obtain the third feature map.

[0127] In another specific embodiment, the first feature map and the second feature map are image fused to obtain the third feature map, which can include:

[0128] (1) Obtain the second residual block to perform convolution operation on the first feature map to obtain the fourth feature map.

[0129] For example, using Block3 to perform convolution operation on the first feature map of 56*56*256 to obtain the fourth feature map of 28*28*512. The channel number of the fourth feature map is greater than that of the first feature map, and the size of the fourth feature map is smaller than that of the first feature map.

[0130] (2) Image fusion based on the fourth feature map and the second feature map is obtained The third feature map.

[0131] Specifically, the second feature map is down-sampled to obtain a down-sampled second feature map; and the fourth feature map and the down-sampled second feature map are fused to obtain a third feature map. More specifically, the second feature map of 56*56*256 is down-sampled by 2 to obtain a down-sampled second feature map, and the fourth feature map and the down-sampled second feature map are fused to obtain a third feature map. The main purposes of down-sampling are two-fold: 1. to make the image conform to the size of the display area; and 2. to generate a thumbnail image corresponding to the image.

[0132] S204, performing product identification based on the third feature map to obtain a target product identification result of the to-be-identified product image.

[0133] In one specific embodiment, the target residual network model includes a pooling layer and a fully connected layer, and the third feature map is sequentially input into the pooling layer and the fully connected layer to obtain the target product identification result. The target product identification result includes a category of the to-be-identified product image.

[0134] In another specific embodiment, performing product identification based on the third feature map to obtain a target product identification result of the to-be-identified product image includes:

[0135] (1) up-sampling the third feature map to obtain an up-sampled third feature map.

[0136] Specifically, the third feature map is up-sampled by 2 to obtain an up-sampled third feature map. Figure 2 up-sampling the third feature map to obtain an up-sampled third feature map. The main purpose of up-sampling is to enlarge the original image so that it can be displayed on a display device with higher resolution.

[0137] S204, performing product identification based on the third feature map to obtain a target product identification result of the to-be-identified product image.

[0138] In one specific embodiment, performing product identification based on the third feature map to obtain a target product identification result of the to-be-identified product image includes:

[0139] (2) inputting the up-sampled third feature map into a preset attention model to adjust the weight to obtain a fifth feature map.

[0140] (3) down-sampling the fifth feature map to obtain a down-sampled fifth feature map.

[0141] Specifically, the fifth feature map is down-sampled by 2 to obtain a down-sampled fifth feature map.

[0142] (4) fusing based on the fifth feature map after down-sampling processing and the third feature map to obtain a target feature map.

[0143] (5) performing product identification based on the target feature map to obtain a target product identification result of the to-be-identified product image.

[0144] In one specific embodiment, the target feature map is sequentially input into a pooling layer and a fully connected layer to obtain the target product identification result.

[0145] In another specific embodiment, the target feature map is input into a preset attention model to adjust weights to obtain a sixth feature map; the sixth feature map and the fourth feature map are fused to obtain a seventh feature map; and product identification is performed based on the seventh feature map to obtain a target product identification result of the to-be-identified product image. That is, the seventh feature map obtained by fusing the fourth feature map without being processed by the preset attention model and the sixth feature map processed by the preset attention model has more abundant features, which can improve the product identification accuracy.

[0146] In yet another specific embodiment, the target residual network model includes a first prediction branch and a second prediction branch, the first prediction branch includes a pooling layer and a fully connected layer, and the second prediction branch includes a pooling layer and a fully connected layer. The loss function of the first prediction branch is the total loss function. The loss function of the second prediction branch is the total loss function. The target product identification result of the to-be-identified product image obtained by performing product identification based on the target feature map includes: inputting the fourth feature map into the first prediction branch to obtain a first product identification result; inputting the target feature map into the second prediction branch to obtain a second product identification result; and determining the target product identification result based on the first product identification result and the second product identification result.

[0147] Specifically, the second product identification result includes a first confidence of each product prediction category; and the target product identification result includes a second confidence of each product prediction category. The first confidence and the second confidence are used to represent the probability that the product is the product prediction category. For each product prediction category, the first confidence and the second confidence are weighted and summed according to a preset result weight coefficient to obtain a target confidence of each product prediction category, and the product prediction category with the highest target confidence in each product prediction category is determined as the target product identification result. The preset result weight coefficient includes a weight coefficient of the first prediction branch and a weight coefficient of the second prediction branch. Preferably, the weight coefficient of the first prediction branch is smaller than the weight coefficient of the second prediction branch.

[0148] Further, refer to Figure 3 , Figure 3Fig. 1 is a structural schematic diagram of a target residual network model in an embodiment of a product identification method in the present application. The target residual network model comprises a first residual block Block2, a third residual block Block3, a fourth residual block Block4, and a second residual block Block5.

[0149] The image of the product to be identified is sequentially input into the first residual block Block2, the third residual block Block3, the fourth residual block Block4, and the second residual block Block5. The first residual block Block2 outputs a first feature map of 56*56*256; the third residual block Block3 outputs a feature map of 28*28*512; the fourth residual block Block4 outputs a feature map of 14*14*1024; and the second residual block Block5 outputs a fourth feature map of 7*7*2048.

[0150] The first feature map of 56*56*256 output by the first residual block Block2 is input into a preset attention model for weight adjustment, 2 times down-sampling, and fusion with the feature map of 28*28*512 output by the third residual block Block3, to obtain a first fusion feature map. The first fusion feature map is then input into the preset attention model for weight adjustment, 2 times down-sampling, and fusion with the feature map of 14*14*1024 output by the fourth residual block Block4, to obtain a second fusion feature map. The second fusion feature map is then input into the preset attention model for weight adjustment, 2 times down-sampling, and fusion with the fourth feature map of 7*7*2048 output by the second residual block Block5, to obtain a third feature map.

[0151] Further, the third feature map is subjected to 2 times up-sampling and fusion with the second fusion feature map, to obtain a third fusion feature map; the third fusion feature map is subjected to 2 times up-sampling and fusion with the first fusion feature map, to obtain a fourth fusion feature map; and the fourth fusion feature map is subjected to 2 times up-sampling and fusion with the first feature map of 56*56*256 output by the first residual block Block2, to obtain a fifth fusion feature map.

[0152] The fifth fusion feature map is input into the preset attention model for weight adjustment, 2 times down-sampling, and fusion with the fourth fusion feature map, to obtain a sixth fusion feature map; the sixth fusion feature map is then input into the preset attention model for weight adjustment, 2 times down-sampling, and fusion with the third fusion feature map, to obtain a seventh fusion feature map; and the seventh fusion feature map is then input into the preset attention model for weight adjustment, 2 times down-sampling, and fusion with the fourth feature map of 7*7*2048 output by the second residual block Block5, to obtain a target feature map. Product identification is performed based on the target feature map.

[0153] In order to better implement the goods identification method in the embodiments of the present application, on the basis of the goods identification method, the embodiments of the present application also provide a goods identification device, as shown in Figure 4 The goods identification device 400 comprises:

[0154] The acquisition unit 401 is configured to acquire a to-be-identified goods image.

[0155] The feature extraction unit 402 is configured to perform feature extraction on the to-be-identified goods image to obtain a first feature map.

[0156] The weight adjustment unit 403 is configured to input the first feature map into a preset attention model to perform weight adjustment, and obtain a second feature map.

[0157] The fusion unit 404 is configured to perform image fusion based on the first feature map and the second feature map, and obtain a third feature map.

[0158] The identification unit 405 is configured to perform goods identification based on the third feature map, and obtain a target goods identification result of the to-be-identified goods image.

[0159] Optionally, the feature extraction unit 402 is configured to:

[0160] acquire a preset target residual network model, wherein the target residual network model comprises a first residual block and a second residual block, and the second residual block is configured to perform convolution operation on an output of the first residual block,

[0161] input the to-be-identified goods image into the target residual network model to obtain a first feature map output by the first residual block for feature extraction on the to-be-identified goods image;

[0162] The fusion unit 404 is configured to:

[0163] acquire a fourth feature map obtained by performing convolution operation on the first feature map by the second residual block;

[0164] perform image fusion based on the fourth feature map and the second feature map to obtain the third feature map.

[0165] Optionally, the fusion unit 404 is configured to:

[0166] perform down-sampling processing on the second feature map to obtain a second feature map after down-sampling processing;

[0167] fuse the fourth feature map and the second feature map after down-sampling processing to obtain the third feature map.

[0168] Optionally, the identification unit 405 is configured to:

[0169] perform up-sampling processing on the third feature map to obtain a third feature map after up-sampling processing;

[0170] input the third feature map after the up-sampling processing into a preset attention model for weight adjustment to obtain a fifth feature map;

[0171] perform down-sampling processing on the fifth feature map to obtain a fifth feature map after the down-sampling processing;

[0172] fuse the fifth feature map after the down-sampling processing and the third feature map to obtain a target feature map;

[0173] perform product identification based on the target feature map to obtain a target product identification result of the product image to be identified.

[0174] Optionally, the identification unit 405 is configured to:

[0175] input the target feature map into a preset attention model for weight adjustment to obtain a sixth feature map;

[0176] fuse the sixth feature map and the fourth feature map to obtain a seventh feature map;

[0177] perform product identification based on the seventh feature map to obtain a target product identification result of the product image to be identified.

[0178] Optionally, the feature extraction unit 402 is configured to:

[0179] obtain a preset residual network model, wherein the preset residual network model is any one of resnet18, resnet34, resnet50, resnet101 and resnet152;

[0180] weight the cross-entropy loss, the triplet loss and the center loss function according to a preset weight coefficient to obtain a total loss function;

[0181] train the preset residual network model based on the total loss function and a preset training set to obtain a target residual network model.

[0182] Optionally, the preset attention model is a Transformer model.

[0183] Embodiments of the present application also provide a computer device integrating any one of the product identification apparatuses provided by embodiments of the present application, and the computer device comprises:

[0184] one or more processors;

[0185] a memory; and

[0186] One or more application programs, wherein the one or more application programs are stored in the memory and configured to perform steps of the item identification method in any of the above item identification method embodiments by the processor.

[0187] As shown in Figure 5 the structural schematic diagram of a computer device related to the embodiments of the present application, in particular:

[0188] The computer device can include a processor 501 with one or more processing cores, a memory 502 with one or more computer readable storage media, a power supply 503, an input unit 504, and the like. Those skilled in the art can understand that the computer device structure shown in the figure does not constitute a limitation on the computer device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements. Among them:

[0189] The processor 501 is the control center of the computer device, which connects various parts of the computer device through various interfaces and lines, executes the software programs and / or modules stored in the memory 502 and the data stored in the memory 502, processes the data of the computer device, and thus monitors the computer device as a whole. Optionally, the processor 501 can include one or more processing cores; the processor 501 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like, and preferably, the processor 501 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 501.

[0190] The memory 502 can be used to store software programs and modules, and the processor 501 executes various function applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the computer device, etc. In addition, the memory 502 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 502 can also include a memory controller to provide access for the processor 501 to the memory 502.

[0191] The computer device further includes a power supply 503 for powering various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 503 can also include one or more than one direct current or alternating current power supply, a recharging system, a power failure detection circuit, a power converter or inverter, a power state indicator, and the like.

[0192] The computer device can further include an input unit 504, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0193] Although not shown, the computer device can also include a display unit and the like, which will not be described here. Specifically, in the present embodiment, the processor 501 in the computer device will load the executable file corresponding to the process of one or more than one application program into the memory 502 according to the following instructions, and run the application program stored in the memory 502 by the processor 501, so as to realize various functions, as follows:

[0194] Obtaining a to-be-recognized product image;

[0195] Performing feature extraction on the to-be-recognized product image to obtain a first feature map;

[0196] Inputting the first feature map into a preset attention model to adjust the weight, and obtaining a second feature map;

[0197] Performing image fusion based on the first feature map and the second feature map to obtain a third feature map;

[0198] Performing product recognition based on the third feature map to obtain a target product recognition result of the to-be-recognized product image.

[0199] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions or by controlling relevant hardware by instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0200] To this end, the embodiments of the present application provide a computer readable storage medium, which can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any of the product identification methods provided by the embodiments of the present application. For example, the computer program loaded by the processor can execute the following steps:

[0201] Obtaining a to-be-identified product image;

[0202] Performing feature extraction on the to-be-identified product image to obtain a first feature map;

[0203] Inputting the first feature map into a preset attention model to adjust the weight and obtain a second feature map;

[0204] Performing image fusion based on the first feature map and the second feature map to obtain a third feature map;

[0205] Performing product identification based on the third feature map to obtain a target product identification result of the to-be-identified product image.

[0206] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the detailed description of other embodiments above, which will not be repeated here.

[0207] In specific implementation, the above various units or structures can be implemented as independent entities, or can be combined as the same or several entities, and the specific implementation of the above various units or structures can be referred to the method embodiments above, which will not be repeated here.

[0208] The specific implementation of the above various operations can be referred to the embodiments above, which will not be repeated here.

[0209] The above has introduced in detail a product identification method and device provided by the embodiments of the present application, and the principle and implementation manner of the present application have been described by applying specific examples; the above embodiment description is only for helping to understand the method and its core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and in view of the above, the content of the specification should not be understood as the limitation of the present application.

Claims

1. A method of identifying an item of merchandise, the method comprising: The method comprises the following steps: obtaining a to-be-recognized product image; performing feature extraction on the to-be-recognized product image to obtain a first feature map; wherein a preset target residual network model is obtained, the target residual network model comprises a first residual block and a second residual block, the second residual block is used for performing convolution operation on the output of the first residual block; the to-be-recognized product image is input into the target residual network model to obtain the first feature map output by the first residual block in the process of feature extraction on the to-be-recognized product image; inputting the first feature map into a preset attention model to adjust the weight, and obtaining a second feature map; performing image fusion based on the first feature map and the second feature map to obtain a third feature map; wherein a fourth feature map is obtained by performing convolution operation on the first feature map by the second residual block; the fourth feature map and the second feature map are fused to obtain the third feature map; performing product recognition based on the third feature map to obtain a target product recognition result of the to-be-recognized product image.

2. The article identification method of claim 1, wherein, The image fusion based on the fourth feature map and the second feature map to obtain the third feature map comprises: performing down-sampling processing on the second feature map to obtain a down-sampling processed second feature map; fusing the fourth feature map and the down-sampling processed second feature map to obtain the third feature map.

3. The article identification method of claim 2, wherein, The product recognition based on the third feature map to obtain the target product recognition result of the to-be-recognized product image comprises: performing up-sampling processing on the third feature map to obtain an up-sampling processed third feature map; inputting the up-sampling processed third feature map into the preset attention model to adjust the weight, and obtaining a fifth feature map; performing down-sampling processing on the fifth feature map to obtain a down-sampling processed fifth feature map; fusing the down-sampling processed fifth feature map and the third feature map to obtain a target feature map; performing product recognition based on the target feature map to obtain a target product recognition result of the to-be-recognized product image.

4. The article identification method of claim 3, wherein, The product recognition based on the target feature map to obtain the target product recognition result of the to-be-recognized product image comprises: inputting the target feature map into a preset attention model to adjust the weight, and obtaining a sixth feature map; fusing the sixth feature map and the fourth feature map to obtain a seventh feature map; performing product recognition based on the seventh feature map to obtain a target product recognition result of the to-be-recognized product image.

5. The product recognition method of claim 1, wherein, The preset target residual network model comprises: obtaining a preset residual network model, wherein the preset residual network model is any one of resnet18, resnet34, resnet50, resnet101 and resnet152; weighting cross-entropy loss, triplet loss and center loss function according to preset weight coefficients to obtain a total loss function; training the preset residual network model based on the total loss function and a preset training set to obtain the target residual network model.

6. The product recognition method of claim 1, wherein, The preset attention model is a Transformer model.

7. An item identification apparatus, characterized by, The goods identification device comprises: An acquisition unit configured to acquire a to-be-identified goods image; A feature extraction unit configured to perform feature extraction on the to-be-identified goods image to obtain a first feature map; wherein a preset target residual network model is acquired, the target residual network model comprising a first residual block and a second residual block, the second residual block being configured to perform convolution operation on an output of the first residual block; the to-be-identified goods image is input into the target residual network model to obtain the first feature map output by the first residual block in feature extraction on the to-be-identified goods image; A weight adjustment unit configured to input the first feature map into a preset attention model to perform weight adjustment, thereby obtaining a second feature map; A fusion unit configured to perform image fusion based on the first feature map and the second feature map, thereby obtaining a third feature map; wherein a fourth feature map is obtained by performing convolution operation on the first feature map by the second residual block; image fusion is performed based on the fourth feature map and the second feature map, thereby obtaining the third feature map; An identification unit configured to perform goods identification based on the third feature map, thereby obtaining a target goods identification result of the to-be-identified goods image.

8. A computer device, comprising: The computer device comprises: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the goods identification method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the goods identification method of any one of claims 1 to 6.

Images