Commodity identification method, device and equipment and computer readable storage medium

Abstract

The application provides a commodity identification method, device and equipment and a computer readable storage medium, which are used to improve the identification accuracy of commodities in an image to a certain extent. The commodity identification method provided by the application comprises the following steps: acquiring an image to be identified, wherein the image to be identified is used to identify commodities contained in the image; inputting the image to be identified into a commodity identification model to perform commodity identification processing and obtaining commodity feature information output by the commodity identification model, wherein the commodity identification model comprises a first convolution layer, a second convolution layer and a full connection layer in sequence, a transmission channel is arranged between the first convolution layer and the full connection layer, the transmission channel is used to output a feature vector output by the first convolution layer to the full connection layer, so that the full connection layer performs optimization processing according to the feature vector output by the first convolution layer; and determining commodities contained in the image to be identified according to the commodity feature information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a commodity recognition method and device, equipment and a computer readable storage medium. BACKGROUND

[0002] Behind the continuous development of unmanned supermarkets and supermarket intelligent management, image recognition technology provides strong technical support and plays an important role.

[0003] Image recognition technology, which can also be understood as artificial intelligence (AI) technology, in the images collected on site in the supermarket, contains various commodities on the supermarket shelves. Image recognition of these images can identify the commodity situation on the supermarket shelves, and thus can monitor the display and sales of the commodities in the supermarket, and assist in adjusting the commodity supply strategy of the supermarket.

[0004] In the process of studying the existing related technology, the inventors found that the recognition accuracy of the commodity recognition achieved by the existing image recognition technology still needs to be improved, which limits the application effect to some extent. SUMMARY

[0005] The present application provides a commodity recognition method, device, equipment and computer readable storage medium, which can improve the recognition accuracy of commodities in the image to some extent.

[0006] In a first aspect, the present application provides a commodity recognition method, comprising:

[0007] obtaining a to-be-recognized image, wherein the to-be-recognized image is used to recognize the commodities contained in the image;

[0008] inputting the to-be-recognized image into a commodity recognition model to perform commodity recognition processing, and obtaining commodity feature information output by the commodity recognition model, wherein the commodity recognition model comprises a first convolutional layer, a second convolutional layer and a fully connected layer in sequence, a transmission channel is configured between the first convolutional layer and the fully connected layer, and the transmission channel is used to output a feature vector output by the first convolutional layer to the fully connected layer, so that the fully connected layer performs optimization processing according to the feature vector output by the first convolutional layer;

[0009] determining the commodities contained in the to-be-recognized image according to the commodity feature information.

[0010] In a first possible implementation manner of the first aspect of the present application, the full connection layer comprises a first full connection layer and a second full connection layer, the first full connection layer is configured to connect the second convolutional layer, and the second full connection layer is configured to connect the first convolutional layer through the transmission channel and receive the feature vector output by the first convolutional layer, and the full connection layer is configured to output both the product of the feature vector output by the second full connection layer and the correction factor and the feature vector output by the first full connection layer.

[0011] In a second possible implementation manner of the first aspect of the present application, the optimization processing comprises:

[0012] transforming the feature vector output by the first convolutional layer into a first feature vector with a dimension corresponding to a dimension of the full connection layer;

[0013] multiplying the first feature vector and a second feature vector corresponding to the full connection layer to obtain a product as a weight coefficient;

[0014] multiplying the weight coefficient and the second feature vector corresponding to the full connection layer to obtain a third feature vector as an output of the full connection layer.

[0015] In a third possible implementation manner of the first aspect of the present application, when the first convolutional layer comprises a plurality of sub-convolutional layers, a transmission channel is configured between each sub-convolutional layer and the full connection layer, so that the full connection layer performs optimization processing according to the feature vectors output by the plurality of sub-convolutional layers respectively.

[0016] In a fourth possible implementation manner of the first aspect of the present application, the obtaining of the image to be recognized comprises:

[0017] obtaining an initial image to be recognized;

[0018] inputting the initial image into the binary classification model to perform commodity image screening processing, obtaining a commodity image after removing a background image from the initial image as the image to be recognized.

[0019] In a fifth possible implementation manner of the first aspect of the present application, the commodity recognition model is obtained by training different commodity image corresponding to commodity feature information based on an initial neural network model of a residual network model structure type.

[0020] In a sixth possible implementation manner of the first aspect of the present application, the determining of the commodity contained in the image to be recognized according to the commodity feature information comprises:

[0021] matching the commodity feature information with different commodity feature information in a preset commodity feature information set;

[0022] The target commodity corresponding to the matched target commodity feature information is determined as a commodity recognition result of the to-be-recognized image.

[0023] In a second aspect, the present application provides a commodity recognition device, the device comprising:

[0024] An acquisition unit is configured to acquire a to-be-recognized image, wherein the to-be-recognized image is used to recognize a commodity contained in the image.

[0025] An identification unit is configured to input the to-be-recognized image into a commodity recognition model to perform commodity recognition processing and obtain commodity feature information output by the commodity recognition model, wherein the commodity recognition model comprises a first convolutional layer, a second convolutional layer and a full connection layer in sequence, a transmission channel is configured between the first convolutional layer and the full connection layer, and the transmission channel is configured to output a feature vector output by the first convolutional layer to the full connection layer, so that the full connection layer performs optimization processing according to the feature vector output by the first convolutional layer.

[0026] A determination unit is configured to determine a commodity contained in the to-be-recognized image according to the commodity feature information.

[0027] In a first possible implementation manner of the second aspect of the present application, the full connection layer comprises a first full connection layer and a second full connection layer, the first full connection layer is configured to connect the second convolutional layer, the second full connection layer is configured to connect the first convolutional layer through the transmission channel and receive the feature vector output by the first convolutional layer, and the full connection layer is configured to output both a product of a feature vector output by the second full connection layer and a correction factor and a feature vector output by the first full connection layer.

[0028] In a second possible implementation manner of the second aspect of the present application, the optimization processing comprises:

[0029] The feature vector output by the first convolutional layer is converted into a first feature vector with a dimension corresponding to a dimension of the full connection layer;

[0030] The first feature vector is multiplied by a second feature vector corresponding to the full connection layer to obtain a product as a weight coefficient;

[0031] The weight coefficient is multiplied by the second feature vector corresponding to the full connection layer to obtain a third feature vector as an output of the full connection layer.

[0032] In a third possible implementation manner of the second aspect of the present application, when the first convolutional layer comprises a plurality of sub-convolutional layers, a transmission channel is configured between each sub-convolutional layer and the full connection layer, so that the full connection layer performs optimization processing according to feature vectors respectively output by the plurality of sub-convolutional layers.

[0033] In a fourth possible implementation manner of the second aspect of the present application, the acquisition unit is specifically configured to:

[0034] acquire the initial image to be recognized;

[0035] input the initial image into the binary classification model for commodity image screening processing, to obtain a commodity image after removing a background image from the initial image, as the recognized image.

[0036] In a fifth possible implementation manner of the second aspect of the present application, the commodity recognition model is obtained by training commodity images corresponding to different commodity feature information based on an initial neural network model of a residual network model structure type.

[0037] In a sixth possible implementation manner of the second aspect of the present application, the determination unit is specifically configured to:

[0038] match the commodity feature information with different commodity feature information in the preset commodity feature information set;

[0039] determine a commodity corresponding to the matched target commodity feature information as the commodity recognition result of the recognized image.

[0040] In a third aspect, the present application further provides a commodity recognition device, comprising a processor and a memory, the memory storing a computer program, and the processor executing the computer program in the memory to perform the method provided in the first aspect of the present application or any possible implementation manner of the first aspect.

[0041] In a fourth aspect, the present application further provides a computer readable storage medium, the computer readable storage medium storing a plurality of instructions, the instructions being adapted to be loaded by a processor to execute the method provided in the first aspect of the present application or any possible implementation manner of the first aspect.

[0042] From the above, the present application has the following beneficial effects:

[0043] For commodity identification, a transmission channel is constructed between non-adjacent full connection layers and convolution layers in the commodity identification model. The feature vector output by the convolution layer at the bottom of the model during normal feature extraction can be output to the convolution layer at the next level and also output to the non-adjacent full connection layer at the higher level through the transmission channel. Since the convolution layer at the bottom pays more attention to basic information in the image, such as points, lines and angle elements, the full connection layer at the higher level has higher attention and pertinence to the contour or appearance of the commodity, and the feature processing of the full connection layer has smaller granularity, so that the model can output commodity feature information with higher accuracy to a certain extent according to the input image to be identified, and the commodity identification result with higher accuracy can be obtained. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present application, the drawings needed to be used in the description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0045] Figure 1 A scene schematic diagram of the commodity identification method of the present application;

[0046] Figure 2 A flowchart of the commodity identification method of the present application;

[0047] Figure 3 A structure schematic diagram of the commodity identification model of the present application;

[0048] Figure 4 Another structure schematic diagram of the commodity identification model of the present application;

[0049] Figure 5 Another structure schematic diagram of the commodity identification model of the present application;

[0050] Figure 6 A structure schematic diagram of the residual network model of the present application;

[0051] Figure 7 A flowchart of obtaining the image to be identified of the present application;

[0052] Figure 8 A flowchart of determining the commodity of the present application;

[0053] Figure 9 A structure schematic diagram of the commodity identification device of the present application;

[0054] Figure 10Fig. 1 is a schematic diagram of a structure of a commodity identification device according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0056] In the following description, specific embodiments of the present application will be described with reference to steps and symbolic representations of operations that are performed by one or more computers. Unless specifically stated otherwise, as is apparent from the following discussions, it is appreciated that throughout the specification, discussions utilizing terms such as "processing", "computing", "calculating", "determining", "displaying", "generating", "identifying", "selecting", "creating", "updating", "receiving", "sending", "accessing" or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical electronic signals.

[0057] The principles of the present application can be employed in many other general purpose or special purpose computing environments or configurations. Examples of well known computing systems, environments, and configurations that can be suitable for use with the present application include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microcomputer-based systems, programmable consumer electronics, network PC systems, minicomputers, mainframe computers, and distributed computing environments that include any of the above systems or devices.

[0058] The terms "first", "second", and "third" and the like in the present application are used to distinguish different objects, and are not used to describe a particular order. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0059] Before introducing the present application, the related content about the application background of the present application is introduced first.

[0060] The commodity identification method, device and computer readable storage medium provided by the present application can be applied to a commodity identification device, and are used to improve the identification accuracy of commodities in an image to a certain extent.

[0061] The method mentioned in the present application can be executed by a commodity identification device, or a server device, a physical host, a user equipment (UE) or other commodity identification devices of different types integrated with the device, wherein the device can be implemented in hardware or software, and the UE can be a terminal device such as a smartphone, a tablet computer, a notebook computer, a palm computer, a desktop computer or a personal digital assistant (PDA).

[0062] The commodity identification device can also be divided into multiple devices to form a device cluster and collectively execute the commodity identification method provided by the present application, such as Figure 1 A schematic diagram of a scene of the commodity identification method of the present application is shown, wherein the on-site deployment of the camera device of the unmanned supermarket, such as the camera 101, the PDA 102 and the tablet computer 103, can collect the images of the commodities on the commodity shelves, and upload the to-be-identified images to the server 105 through the network 104 for commodity identification, and further, the server 105 can also continue to analyze the commodity conditions and supply strategies of the supermarket shelves according to the commodity identification results.

[0063] The camera device can be a camera, or a device including a camera, such as the camera 101, the PDA 102 and the tablet computer 103 mentioned above; the network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables; the server 105 can be a server cluster composed of multiple servers, and the server 105 can also be a cloud server. According to actual needs, there can be any number of camera devices, networks 104 and servers 105.

[0064] The commodity identification method can be applied not only to the unmanned supermarket scenario, but also to the ordinary supermarket scenario, for example, the staff can collect the images of the commodities on the commodity shelves in the supermarket on site by using the PDA, the smartphone and other devices with cameras, and upload the to-be-identified images to the server or transmit them to the physical host in the supermarket for commodity identification. Of course, the device can also collect the images of the commodities on the commodity shelves on site and perform commodity identification.

[0065] In the existing related art, due to the low recognition accuracy of the recognition model used in the commodity identification process, the commodities in the to-be-identified images cannot be accurately identified, especially when the commodity display is complex, the staff still need to participate to meet the accuracy requirements of commodity identification.

[0066] Based on the above-mentioned defects of the existing related art, the present application provides a commodity identification method with high commodity identification accuracy, which at least partially overcomes the defects of the existing related art.

[0067] Referring to Figure 2 a flowchart of a commodity identification method of the present application is shown, and Figure 3 a structural diagram of a commodity identification model of the present application is shown, the commodity identification method provided by the present application can specifically include the following steps:

[0068] In step S201, an image to be identified is acquired, wherein the image to be identified is used to identify the commodity contained in the image;

[0069] In step S202, the image to be identified is input into the commodity identification model for commodity identification processing, and commodity feature information output by the commodity identification model is obtained, wherein the commodity identification model sequentially includes a first convolutional layer, a second convolutional layer and a fully connected layer (FC), a transmission channel is configured between the first convolutional layer and the fully connected layer, the transmission channel is used to output the feature vector output by the first convolutional layer to the fully connected layer, so that the fully connected layer performs optimization processing according to the feature vector output by the first convolutional layer;

[0070] In step S203, the commodity contained in the image to be identified is determined according to the commodity feature information.

[0071] From the above Figure 2 As can be seen from the embodiments shown, for commodity identification, a transmission channel is constructed between non-adjacent fully connected layers and convolutional layers in the commodity identification model. The feature vector output by the convolutional layer at the bottom of the model when normally performing feature extraction can not only be output to the convolutional layer at the next layer, but also be output to the non-adjacent fully connected layer at the higher layer through the transmission channel. Since the convolutional layer at the bottom pays more attention to the basic information in the image, such as point, line and angle elements, the fully connected layer at the higher layer can have higher attention and pertinence to the outline or appearance of the commodity. Therefore, the feature processing of the fully connected layer has smaller granularity, and the model can output commodity feature information with higher accuracy to a certain extent according to the input image to be identified, and the commodity identification result with higher accuracy can be obtained according to the commodity feature information.

[0072] In addition, since a transmission channel is constructed between non-adjacent fully connected layers and convolutional layers in the commodity identification model, the model has higher attention and pertinence to the outline or appearance of the commodity. Therefore, compared with ordinary models, better identification accuracy and identification effect can be obtained in the training process of the model, so that the model can achieve better learning effect and convergence effect, and the purpose of improving learning efficiency is achieved.

[0073] The following continues to describe Figure 2The various steps in the illustrated embodiments and their possible implementation in practical applications are described in detail:

[0074] In this application, obtaining the image to be identified can be understood not only as a collection process of collecting images through a camera, for example, a commodity identification device can collect the image to be identified on site through its own camera or an external camera; It can also be understood as a retrieval process of retrieving images from related devices that store images, for example, a commodity identification device can retrieve the image to be identified collected by a specific supermarket from a server specially storing the image to be identified; Or it can also be understood as a receiving process of receiving images sent by related devices that store images, for example, a commodity identification device can receive in real time the image to be identified collected on site uploaded by a local PDA, and the specific acquisition method is not limited herein.

[0075] After obtaining the image to be identified, it can be input into the commodity identification model of the present application. It is easy to understand that the commodity identification model includes multiple layers, and after the image to be identified is input into the commodity identification model, it can be converted into a feature vector that can be identified by the model through the input layer in the commodity identification model. The feature vector is sequentially extracted by the multiple convolution layers. The bottom layer of the commodity feature (the commodity feature referred to in the present application can be represented by Feature Map), and the bottom layer of the commodity feature extracted by the last convolution layer is output to the adjacent fully connected layer, and is converted into a high layer of commodity feature by the fully connected layer. Different from the commodity identification model involved in the prior art, the bottom layer of the commodity feature obtained by the previous convolution layer in the model of the present application can also be output to the fully connected layer through the transmission channel constructed by the present application between the fully connected layer and the non-adjacent convolution layer.

[0076] At this time, the feature vector extracted by the fully connected layer can be dot multiplied with the feature vector transmitted by the non-adjacent convolution layer through the transmission channel as a correlation indicating weight coefficient. The greater the correlation, the greater the value, and the smaller the correlation, the smaller the value, so that the weight coefficient can be continuously dot multiplied with the feature vector extracted by the fully connected layer itself. The feature region with large weight is retained, and the interference region with no feature or small weight is filtered, so as to achieve the purpose of using the bottom layer of semantic information to guide the high layer to pay attention to the place with high value features, that is, to achieve the purpose of optimization processing.

[0077] That is, the optimization processing can be understood as:

[0078] Converting the feature vector output by the first convolution layer into a first feature vector with a dimension corresponding to the dimension of the fully connected layer;

[0079] Multiplying the first feature vector and the second feature vector corresponding to the fully connected layer to obtain the product as a weight coefficient;

[0080] The weight coefficient is multiplied by the second feature vector corresponding to the full connection layer to obtain a third feature vector as an output of the full connection layer.

[0081] In the actual application, the feature vector output by the convolution layer can be first converted in dimension to a feature vector with the same dimension as the full connection layer, and then the dot product is performed.

[0082] For example, if the dimension of the full connection layer is 1000 (the size of the Feature Map is h*w, h*w=1000), the feature vector output by the convolution layer is converted to a 1000-dimensional feature vector; if the dimension of the full connection layer is 500 (the size of the Feature Map is h*w, h*w=500), the feature vector output by the convolution layer is converted to a 500-dimensional feature vector. The feature vector output by the convolution layer can be understood as a feature matrix, such as a 2000-dimensional feature vector, which is converted to a 1000-dimensional feature vector. After conversion, the size of the feature vector output by the convolution layer is 2000*1000.

[0083] It can be understood that the first convolution layer not adjacent to the full connection layer in the present application can include any number of sub-convolution layers according to actual needs in the actual application, so that a transmission channel can be constructed between each sub-convolution layer and the non-adjacent full connection layer, such as Figure 4 Another structure diagram of the commodity identification model of the present application is shown, each sub-convolution layer can output to the next layer of sub-convolution layer or the second convolution layer after the first convolution layer when extracting the normal feature vector, and the extracted feature vector can also be output to the non-adjacent full connection layer behind through the transmission channel, and the full connection layer can optimize the feature vectors output by the plurality of sub-convolution layers.

[0084] Further, referring to Figure 5 Another structure diagram of the commodity identification model of the present application is shown, in the actual application, the full connection layer in the commodity identification model can also be divided into a first full connection layer and a second full connection layer according to the connection relationship with the front convolution layer, the first full connection layer is used to connect the second convolution layer, that is, the normal model architecture, and the second full connection layer is used to connect the first convolution layer through the transmission channel in the present application and receive the feature vector output by the first convolution layer for optimization processing.

[0085] In this case, according to the characteristics of the commodity hierarchical classification, the application forms a multi-level classification network structure. At the first convolutional layer, a second fully connected layer is configured, followed by a Softmax layer, and the output class number is the class number of the coarse-grained layer of commodity classification. For example, if the class number of the coarse-grained layer of the model is 3, then the output of the model here is 3. On the other hand, at the second convolutional layer, a first fully connected layer is configured, and finally connected to a Softmax layer, and the output class number is the class number of the fine-grained layer of commodity classification. For example, if the class number of the fine-grained layer of the model is 5, then the output of the model here is 5. Under this classification network structure, the output commodity feature information can have both coarse-grained and fine-grained recognition accuracy.

[0086] In the application, the optimization process can also be further optimized. Specifically, a correction factor can be configured for the second fully connected layer. The correction factor is used to multiply the feature vector output after the optimization process of the second fully connected layer to achieve the purpose of correction. The correction factor a can be understood as an empirical coefficient and can be obtained through historical experience or manually configured to achieve the purpose of flexibly adjusting the optimization effect of the optimization process. At this time, the fully connected layer can use the product of the feature vector output by the second fully connected layer and the correction factor and the feature vector output by the first fully connected layer as the output. The output feature vector can be more robust and distinguishable to form a multi-layer cross-learning mechanism based on attention.

[0087] The training process of the commodity recognition model in the application will be introduced below.

[0088] First, a large number of commodity images can be prepared, and the corresponding commodity feature information of the commodities contained in the images is labeled as a training set for training the model. For example, the commodity feature corresponding to the commodity image of each minimum inventory unit can be configured according to the Stock Keeping Unit (SKU) in the supermarket, or the image corresponding to the commodity feature can be configured according to the commodity that can be stored and planned to be stored in the supermarket, and the training set is formed after labeling the commodity feature information.

[0089] Specifically, these commodity images can be collected by a camera or a device including a camera at the supermarket site, and uploaded to the server. The server will extract the commodity features according to the collected commodity images, and combine the commodity semantic information given by the staff to form images of different commodity feature information; or the commodity recognition device or commodity recognition equipment executing the commodity recognition method of the application can collect commodity images at the supermarket site, and combine the commodity semantic information given by the staff to form images of different commodity feature information.

[0090] After obtaining the training set, the model can be trained in combination with the loss function, that is, the images in the training set are input into the initial model, forward propagation is performed, the loss function is calculated according to the product feature recognition result output by the model, and the loss function is used for backward propagation to adjust the parameters of the model. When the training requirements such as the number of training times, training time, recognition accuracy, etc. are met, the training of the model is completed, and the model can be used as the product recognition model in the present application, which can be used to identify the product in the input image and output the corresponding product feature information.

[0091] Among them, for example, the loss function can include at least one of a cross-entropy loss function, a Triplet loss function and a center loss function.

[0092] Cross-entropy loss function:

[0093]

[0094] Where y and i are two sample labels, p i represents the probability distribution of the true label, q i is the probability distribution of the predicted label of the trained model, L ID is used to measure the similarity of p i and q i , or in other words, the difficulty of expressing the probability distribution p i through the probability distribution q i , the smaller the L ID , the smaller the difference between the two distributions.

[0095] Triplet loss function:

[0096] L Triplet = [d p -d n + α] + ,

[0097] Where d p and d n are the feature distances of the positive sample pair and the negative sample pair, and α is the margin setting for selecting difficult sample pairs, generally set to 0.3. The tripletloss loss function can make the distance between the same objects smaller and the distance between different objects farther.

[0098] Center loss function:

[0099]

[0100] Where yi is the label of the jth image, c yi is the clustering center of the yi label, and f tjrepresents the features before the full connection layer, and β represents the size of the mini-batch of the training data segmentation, i.e., the size of the mini-batch. L C The centerloss loss function is used to represent the sum of squares of distances between the features of each sample in each portion of training data and the center of the features, and minimizing the centerloss loss function can increase the distance between classes and reduce the distance within the class.

[0101] wherein, since the tripletloss loss function only considers the difference between d p and d n , the absolute distance between the positive sample pair and the negative sample pair is ignored, for example, d p = 0.3 and d n = 0.5, the loss value is 0.3-0.5+0.3 = 0.1, and the loss value of another pair d p = 1.3 and d n = 1.5 is also 0.1, although the loss values are the same, the latter pair obviously fails to cluster the positive sample pair, and the former fails to separate the negative sample pair, so the centerloss loss function can be combined to form a set of loss functions to compensate for the defects of the tripletloss loss function.

[0102] If the cross-entropy loss function, the triplet loss function and the center loss function are combined as the final loss function, the final loss function can be:

[0103] L f = L ID + L Triplet + βL C ,

[0104] wherein β can be 0.0005.

[0105] Further, in the present application, the commodity recognition model can be specifically obtained by training commodity images corresponding to different commodity feature information on the basis of an initial neural network model of a residual network model structure type.

[0106] That is to say, the initial model before the commodity recognition model of the present application is trained, and the model structure thereof is a residual network model structure, and in the residual network model structure, a residual block structure can be formed between the same layers or different layers, such as between two convolutional layers, between a convolutional layer and a fully connected layer, etc.

[0107] In combination with the structure of the residual network model shown in Figure 6 , x represents the input of the residual block, F(x) represents the output of the residual block before the second layer activation function, i.e., F(x) = W2σ(W1x), wherein W1 and W2 represent the weights of the first layer and the second layer, σ represents the relu activation function, and the output of the residual block is σ(F(x)+x).

[0108] When there is no shortcut connection (i.e. Figure 6 the right side from x to the channel), the residual block is a normal 2-layer network. The network in the residual block can be a fully connected layer or a convolutional layer. Let the output of the second layer network before the activation function be H(x). If the optimal output in this 2-layer network is the input x, then for the network without shortcut connection, it needs to be optimized to H(x) = x; for the network with shortcut connection, i.e. the residual block, if the optimal output is x, then only F(x) = H(x) - x needs to be optimized to 0, which is simpler than the former. Since the residual network model structure uses a jump connection, it alleviates the gradient vanishing problem caused by increasing the depth of the deep neural network.

[0109] For example, the residual network model structure of the embodiment of the present application can be a ResNet50 residual network model structure, which contains multiple convolutional layers, and the softmax layer connected to the convolutional layers, and the number of outputs of the softmax layer is the number of categories in the training set, for example, if 2000 types of goods are provided in the training set, then the number of categories is set to 2000, and if it is 10000, then it is set to 10000. Since the residual network model structure is introduced, the low-level detail information and high-level feature information can be further fused, so that the goods recognition model is easy to train, and the problem of gradient explosion can also be avoided.

[0110] In some example implementations, in addition to the optimization of the goods recognition model mentioned above, the input image of the model can also be optimized to improve the recognition efficiency and recognition accuracy.

[0111] For example, referring to Figure 7 Fig. 1 shows a flow diagram of a process for obtaining an image to be recognized according to an embodiment of the present application. For the acquisition process of the image to be recognized, it can specifically include:

[0112] Step S701: obtaining an initial image to be recognized;

[0113] It can be understood that the image to be recognized input into the goods recognition model can also be a pre-processed image, and correspondingly, before pre-processing, an initial image to be recognized is obtained, which contains image information of the goods to be recognized.

[0114] Step S702: inputting the initial image into a binary classification model for goods image screening processing to obtain a goods image after removing the background image from the initial image as the image to be recognized.

[0115] In the present application, for the image to be identified, a binary classification processing can be performed. It should be understood that the image captured on site in the supermarket usually includes not only the goods, but also the shelves and other irrelevant background images. In order to eliminate the interference of the background image on the goods identification, a binary classification model can be trained to classify the dense, irregular and different goods images into one category without distinguishing them, and the regular background images, i.e. the shelf images composed of growth strips (including the product label images in the shelf) and the pure color and large area wall images can be classified into another category. Through the binary classification model, the goods image screening processing is performed to extract the goods image after eliminating the background image in the initial image as the image to be identified, which further improves the effective image information in the image and lays a good foundation for the subsequent goods identification, which can greatly reduce the difficulty of goods identification.

[0116] The binary classification model can be obtained by training an initial model based on a training set including different goods images and background images in the supermarket in combination with a loss function. For example, the binary classification model can be a single-stage Yolov3 model or the like, and the training process can refer to the training of the above-mentioned goods identification model, which will not be described herein again.

[0117] For example, after the binary classification model eliminates the background image in the initial image, a corresponding product identification (ID) can be added to each product in the remaining image. When the product image is input into the goods identification model, the goods identification model can extract the product features one by one according to the product ID to identify the product, so as to further facilitate the goods identification and improve the identification efficiency of the goods identification.

[0118] After obtaining the product feature information corresponding to the image to be identified output by the model, the product corresponding to the product feature information can be determined, i.e. the final product identification result can be determined based on the product feature information.

[0119] In the exemplary implementation, referring to Figure 8 FIG. 1 shows a flow diagram of a process for determining a product according to the present application. In the process of determining a product based on the product features output by the product identification model, the following steps can be included:

[0120] Step S801, matching the product feature information with different product feature information in a preset product feature information set;

[0121] It can be understood that the one-to-one product feature information can be extracted or established according to the products in the product library in advance, and the product feature information set can be formed by summarizing the product feature information for recording the product features and subsequent product matching.

[0122] In step S802, the matched target commodity feature information corresponding commodity is determined as the commodity recognition result of the to-be-recognized image.

[0123] After the commodity feature in the current to-be-recognized image is obtained through the commodity recognition model, it can be matched with different commodity feature information in the commodity feature information set, so that the corresponding target commodity feature information can be matched. The commodity corresponding to the target commodity feature information is the commodity recognition result of the commodity in the to-be-recognized image.

[0124] In order to better implement the commodity recognition method provided in the present application, the present application further provides a commodity recognition device.

[0125] Referring to Figure 9 , Figure 9 A structural schematic diagram of the commodity recognition device of the present application, in the present application, the commodity recognition device 900 can specifically include the following structures:

[0126] The acquisition unit 901 is configured to acquire a to-be-recognized image, wherein the to-be-recognized image is used to recognize the commodity contained in the image;

[0127] The recognition unit 902 is configured to input the to-be-recognized image into a commodity recognition model to perform commodity recognition processing, and obtain commodity feature information output by the commodity recognition model, wherein the commodity recognition model sequentially includes a first convolutional layer, a second convolutional layer and a fully connected layer, a transmission channel is configured between the first convolutional layer and the fully connected layer, and the transmission channel is configured to output a feature vector output by the first convolutional layer to the fully connected layer, so that the fully connected layer performs optimization processing according to the feature vector output by the first convolutional layer.

[0128] The determination unit 903 is configured to determine the commodity contained in the to-be-recognized image according to the commodity feature information.

[0129] In an exemplary implementation, the fully connected layer includes a first fully connected layer and a second fully connected layer, the first fully connected layer is configured to connect the second convolutional layer, the second fully connected layer is configured to connect the first convolutional layer through the transmission channel and receive the feature vector output by the first convolutional layer, and the fully connected layer is configured to output the product of the feature vector output by the second fully connected layer and the correction factor and the feature vector output by the first fully connected layer.

[0130] In another exemplary implementation, the optimization processing includes:

[0131] The feature vector output by the first convolutional layer is converted into a first feature vector with a dimension corresponding to the dimension of the fully connected layer;

[0132] The first feature vector is multiplied by a second feature vector corresponding to the fully connected layer, and the obtained product is used as a weight coefficient;

[0133] The weight coefficient is multiplied by the second feature vector corresponding to the full connection layer to obtain a third feature vector as an output of the full connection layer.

[0134] In yet another exemplary implementation, when the first convolutional layer includes a plurality of sub-convolutional layers, a transmission channel is configured between each sub-convolutional layer and the full connection layer, so that the full connection layer performs optimization processing according to the feature vectors respectively output by the plurality of sub-convolutional layers.

[0135] In yet another exemplary implementation, the acquisition unit 901 is specifically configured to:

[0136] acquire an initial image to be recognized;

[0137] input the initial image into the binary classification model to perform commodity image screening processing, and obtain a commodity image after removing a background image from the initial image as a to-be-recognized image.

[0138] In yet another exemplary implementation, the commodity recognition model is obtained by training commodity images corresponding to different commodity feature information based on an initial neural network model of a residual network model structure type.

[0139] In yet another exemplary implementation, the determination unit 903 is specifically configured to:

[0140] match the commodity feature information with different commodity feature information in a preset commodity feature information set;

[0141] determine a commodity corresponding to the matched target commodity feature information as a commodity recognition result of the to-be-recognized image.

[0142] The application also provides a commodity recognition device, which is described in detail in the following. Figure 10 , Figure 10 A structural schematic diagram of the commodity recognition device of the application is shown, and specifically, the commodity recognition device of the application includes a processor 1001, a memory 1002, and an input / output device 1003. The processor 1001 is configured to execute a computer program stored in the memory 1002 to realize the functions of the steps of the commodity recognition method in any embodiment of the application. Figures 1 to 8 corresponding to the steps of the commodity recognition method in any embodiment; or the processor 1001 is configured to execute a computer program stored in the memory 1002 to realize the functions of the units in the embodiment. Figure 9 corresponding to the steps of the commodity recognition method in any embodiment; or the processor 1001 is configured to execute a computer program stored in the memory 1002 to realize the functions of the units in the embodiment. Figures 1 to 8 corresponding to the steps of the commodity recognition method in any embodiment; or the processor 1001 is configured to execute a computer program stored in the memory 1002 to realize the functions of the units in the embodiment.

[0143] For example, the computer program can be divided into one or more modules / units, one or more modules / units are stored in the memory 1002 and executed by the processor 1001 to complete the present application. One or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the computer device.

[0144] The commodity identification device can include, but is not limited to, the processor 1001, the memory 1002, the input and output device 1003. Those skilled in the art can understand that the schematic is only an example of the commodity identification device, and does not constitute a limitation on the commodity identification device, and can include more or less components than the schematic, or combine certain components, or different components, for example, the commodity identification device can also include a network access device, a bus, etc., and the processor 1001, the memory 1002, the input and output device 1003 and the network access device are connected through the bus.

[0145] The processor 1001 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The processor is the control center of the commodity identification device, and connects various parts of the whole device through various interfaces and lines.

[0146] The memory 1002 can be used to store computer programs and / or modules, and the processor 1001 realizes various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 1002, and calling data stored in the memory 1002. The memory 1002 can mainly include a program storage area and a data storage area, wherein the program storage area can store operating systems, application programs required by at least one function, etc.; and the data storage area can store data created according to the use of the commodity identification device, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0147] The processor 1001 is used to execute the computer programs stored in the memory 1002, and can specifically realize the following functions:

[0148] Obtaining a to-be-identified image, wherein the to-be-identified image is used to identify a commodity contained in the image;

[0149] Inputting the to-be-identified image into a commodity identification model to perform commodity identification processing, and obtaining commodity feature information output by the commodity identification model, wherein the commodity identification model sequentially includes a first convolutional layer, a second convolutional layer, and a fully connected layer, a transmission channel is configured between the first convolutional layer and the fully connected layer, and the transmission channel is used to output a feature vector output by the first convolutional layer to the fully connected layer, so that the fully connected layer performs optimization processing according to the feature vector output by the first convolutional layer.

[0150] Determining the commodity contained in the to-be-identified image according to the commodity feature information.

[0151] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the commodity identification device, the device and the corresponding units described above can be referred to as Figures 1 to 8 The description of the commodity identification method corresponding to any embodiment is not repeated here in detail.

[0152] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by related hardware controlled by the instructions, the instructions can be stored in a computer readable storage medium and loaded and executed by a processor.

[0153] Therefore, the present application provides a computer readable storage medium, which stores a plurality of instructions, the instructions can be loaded by a processor to execute the present application Figures 1 to 8Corresponding to the steps in the commodity identification method in any embodiment, the specific operation can be referred to as Figures 1 to 8 Corresponding to the description of the commodity identification method in any embodiment, it will not be repeated here.

[0154] The computer readable storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0155] Due to the instructions stored in the computer readable storage medium, the application can be executed as Figures 1 to 8 Corresponding to the steps in the commodity identification method in any embodiment, the application can be implemented as Figures 1 to 8 Corresponding to the beneficial effects of the commodity identification method in any embodiment, see the previous description, and will not be repeated here.

[0156] The commodity identification method, device, equipment and computer readable storage medium provided by the application are introduced in detail above, and the principle and implementation mode of the application are described by applying specific examples. The above description of the embodiments is only used to help understand the method and its core idea of the application; at the same time, for those skilled in the art, according to the idea of the application, the specific implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as the limitation of the application.

Claims

1. A commodity identification method, characterized in that: The method comprises: Acquire an image to be identified, wherein the image to be identified is used to identify a commodity contained in the image; Inputting the image to be identified into a commodity recognition model for commodity recognition processing, and obtaining commodity feature information output by the commodity recognition model, wherein the commodity recognition model sequentially includes a first convolutional layer, a second convolutional layer, and a fully connected layer, and a transmission channel is constructed between the non-adjacent fully connected layer and the first convolutional layer. The first convolutional layer located at the bottom layer of the model outputs a feature vector to the high-level, non-adjacent fully connected layer through the transmission channel, so that the fully connected layer performs optimization processing based on the feature vector output by the first convolutional layer; wherein the optimization processing includes: converting the feature vector output by the first convolutional layer into a first feature vector whose dimension corresponds to the dimension of the fully connected layer; multiplying the first feature vector by the second feature vector corresponding to the fully connected layer, and using the obtained product as a weight coefficient; and multiplying the weight coefficient by the second feature vector corresponding to the fully connected layer to obtain a third feature vector as the output of the fully connected layer; The commodity contained in the image to be identified is determined according to the commodity feature information.

2. The method according to claim 1, characterized in that The fully connected layer includes a first fully connected layer and a second fully connected layer, the first fully connected layer is used to connect to the second convolutional layer, the second fully connected layer is used to connect to the first convolutional layer through the transmission channel and receive the feature vector output by the first convolutional layer, and the fully connected layer is used to output the product of the feature vector output by the second fully connected layer and the correction factor and the feature vector output by the first fully connected layer.

3. The method according to claim 1, characterized in that When the first convolutional layer includes multiple sub-convolutional layers, the transmission channel is configured between each sub-convolutional layer and the fully connected layer, so that the fully connected layer performs the optimization processing according to the feature vectors respectively output by the multiple sub-convolutional layers.

4. The method according to claim 1, wherein The acquiring of the image to be identified comprises: Obtaining an initial image to be recognized; The initial image is input into a binary classification model to perform product image screening processing, and a product image is obtained after the background image is removed from the initial image, which is used as the image to be identified.

5. The method according to claim 1, wherein The product recognition model is obtained by training product images corresponding to different product feature information based on an initial neural network model of a residual network model structure type.

6. The method according to claim 1, characterized in that The determining, based on the commodity feature information, the commodity contained in the image to be identified includes: Matching the product feature information with different product feature information in a preset product feature information set; The commodity corresponding to the matched target commodity feature information is determined as the commodity recognition result of the image to be recognized.

7. A commodity identification device, characterized in that: The device comprises: an acquisition unit, configured to acquire an image to be identified, wherein the image to be identified is used to identify a commodity contained in the image; an identification unit, configured to input the image to be identified into a commodity identification model for commodity identification processing, and obtain commodity feature information output by the commodity identification model, wherein the commodity identification model sequentially includes a first convolutional layer, a second convolutional layer, and a fully connected layer, a transmission channel being constructed between non-adjacent fully connected layers and the first convolutional layer, and the first convolutional layer at the bottom layer of the model outputs a feature vector to a higher-level, non-adjacent fully connected layer through the transmission channel, so that the fully connected layer performs optimization processing based on the feature vector output by the first convolutional layer; wherein the optimization processing includes: converting the feature vector output by the first convolutional layer into a first feature vector having a dimension corresponding to the dimension of the fully connected layer; multiplying the first feature vector by the second feature vector corresponding to the fully connected layer, and using the obtained product as a weight coefficient; and multiplying the weight coefficient by the second feature vector corresponding to the fully connected layer to obtain a third feature vector as the output of the fully connected layer; A determination unit is configured to determine the commodity contained in the image to be identified based on the commodity feature information.

8. A commodity identification device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the method according to any one of claims 1 to 6 is executed.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the method according to any one of claims 1 to 6.

Images