An intelligent refrigerator and a method for identifying types of objects

Abstract

The present invention discloses an intelligent refrigerator and a method for identifying the types of objects, which avoids the problem of low recognition accuracy caused by directly identifying fast-moving consumer goods due to their rapid update. It includes a cabinet body, a camera unit, and a controller, wherein: the cabinet body is used for storing objects; the camera unit is used for taking the original image when the user accesses or stores objects; the controller is configured to execute: obtaining an image containing an object; using a siamese network to extract features from the image to determine the target feature vector of the object; determining the similarity between the target feature vector and each sample feature vector in the feature set; and determining the object category of the sample image corresponding to the sample feature vector with the maximum similarity as the category of the object.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent refrigerators, and particularly relates to an intelligent refrigerator and a method for identifying object types thereof. Background Art

[0002] With the rapid development of artificial intelligence technology, deep neural network algorithms are increasingly applied to real life. In the field of intelligent refrigerators, the core basis for intelligent development is the automatic identification of food ingredients. Due to the characteristics of a wide variety of fast-moving consumer goods, frequent packaging changes, and emerging new types, when using deep learning models to automatically identify fast-moving consumer goods, the types of fast-moving consumer goods are usually directly identified, making it difficult to ensure the identification accuracy of fast-moving consumer goods, thereby affecting the food ingredient management effect of intelligent refrigerators. Summary of the Invention

[0003] The present invention provides an intelligent refrigerator and a method for identifying object types thereof. By judging the similarity between the image of the object to be judged and the sample image, the object category of the most similar sample image is determined as the category of the object. Since only the similarity is judged, the problem of low identification accuracy caused by directly identifying fast-moving consumer goods due to their rapid updates is avoided.

[0004] In a first aspect, an intelligent refrigerator provided by an embodiment of the present invention includes a cabinet body, a camera unit, and a controller, wherein:

[0005] The cabinet body is used for storing objects;

[0006] The camera unit is used for taking an original image when a user accesses or stores an object;

[0007] The controller is configured to execute:

[0008] Obtain an image containing an object captured by the intelligent refrigerator;

[0009] Use a siamese network to extract features from the image to determine the target feature vector of the object;

[0010] Determine the similarity between the target feature vector and each sample feature vector in the feature set, where the feature set is obtained by using the siamese network to extract features from sample images of each labeled object category, and the feature set contains at least one sample feature vector of the sample images of each labeled object category;

[0011] Determine the object category of the sample image corresponding to the sample feature vector that satisfies the maximum similarity as the category of the object.

[0012] The intelligent refrigerator provided by the embodiment of the present invention uses a siamese network to extract features from the captured image, then determines the similarity between the image and the sample image, so as to select the most similar sample image, and determines the object category marked by the sample image as the object category in the image. Since the siamese network is used to judge the similarity between two input feature vectors, the siamese network only needs to be trained once and does not need to be repeatedly trained according to the update of the object type. Therefore, by using the method of judging similarity to determine the object category, it avoids directly identifying the image and the need to retrain the deep learning model according to the update of the object type, and can improve the recognition accuracy of the object without repeated training.

[0013] In some examples, the siamese network is trained by using the training sample images marked with object categories as input and the similarity between the object categories marked by the training sample images as output to train the initial siamese network.

[0014] In some examples, the controller is specifically configured to execute:

[0015] Detect the human hand in the original image captured by the intelligent refrigerator according to the object detection algorithm to determine the position of the human hand;

[0016] According to the position of the human hand, determine the object position of the object grabbed by the human hand in the original image in the original image;

[0017] Obtain the image containing the object from the object position of the original image.

[0018] In some examples, the controller is specifically configured to pre-obtain the feature set in the following manner:

[0019] Obtain the sample images of each marked object category;

[0020] Use the siamese network to extract features from the sample images to determine the sample feature vectors of each sample image;

[0021] Determine the feature set according to each sample feature vector and the object category marked by the sample image corresponding to each sample feature vector.

[0022] In some examples, the controller is specifically configured to execute:

[0023] Integrate the target feature vector with each sample feature vector in the feature set respectively to obtain each integrated feature vector;

[0024] Classify the integrated feature vector by using the fully-connected layer and the loss layer in the siamese network, and determine the classification label of the integrated feature vector, where the classification label is used to characterize the similarity between the target feature vector and the sample feature vector.

[0025] In some examples, for any sample feature vector, the controller is specifically configured to obtain an integrated feature vector in the following manner:

[0026] Concatenate the target feature vector and the sample feature vector to obtain an integrated feature vector; or,

[0027] Subtract the corresponding bits of the target feature vector and the sample feature vector to obtain an integrated feature vector, where the corresponding bits are used to characterize the positions of the elements in the target feature vector in the target feature vector and the positions of the elements in the sample feature vector in the sample feature vector are the same.

[0028] In some examples, after determining the similarity between the target feature vector and each sample feature vector in the feature set, the controller is further configured to perform:

[0029] If the similarity between the target feature vector and each sample feature vector is less than the threshold, add the target feature vector and the category of the target feature vector to the feature set.

[0030] In a second aspect, a method for identifying the type of an object provided by an embodiment of the present invention includes:

[0031] Obtain an image containing the object;

[0032] Use a siamese network to extract features from the image and determine the target feature vector of the object;

[0033] Determine the similarity between the target feature vector and each sample feature vector in the feature set, where the feature set is obtained by using the siamese network to extract features from sample images of each labeled object category, and the feature set contains at least one sample feature vector of the sample images of each labeled object category;

[0034] Determine the object category of the sample image corresponding to the sample feature vector with the maximum similarity as the object category of the object.

[0035] In some examples, the siamese network is trained by using training sample images labeled with object categories as input and the similarity between the object categories labeled in the training sample images as output to train the initial siamese network.

[0036] In some examples, the obtaining of the image including the object includes:

[0037] Detect the human hand in the original image captured by the smart refrigerator according to the object detection algorithm, and determine the position of the human hand;

[0038] According to the position of the human hand, determine the object position of the object grabbed by the human hand in the original image in the original image;

[0039] Obtain the image including the object from the object position of the original image.

[0040] In some examples, the feature set is pre-obtained in the following manner:

[0041] Obtain the sample images of each labeled object category;

[0042] Use the siamese network to extract features from the sample images, and determine the sample feature vector of each sample image;

[0043] Determine the feature set according to each sample feature vector and the object category labeled by the sample image corresponding to each sample feature vector.

[0044] In some examples, the determination of the similarity between the target feature vector and each sample feature vector in the feature set includes:

[0045] Integrate the target feature vector and each sample feature vector in the feature set respectively to obtain each integrated feature vector;

[0046] Use the fully connected layer and loss layer in the siamese network to classify the integrated feature vector, and determine the classification label of the integrated feature vector, where the classification label is used to characterize the similarity between the target feature vector and the sample feature vector.

[0047] In some examples, for any one sample feature vector, an integrated feature vector is obtained in the following manner:

[0048] Concatenate the target feature vector and the sample feature vector to obtain an integrated feature vector; or,

[0049] Subtract the corresponding bits of the target feature vector and the sample feature vector to obtain an integrated feature vector, where the corresponding bits are used to characterize the position of the element in the target feature vector in the target feature vector and the position of the element in the sample feature vector in the sample feature vector is the same.

[0050] In some examples, after determining the similarity between the target feature vector and each sample feature vector in the feature set, the method further includes:

[0051] If the similarity between the target feature vector and each sample feature vector is less than the threshold, add the target feature vector and the category of the target feature vector to the feature set.

[0052] In a third aspect, an embodiment of the present invention further provides a device for identifying the type of an object. The device includes:

[0053] An image acquisition unit, configured to acquire an image containing an object;

[0054] A feature extraction unit, configured to extract features from the image by using a siamese network, and determine a target feature vector of the object;

[0055] A similarity decision unit, configured to determine the similarity between the target feature vector and each sample feature vector in the feature set, where the feature set is obtained by extracting features from sample images of each labeled object category by using the siamese network, and the feature set includes at least one sample feature vector of the sample images of each labeled object category;

[0056] A category determination unit, configured to determine the object category of the sample image corresponding to the sample feature vector with the maximum similarity as the category of the object.

[0057] In some examples, the siamese network is trained by using training sample images labeled with object categories as inputs and the similarity between the object categories labeled in the training sample images as outputs to train an initial siamese network.

[0058] In some examples, the image acquisition unit is specifically configured to:

[0059] Detect the human hand in the original image captured by the smart refrigerator according to an object detection algorithm, and determine the position of the human hand;

[0060] Determine the object position of the object grabbed by the human hand in the original image according to the position of the human hand;

[0061] Acquire an image containing the object from the object position of the original image.

[0062] In some examples, the similarity decision unit is specifically configured to pre-obtain the feature set in the following manner:

[0063] Acquire sample images of each labeled object category;

[0064] Use the siamese network to extract features from the sample images and determine the sample feature vectors of each sample image;

[0065] Determine the feature set according to each sample feature vector and the object category labeled for each sample feature vector.

[0066] In some examples, the similarity decision unit is specifically configured to:

[0067] Integrate the target feature vector with each sample feature vector in the feature set respectively to obtain respective integrated feature vectors;

[0068] Use the fully connected layer and the loss layer in the siamese network to classify the integrated feature vectors, and determine the classification labels of the integrated feature vectors, where the classification labels are used to characterize the similarity between the target feature vector and the sample feature vectors.

[0069] In some examples, the similarity decision unit is specifically configured to obtain an integrated feature vector for any one sample feature vector in the following manner:

[0070] Concatenate the target feature vector and the sample feature vector to obtain an integrated feature vector; or,

[0071] Subtract the corresponding positions of the target feature vector and the sample feature vector to obtain an integrated feature vector, where the corresponding positions are used to represent the positions of the elements in the target feature vector in the target feature vector and the positions of the elements in the sample feature vector in the sample feature vector are the same.

[0072] In some examples, after determining the similarity between the target feature vector and each sample feature vector in the feature set, the similarity decision unit is further specifically configured to:

[0073] If the similarity between the target feature vector and each sample feature vector is less than the threshold, add the target feature vector and the category of the target feature vector to the feature set.

[0074] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, on which a computer program is stored, and when the program is executed by a processor, it is used to implement the steps of the method described in the first aspect above.

[0075] These aspects or other aspects of the present application will be more clearly understood in the following description of the embodiments. Description of the Drawings

[0076] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0077] Figure 1 Schematic diagram of an intelligent refrigerator provided by an embodiment of the present invention;

[0078] Figure 2 Schematic diagram of an image containing an object provided by an embodiment of the present invention;

[0079] Figure 3 Schematic diagram of the structure of a twin network provided by an embodiment of the present invention;

[0080] Figure 4 Schematic diagram of the comparison between the first image and the sample image provided by an embodiment of the present invention;

[0081] Figure 5 Schematic diagram of the comparison between the second image and the sample image provided by an embodiment of the present invention;

[0082] Figure 6 Schematic diagram of the comparison between the third image and the sample image provided by an embodiment of the present invention;

[0083] Figure 7 Flowchart of the implementation for identifying the type of an object provided by an embodiment of the present invention;

[0084] Figure 8 Flowchart of the method implementation for identifying the type of an object provided by an embodiment of the present invention;

[0085] Figure 9 Schematic diagram of a device for identifying the type of an object provided by an embodiment of the present invention. Detailed implementation manners

[0086] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will further describe the present invention in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0087] In the embodiments of the present invention, the term "and / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0088] The application scenarios described in the embodiments of the present invention are for more clearly illustrating the technical solutions of the embodiments of the present invention, and do not constitute a limitation to the technical solutions provided by the embodiments of the present invention. Those of ordinary skill in the art will know that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems. Among them, in the description of the present invention, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0089] In the field of smart refrigerators, when using a deep learning model for automatic identification of food ingredients, the types of food ingredients are directly identified. Due to the characteristics of a large variety of food ingredients, frequent changes in packaging, and the continuous emergence of new types, when the deep learning model conducts classification training on new data, the accuracy of the classification results for old data usually decreases. Therefore, in order to ensure the recognition accuracy, the parameters of the deep learning model are usually retrained based on all known data categories to adapt to the changes in the data distribution. Due to the characteristics of a large variety of food ingredients such as fast-moving consumer goods, frequent changes in packaging, and the continuous emergence of new types, a large amount of data needs to be continuously collected and labeled for retraining the model to ensure the classification accuracy of fast-moving consumer goods. Therefore, there is a relatively large consumption in terms of time resources for data collection, labeling, and model training. And in actual applications, it is difficult to obtain a large number of training samples of new categories in a short period of time to retrain the deep learning model, making it difficult to ensure the classification accuracy of fast-moving consumer goods, thereby affecting the food ingredient management effect of the smart refrigerator.

[0090] The embodiments of the present invention provide a brand-new way of identifying types. Instead of directly identifying the image, the similarity between images is judged through a siamese network, and the object category labeled by the sample image with the maximum similarity is used as the category of the object in the image. Since the siamese network is used to judge the similarity between two input feature vectors, the siamese network only needs to be trained once and does not need to be repeatedly trained according to the update of the object type. Therefore, using the method of judging similarity to determine the category of the object avoids directly identifying the image and the need to retrain the deep learning model according to the update of the object type, and can improve the recognition accuracy of the object without repeated training.

[0091] The embodiments of the present invention provide a smart refrigerator that can capture an image of an object during the process of a user accessing the object, and use a siamese network to judge the similarity between the image of the object and the sample image, so as to use the object category labeled by the sample image with the maximum similarity as the category of the object in the image. As Figure 1 shown, the smart refrigerator includes a cabinet body 100, a camera unit 101, and a controller 102, where:

[0092] The cabinet 100 is used to store objects; generally, the objects in this embodiment include, but are not limited to, fast-moving consumer goods, food, ingredients, etc. that need refrigeration.

[0093] The imaging unit 101 is used to capture the original images when the user accesses the objects; in practice, the imaging unit can be set at the top of the door frame of the refrigerating chamber of the smart refrigerator, and / or at the top of the door frame of the freezing chamber of the smart refrigerator. The number of cameras included in the imaging unit in this embodiment is at least one, and the opening and closing operations of the smart refrigerator door body can trigger the opening and closing of the imaging unit. Optionally, when the smart refrigerator door body is opened, the camera in the imaging unit pops out and starts shooting. In this embodiment, the number of original images captured by the imaging unit for the same object is at least one, and the schematic diagram of the images of the object captured is as Figure 2 shown, where area A is the image of the interior of the refrigerator captured, area B is the image containing the object captured, and area C is the image containing information such as the human body and the background captured. In some examples, the objects in this embodiment include, but are not limited to, objects such as fast-moving consumer goods that can be stored in the smart refrigerator.

[0094] The controller 102 is configured to perform the following steps:

[0095] Step 1: Obtain the image containing the object captured by the smart refrigerator; wherein, the image containing the object is obtained after processing the captured original image. Since the original image is captured by the camera and contains information such as the refrigerator, the human body, the background, and the object, in order to accurately locate the position of the object in the image, the original image needs to be processed so as to remove the remaining information irrelevant to the object in the subsequent feature extraction process of the image and extract the feature information about the object. Such as obtaining Figure 2 the image of area B in.

[0096] As an optional implementation manner, the specific processing implementation process is as follows:

[0097] 1a. Detect the human hand in the original image captured by the smart refrigerator according to the object detection algorithm to determine the position of the human hand;

[0098] 1b. Determine the object position of the object grabbed by the human hand in the original image according to the position of the human hand;

[0099] 1c. Obtain the image containing the object from the object position of the original image.

[0100] After obtaining the image containing the object, the size and grayscale of the image can also be normalized to facilitate the feature extraction operation.

[0101] Based on the usage scenario of the smart refrigerator, the object position of the object grasped by the human hand can be finally determined by detecting the hand position, so as to crop or cut out the image containing the object at the object position from the original image, so that when performing feature extraction subsequently, the features of the grasped object can be extracted more accurately and effectively, thus ensuring the accuracy of recognition and improving the recognition precision. An alternative implementation is to use a trained Single Shot MultiBox Detector (SSD) model, YOLO model, etc. to identify the position information of the object grasped by the hand in the image, and finally determine the image of the object at that position.

[0102] Step 2: Use a Siamese network to extract features from the image and determine the target feature vector of the object.

[0103] In some examples, the structure of the Siamese network in this embodiment is as Figure 3 shown, including two feature extraction networks 300, 301 with the same structure and parameters, a feature integration network 302, and a similarity decision network 303. The feature extraction network may include several convolutional layers, batch normalization layers, pooling layers, etc. The network receives two images of a sample pair as inputs. The input images need to undergo normalization operations for size and grayscale, and then enter the two feature extraction networks respectively to generate two feature vectors. Among them, batch normalization belongs to a type of normalization. The batch normalization layer is used when training the Siamese network. Each time, several training sample images are taken for training to obtain the mean and variance of the training data for each training batch, and then the training data of this batch is normalized. Using the batch normalization layer can make the learning speed of the Siamese network faster and achieve better results in classification tasks. The feature integration network is used to integrate the two feature vectors output by the two feature extraction networks into one feature vector. The similarity decision network includes multiple fully connected layers and a soft max Loss layer. Using the feature vector output by the feature integration network, after several fully connected operations, it is mapped into a two-dimensional feature vector. The two dimensions of this feature vector respectively represent the confidence that the two input training sample images belong to the same object category, and the confidence that the two input training sample images belong to different object categories.

[0104] In some examples, the Siamese network is trained using training sample images labeled with object categories, with the similarity between the object categories labeled in the training sample images as the output. Among them, the two training sample images are respectively input into the two feature extraction networks, and the weights of the two feature extraction networks are shared. These two training sample images can be training sample images of the same object category or training sample images of different object categories.

[0105] In some examples, the training sample images include positive sample pairs and negative sample pairs, and the process of obtaining the training sample images is as follows:

[0106] (1) Select images of X different object categories, obtain Y pictures of different exposure surfaces of the original images of each object category, extract the images containing the objects and label the object categories, and determine that the number of training sample images is Y*X, where X is a positive integer greater than or equal to 100, and Y is a positive integer greater than or equal to 6.

[0107] (2) Screen out positive sample pairs and negative sample pairs from the Y*X training sample images. Among them, a positive sample pair is a pair of samples composed of two randomly selected images of the same object category from the Y*X training sample images, and a negative sample pair is a pair of samples composed of two randomly selected images of different object categories from the Y*X training sample images.

[0108] The training sample images contain X object categories, and each object category contains Y pictures, that is, the number of positive sample pairs is The negative sample pair is a group of samples composed of two fast-moving consumer goods pictures of different categories, and the category label of the negative sample pair is set to 0. To ensure the balance of the number of positive and negative samples, the randomly selected number of negative sample pairs is also

[0109] (3) Set the object category label of the positive sample pair to 1, and the object category label of the negative sample pair to 0; among them, the label 1 is used to represent that the two input training sample images belong to the same object category, and the label 0 is used to represent that the two input training sample images belong to different object categories.

[0110] Using the method of judging similarity by the siamese network in this embodiment, during the training of the siamese network to obtain training samples, a large amount of acquisition and annotation work can be saved. For example, the current recognition method requires Y*X training sample images, but this embodiment can use the labeled Y*X training sample images to generate Training sample images far greater than Y*X, thus saving a large amount of acquisition and annotation work.

[0111] Use the above positive sample pairs and negative sample pairs to train the siamese network. After the training is completed, randomly select several training sample images from the training samples of each object category and input them into the trained siamese network. Generally, only input them into one feature extraction network of the siamese network to obtain the feature vectors of each object category, and label the corresponding object category for the feature vectors, so as to determine the feature set.

[0112] In some examples, the controller is specifically configured to pre-obtain the feature set in the following manner:

[0113] 2a. Obtain sample images of each labeled object category;

[0114] 2b. Use the siamese network to extract features from the sample images and determine the sample feature vector of each sample image;

[0115] 2c. Determine the feature set according to each sample feature vector and the object category labeled for the sample image corresponding to each sample feature vector.

[0116] Step 3. Determine the similarity between the target feature vector and each sample feature vector in the feature set, where the feature set is obtained by using the siamese network to extract features from sample images of each labeled object category, and the feature set contains at least one sample feature vector of the sample images of each labeled object category.

[0117] In some examples, after obtaining the target feature vector, it is necessary to integrate the target feature vector and each sample feature vector in the feature set respectively, and input the integrated feature vectors into the fully connected layer and loss layer in the siamese network for classification. The controller is specifically configured to execute:

[0118] 3a. Integrate the target feature vector and each sample feature vector in the feature set respectively to obtain respective integrated feature vectors;

[0119] In implementation, the integration method in this embodiment can be carried out through the integration network of the siamese network, or the feature vectors can be integrated and then provided to the siamese network. This embodiment does not make too many limitations on this.

[0120] In some examples, for any one sample feature vector, the controller is specifically configured to obtain an integrated feature vector by any one of the following methods:

[0121] Method 1. Concatenate the target feature vector and the sample feature vector to obtain an integrated feature vector;

[0122] Method 2. Subtract the corresponding bits of the target feature vector and the sample feature vector to obtain an integrated feature vector, where the corresponding bits are used to represent the positions of the elements in the target feature vector in the target feature vector and the positions of the elements in the sample feature vector in the sample feature vector are the same.

[0123] 3b. Use the fully connected layer and loss layer in the siamese network to classify the integrated feature vector and determine the classification label of the integrated feature vector, where the classification label is used to represent the similarity between the target feature vector and the sample feature vector.

[0124] Step 4: Determine the object category of the sample image corresponding to the sample feature vector with the maximum similarity as the category of the object. As Figure 4 , Figure 5 , Figure 6 shown, when the image containing the object is the same as the sample image, the obtained similarity is 1; when the object types of the image containing the object and the sample image are the same, the obtained similarity is 0.66 which is greater than the threshold; when the object types of the image containing the object and the sample image are different, the obtained similarity is 0.28 which is less than the threshold.

[0125] In implementation, if there are multiple sample feature vectors similar to the target feature vector (i.e., the similarities are all greater than the threshold), then select the sample feature vector with the maximum similarity, and determine the object category of the sample image corresponding to this sample feature vector as the category of the object; in some examples, if the similarity between the target feature vector and each sample feature vector is less than or equal to the threshold (i.e., they are all not similar), then add the target feature vector and the category of the target feature vector to the feature set, that is, after performing object category annotation on the target feature vector of the new category, add the annotated target feature vector to the feature set, so that when performing object type recognition again later, it can be compared with the feature vector of the new image, and there is no need to repeat the training of the siamese network using the new image.

[0126] As Figure 7 shown, an implementation process for identifying object types provided by an embodiment of the present invention is as follows:

[0127] Step 700: Use the training sample images with annotated object categories as the input, and the similarity between the object categories annotated in the training sample images as the output, and train the initial siamese network to obtain a trained siamese network;

[0128] Step 701: Obtain sample images of each annotated object category, use the siamese network to extract features from the sample images, determine the sample feature vector of each sample image, and determine the feature set according to each sample feature vector and the object category annotated in the sample image corresponding to each sample feature vector;

[0129] Step 702: Detect the human hand in the original image captured by the smart refrigerator according to the object detection algorithm, determine the position of the human hand, determine the object position of the object grabbed by the human hand in the original image based on the position of the human hand, and obtain the image containing the object from the object position in the original image;

[0130] Step 703: Use the siamese network to extract features from the image, and determine the target feature vector of the object;

[0131] Step 704: Integrate the target feature vector with each sample feature vector in the feature set to obtain respective integrated feature vectors;

[0132] Step 705: Classify the integrated feature vectors by using the fully connected layer and the loss layer of the siamese network to determine the classification labels of the integrated feature vectors;

[0133] Step 706: If the similarity between the target feature vector and each of the sample feature vectors is less than the threshold, add the target feature vector and the category of the target feature vector to the feature set.

[0134] In some examples, based on the same inventive concept, an embodiment of the present invention further provides a method for identifying the type of an object. Since the principle of solving problems by this method is similar to that of the above intelligent refrigerator, the implementation of this method can refer to the implementation of the intelligent refrigerator, and the repeated parts will not be elaborated.

[0135] As Figure 8 shown, a method for identifying the type of an object provided by an embodiment of the present invention has the following implementation process:

[0136] Step 800: Obtain an image containing the object;

[0137] Step 801: Use the siamese network to extract features from the image to determine the target feature vector of the object;

[0138] Step 802: Determine the similarity between the target feature vector and each sample feature vector in the feature set, where the feature set is obtained by using the siamese network to extract features from sample images of various labeled object categories, and the feature set contains at least one sample feature vector of the sample images of various labeled object categories;

[0139] Step 803: Determine the object category of the sample image corresponding to the sample feature vector with the maximum similarity as the category of the object.

[0140] In some examples, the siamese network is trained by using training sample images labeled with object categories as input and the similarity between the object categories labeled in the training sample images as output for the initial siamese network.

[0141] In some examples, the obtaining of the image containing the object includes:

[0142] Detect the human hand in the original image captured by the intelligent refrigerator according to the object detection algorithm to determine the position of the human hand;

[0143] Determine the object position of the object grasped by the human hand in the original image according to the position of the human hand;

[0144] Obtain an image containing the object from the object position of the original image.

[0145] In some examples, the feature set is pre-obtained in the following manner:

[0146] Obtain sample images of each labeled object category;

[0147] Use the siamese network to extract features from the sample images and determine the sample feature vectors of each sample image;

[0148] Determine the feature set according to each sample feature vector and the object category labeled for the sample image corresponding to each sample feature vector.

[0149] In some examples, determining the similarity between the target feature vector and each sample feature vector in the feature set includes:

[0150] Integrate the target feature vector with each sample feature vector in the feature set respectively to obtain respective integrated feature vectors;

[0151] Use the fully connected layer and loss layer in the siamese network to classify the integrated feature vectors and determine the classification labels of the integrated feature vectors, where the classification labels are used to characterize the similarity between the target feature vector and the sample feature vector.

[0152] In some examples, for any one sample feature vector, an integrated feature vector is obtained in the following manner:

[0153] Concatenate the target feature vector with the sample feature vector to obtain an integrated feature vector; or,

[0154] Subtract the corresponding bits of the target feature vector from the sample feature vector to obtain an integrated feature vector, where the corresponding bits are used to represent the positions of the elements in the target feature vector in the target feature vector and the positions of the elements in the sample feature vector in the sample feature vector are the same.

[0155] In some examples, after determining the similarity between the target feature vector and each sample feature vector in the feature set, it further includes: [[ID=,]]

[0156] If the similarity between the target feature vector and each sample feature vector is less than the threshold, add the target feature vector and the category of the target feature vector to the feature set.

[0157] In some examples, based on the same inventive concept, embodiments of the present invention further provide a device for identifying the type of an object. Since this device is the same as the device in the method of the embodiments of the present invention, and the principle of this device for solving problems is similar to that of the method, the implementation of this device can refer to the implementation of the smart refrigerator, and the repeated parts will not be elaborated.

[0158] As Figure 9 shown, embodiments of the present invention further provide a device for identifying the type of an object, and the device includes:

[0159] An image acquisition unit 900, configured to acquire an image including an object;

[0160] A feature extraction unit 901, configured to extract features from the image by using a siamese network to determine a target feature vector of the object;

[0161] A similarity decision unit 902, configured to determine the similarity between the target feature vector and each sample feature vector in a feature set, where the feature set is obtained by extracting features from sample images of each labeled object category by using the siamese network, and the feature set includes at least one sample feature vector of the sample images of each labeled object category;

[0162] A category determination unit 903, configured to determine the object category of the sample image corresponding to the sample feature vector that satisfies the maximum similarity as the category of the object.

[0163] In some examples, the siamese network is obtained by training an initial siamese network by using training sample images labeled with object categories as inputs and the similarity between the object categories labeled in the training sample images as outputs.

[0164] In some examples, the image acquisition unit 900 is specifically configured to:

[0165] Detect the human hand in the original image captured by the smart refrigerator according to an object detection algorithm to determine the position of the human hand;

[0166] Determine the object position of the object grabbed by the human hand in the original image according to the position of the human hand;

[0167] Acquire an image including the object from the object position of the original image.

[0168] In some examples, the similarity decision unit 902 is specifically configured to pre-acquire the feature set in the following manner:

[0169] Acquire sample images of each labeled object category;

[0170] Use the siamese network to extract features from the sample images, and determine the sample feature vectors of each sample image;

[0171] Determine the feature set according to each sample feature vector and the object category labeled by the sample image corresponding to each sample feature vector.

[0172] In some examples, the similarity decision unit 902 is specifically configured to:

[0173] Integrate the target feature vector with each sample feature vector in the feature set respectively to obtain respective integrated feature vectors;

[0174] Use the fully connected layer and the loss layer in the siamese network to classify the integrated feature vectors, and determine the classification labels of the integrated feature vectors, where the classification labels are used to characterize the similarity between the target feature vector and the sample feature vectors.

[0175] In some examples, the similarity decision unit 902 is specifically configured to obtain an integrated feature vector for any one sample feature vector in the following manner:

[0176] Concatenate the target feature vector with the sample feature vector to obtain an integrated feature vector; or,

[0177] Subtract the corresponding bits of the target feature vector from the sample feature vector to obtain an integrated feature vector, where the corresponding bits are used to represent the positions of the elements in the target feature vector in the target feature vector and the positions of the elements in the sample feature vector in the sample feature vector are the same.

[0178] In some examples, after determining the similarity between the target feature vector and each sample feature vector in the feature set, the similarity decision unit 902 is further specifically configured to:

[0179] If the similarity between the target feature vector and each sample feature vector is less than the threshold, add the target feature vector and the category of the target feature vector to the feature set.

[0180] Based on the same inventive concept, an embodiment of the present invention further provides a computer storage medium, on which a computer program is stored, and when the program is executed by a processor, the following steps are implemented:

[0181] Obtain an image containing an object;

[0182] Use the siamese network to extract features from the image, and determine the target feature vector of the object;

[0183] Determine the similarity between the target feature vector and each sample feature vector in the feature set, where the feature set is obtained by extracting features from sample images of each labeled object category using the siamese network, and the feature set contains at least one sample feature vector of the sample images of each labeled object category;

[0184] Determine the object category of the sample image corresponding to the sample feature vector that satisfies the maximum similarity as the category of the object.

[0185] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) that contain computer-usable program code.

[0186] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0187] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0188] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0189] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. An intelligent refrigerator, characterized in that, Comprising a cabinet body, a camera unit, and a controller, wherein: The cabinet body is used for storing objects; The camera unit is used for capturing an original image when a user accesses or stores an object; The controller is configured to execute: Obtain an image containing an object captured by the smart refrigerator; Extract features from the image using a siamese network to determine a target feature vector of the object; Determine the similarity between the target feature vector and each sample feature vector in the feature set, where the feature set is obtained by extracting features from sample images of each labeled object category using the siamese network, and the feature set contains at least one sample feature vector of the sample images of each labeled object category; Determine the object category of the sample image corresponding to the sample feature vector with the maximum similarity as the object category of the object.

2. The intelligent refrigerator according to claim 1, characterized in that, The siamese network is trained with the training sample images labeled with object categories as the input and the similarity between the object categories labeled by the training sample images as the output to obtain the initial siamese network.

3. The intelligent refrigerator according to claim 1, wherein The controller is specifically configured to execute: Detect the human hand in the original image captured by the smart refrigerator according to an object detection algorithm to determine the position of the human hand; Determine the object position of the object grabbed by the human hand in the original image according to the position of the human hand; Obtain an image containing the object from the object position of the original image.

4. The intelligent refrigerator according to claim 1, wherein The controller is specifically configured to pre-obtain the feature set in the following manner: Obtain sample images of each labeled object category; Extract features from the sample images using the siamese network to determine the sample feature vector of each sample image; Determine the feature set according to each sample feature vector and the object category labeled by the sample image corresponding to each sample feature vector.

5. The intelligent refrigerator according to claim 1, wherein The controller is specifically configured to execute: Integrate the target feature vector with each sample feature vector in the feature set respectively to obtain respective integrated feature vectors; Classify the integrated feature vectors using the fully connected layer and the loss layer in the siamese network to determine the classification label of the integrated feature vector, where the classification label is used to characterize the similarity between the target feature vector and the sample feature vector.

6. The intelligent refrigerator according to claim 5, wherein For any one sample feature vector, the controller is specifically configured to obtain an integrated feature vector in the following manner: Concatenate the target feature vector and the sample feature vector to obtain an integrated feature vector; or, Subtract the corresponding bits of the target feature vector and the sample feature vector to obtain an integrated feature vector, where the corresponding bits are used to represent the position of the element in the target feature vector in the target feature vector and the position of the element in the sample feature vector in the sample feature vector is the same.

7. The intelligent refrigerator according to any one of claims 1 to 6, characterized in that, After determining the similarity between the target feature vector and each sample feature vector in the feature set, the controller is further configured to execute: If the similarity between the target feature vector and each sample feature vector is less than the threshold, add the target feature vector and the category of the target feature vector (which is obtained through annotation) to the feature set.

8. A method for identifying the type of an object, characterized in that, The method includes: Obtain an image containing an object; Use a siamese network to extract features from the image and determine the target feature vector of the object; Determine the similarity between the target feature vector and each sample feature vector in the feature set, where the feature set is obtained by using the siamese network to extract features from sample images of each annotated object category, and the feature set contains at least one sample feature vector of the sample images of each annotated object category; Determine the object category of the sample image corresponding to the sample feature vector with the maximum similarity as the category of the object.

9. The method according to claim 8, wherein The siamese network is trained by using training sample images with annotated object categories as input and the similarity between the object categories annotated in the training sample images as output on an initial siamese network.

10. The method according to claim 8, characterized in that, The determination of the similarity between the target feature vector and each sample feature vector in the feature set includes: Integrate the target feature vector with each sample feature vector in the feature set respectively to obtain respective integrated feature vectors; Use the fully connected layer and the loss layer in the siamese network to classify the integrated feature vectors and determine the classification labels of the integrated feature vectors, where the classification labels are used to represent the similarity between the target feature vector and the sample feature vectors.

Images