Item recognition method and device for storage device, and storage device

Abstract

The application discloses a kind of storage equipment article identification method, device and storage equipment, belong to storage equipment technical field.The method comprises: obtaining the color image data and depth image data of the storage space of storage equipment;Based on the color image data and the depth image data, obtain first image data feature;Based on the depth image data, obtain second image data feature;Based on the first image data feature and the second image data feature, obtain the article category information and article position information of the storage equipment.The method is independently processed by depth image data, extracts second image data feature, and the color image data and depth image data are fused, first image data feature is extracted, shallow feature and deep feature are fused, depth information is effectively utilized, article category and position can be accurately identified, position identification does not need to additionally set gravity sensor and the like device, system is simple and relatively low in cost.

Description

Technical Field

[0001] This application belongs to the field of storage equipment technology, and particularly relates to a method, device and storage equipment for identifying items in a storage device. Background Technology

[0002] With the development of smart technology, more and more storage devices are equipped with item recognition functions, which can automatically identify the stored items and provide users with item management suggestions. For example, smart refrigerators have food recognition functions, which can identify the food stored in the refrigerator and provide users with recipes and food purchasing suggestions.

[0003] While visual technology can be used to identify objects, it is mostly used to identify the type of object. The accuracy of identifying the location of an object is relatively low, and location identification usually requires the additional installation of a gravity sensor, making the system complex and costly. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a method, apparatus, and storage device for identifying items in a storage device, which can accurately identify the type and location of items, and the system is simple and relatively low in cost.

[0005] In a first aspect, this application provides a method for identifying items in a storage device, the method comprising:

[0006] Acquire color and depth image data of the storage space of the storage device;

[0007] Based on the color image data and the depth image data, the first image data features are obtained;

[0008] Based on the depth image data, the second image data features are obtained;

[0009] Based on the first image data features and the second image data features, the item type information and item location information of the storage device are obtained.

[0010] According to the item identification method of the storage device of this application, by independently processing the depth image data, extracting the second image data features, and by fusing the color image data and the depth image data, extracting the first image data features, and fusing the shallow features and deep features, the depth information is effectively utilized, which can accurately identify the type and location of the item. The location identification does not require additional gravity sensors or other devices, and the system is simple and relatively low in cost.

[0011] According to one embodiment of this application, before acquiring the color image data and depth image data of the storage space of the storage device, the method further includes:

[0012] Obtain the target depth data of the storage device;

[0013] In response to a step signal in the target depth data, the color image data and the depth image data are acquired.

[0014] According to one embodiment of this application, obtaining the first image data feature based on the color image data and the depth image data includes:

[0015] The color image data and the depth image data are stitched together to obtain the first fused image data;

[0016] Feature extraction is performed on the first fused image data to obtain the features of the first image data.

[0017] According to one embodiment of this application, the second image data features include at least one of the item depth detail features and the item region of interest.

[0018] According to one embodiment of this application, obtaining the item type information and item location information of the storage device based on the first image data features and the second image data features includes:

[0019] The first image data features and the second image data features are concatenated to obtain the second fused image data;

[0020] The second fused image data is decoupled and feature-enhanced to obtain target classification information, target location information, and regression of target detection boxes;

[0021] Based on the target classification information, the target location information, and the regression of the target detection box, the item type information and the item location information are obtained.

[0022] According to one embodiment of this application, obtaining the first image data feature based on the color image data and the depth image data includes:

[0023] The color image data and the depth image data are input into the backbone network of the object recognition model to obtain the first image data features output by the backbone network;

[0024] Based on the depth image data, second image data features are obtained, including:

[0025] The depth image data is input into the first convolutional network of the object recognition model to obtain the second image data features output by the first convolutional network;

[0026] The step of obtaining the item type information and item location information of the storage device based on the first image data features and the second image data features includes:

[0027] The first image data features and the second image data features are input into the decoupled head structure of the item recognition model to obtain the item type information and the item location information output by the decoupled head structure.

[0028] The item recognition model is trained based on an item image training set.

[0029] According to one embodiment of this application, the data of the item image training set includes positive item samples and negative item samples, wherein the positive item samples represent item samples within the user's limb range of motion, and the negative item samples represent item samples outside the user's limb range of motion.

[0030] Secondly, this application provides an item identification device for a storage device, the device comprising:

[0031] The acquisition module is used to acquire color image data and depth image data of the storage space of the storage device;

[0032] The first processing module is used to obtain first image data features based on the color image data and the depth image data;

[0033] The second processing module is used to obtain second image data features based on the depth image data;

[0034] The third processing module is used to obtain the item type information and item location information of the storage device based on the first image data features and the second image data features.

[0035] According to the item recognition device of the storage equipment of this application, by independently processing the depth image data, extracting the second image data features, and by fusing the color image data and the depth image data, extracting the first image data features, and fusing the shallow features and deep features, the depth information is effectively utilized, and the type and location of the item can be accurately identified. The location recognition does not require additional gravity sensors or other devices, and the system is simple and relatively low in cost.

[0036] Thirdly, this application provides a storage device, comprising:

[0037] A housing that defines a storage space;

[0038] A data acquisition device is disposed in the housing, and the data acquisition device is used to acquire depth data and color depth data;

[0039] The item identification device for the storage equipment described in the second aspect above is connected to the data acquisition device.

[0040] Fourthly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the item identification method for the storage device as described in the first aspect above.

[0041] Fifthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the item identification method for the storage device as described in the first aspect above.

[0042] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the item identification method for a storage device as described in the first aspect above.

[0043] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0044] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0045] Figure 1 This is one of the flowcharts illustrating the item identification method for storage devices provided in this application embodiment;

[0046] Figure 2 This is a schematic diagram of the structure of the item recognition device for the storage equipment provided in the embodiments of this application;

[0047] Figure 3 This is a schematic diagram of the process of entering items into the storage device provided in the embodiments of this application;

[0048] Figure 4 This is a second schematic flowchart of the item identification method for storage devices provided in the embodiments of this application;

[0049] Figure 5 This is a schematic diagram showing the location of the step line setting for the storage device provided in the embodiments of this application;

[0050] Figure 6 This is one of the structural schematic diagrams of the storage device provided in the embodiments of this application;

[0051] Figure 7 This is a second schematic diagram of the structure of the storage device provided in the embodiments of this application;

[0052] Figure 8 This is the third structural schematic diagram of the storage device provided in the embodiments of this application;

[0053] Figure 9 This is the fourth structural schematic diagram of the storage device provided in the embodiments of this application;

[0054] Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0055] Figure label:

[0056] Housing 310, data acquisition device 320, drawer 330, shelf 340, door 350.

[0057] The first jump line is 410, and the second jump line is 420. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0059] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0060] It should be noted that the storage device in this embodiment can be understood as a refrigeration storage device in a broad sense, including but not limited to refrigerators, freezers, display cases, beverage cabinets, wine cabinets, refrigerated display cases, and refrigerated vending machines. The storage devices have diverse structural forms and a wide range of applications.

[0061] The following description, in conjunction with the accompanying drawings, details the item identification method for storage devices, item identification device for storage devices, storage devices, electronic devices, and readable storage media provided in this application through specific embodiments and application scenarios.

[0062] The storage device in this application embodiment may include a housing 310, a door 350, and a data acquisition device 320.

[0063] The housing 310 defines a storage space for storing items.

[0064] The housing 310 has an opening, and the door 350 is closable at the opening. When the door is open, the user can enter the storage space through the opening to retrieve or place items.

[0065] The data acquisition device 320 can be installed in the housing 310. The data acquisition device 320 is used to acquire depth data and color depth data.

[0066] Color depth data includes color data (also known as color image data) and depth data (also known as depth image data).

[0067] Understandably, for a given scenario, the data acquisition device 320 can collect both depth data and color depth data.

[0068] In actual implementation, the data acquisition device 320 can be an RGBD camera, which includes an RGB sensor and a depth sensor. The RGB sensor is used to acquire RGB images (i.e., color data), and the depth sensor is used to acquire depth images (i.e., depth data). When the RGB sensor and the depth sensor acquire an RGB image and a depth image simultaneously, the resulting RGB-D image pair is the color depth data.

[0069] The field of view of the data acquisition device 320 refers to the range of the scene that the data acquisition device 320 can capture. The field of view of the data acquisition device 320 can cover the edge of the storage space to fully collect the depth data and color depth data of the storage space.

[0070] For example, such as Figure 6 As shown, the data acquisition device 320 can be installed at the top of the housing 310 of the storage device, and the field of view of the data acquisition device 320 (shown by the dotted line in the figure) can cover the left and right edges of the shelf 340.

[0071] It should be noted that if the storage device has a pull-out drawer 330 or other storage components, the field of view of the data acquisition device 320 can cover the range of movement of the drawer 330 or other storage components in order to acquire relevant images of the items inside the storage components.

[0072] For example, such as Figure 7 As shown, the data acquisition device 320 can be installed on the top of the housing 310 of the storage device near the door 350. The field of view of the data acquisition device 320 (shown by the dotted line in the figure) can cover the rear edge of the shelf 340 near the inner liner, as well as the edge of the lower drawer 330 when it is pulled out.

[0073] This application provides a method for identifying items in a storage device, which can accurately identify the type and location of items. The system is simple and relatively low in cost.

[0074] like Figure 1 As shown, the item identification method of the storage device includes steps 110, 120, 130 and 140.

[0075] Step 110: Obtain color image data and depth image data of the storage space of the storage device.

[0076] In this step, color image data and depth image data of the storage space can be collected by the data acquisition device 320 of the storage device.

[0077] In practice, color image data and depth image data can be acquired using an RGBD camera. An RGBD camera includes an RGB sensor and a depth sensor. The RGB image acquired by the RGB sensor is the color image data, and the depth image acquired by the depth sensor is the depth image data.

[0078] When an RGB sensor and a depth sensor simultaneously acquire an RGB image and a depth image, the resulting RGB-D image pair is the color depth data. The color depth data can be divided into the color image data acquired by the RGB sensor and the depth image data acquired by the depth sensor.

[0079] It should be noted that the color image data and depth image data can be aligned on the sensor of the data acquisition device 320, or the color image data and depth image data can be aligned through a pre-processing algorithm.

[0080] Step 120: Based on the color image data and depth image data, obtain the first image data features.

[0081] Among them, color image data represents the color information in the scene. It can record the color and texture of objects through three channels: red (R), green (G), and blue (B), reflecting the visible optical characteristics of objects.

[0082] Depth image data represents the distance information of each pixel in the scene, records the spatial positional relationship between the object and the sensor, and reflects the three-dimensional structure of the object and its surrounding scene.

[0083] In this step, feature extraction is performed based on color image data and depth image data to obtain first image data features. The first image data features may include the visible optical features and three-dimensional structural features of the object. The first image data features belong to the deep feature map and can be used to capture the global semantics of the scene.

[0084] Step 130: Based on the depth image data, obtain the second image data features.

[0085] In this step, feature extraction is performed on the depth image data separately to obtain the second image data features, which include the three-dimensional structural features of the object.

[0086] It should be noted that the second image data feature is a shallow feature map extracted from depth image data, which can capture subtle differences in depth values ​​and local structures in the scene.

[0087] Step 140: Based on the features of the first image data and the features of the second image data, obtain the item type information and item location information of the storage device.

[0088] In this embodiment, the first image data features and the second image data features can be fused. The first image data features are used to capture global semantics, and the second image data features are used to capture subtle differences in depth values ​​and local structures. This allows for accurate identification of items within the storage space, obtaining information on the type and location of items stored in the storage device.

[0089] In related technologies, there are two main methods for vision-based species and location detection. One method is to use visible light images to predict species and location, and depth images to help confirm the location. The other method is to extract RGB-D features by stitching together RGB images and depth images to predict species and location. Neither of these methods effectively utilizes depth information, resulting in low detection accuracy.

[0090] In this embodiment, a shortcut connection is designed in the network inference process. The depth image data is processed independently, and the shallow features (second image data features) of the depth image data are extracted separately. Then, these features are combined with the deep features (first image data features) extracted from the fused color image data and depth image data. This effectively utilizes the information contained in the depth image data and fully captures the depth, color, texture and other features of the items in the storage space. This can achieve higher detection and recognition accuracy while supporting the detection of the item's location.

[0091] According to the item identification method for storage devices provided in the embodiments of this application, by independently processing the depth image data, extracting the second image data features, and fusing the color image data and depth image data, extracting the first image data features, and fusing shallow and deep features, the depth information is effectively utilized, which can accurately identify the type and location of items. The location identification does not require additional gravity sensors or other devices, and the system is simple and relatively low in cost.

[0092] In some embodiments, before step 110, acquiring the color image data and depth image data of the storage space of the storage device, the item identification method for the storage device may further include:

[0093] Acquire target depth data for storage devices;

[0094] In response to a step signal in the target depth data, color image data and depth image data are acquired.

[0095] In this embodiment, target depth data is collected by data acquisition device 320. The target depth data at the opening of the storage device can be collected when the door 350 is open, or the target depth data inside the storage device can be collected when the door 350 is closed.

[0096] Understandably, for the same scene, the amount of depth data is less than the amount of color data. Collecting target depth data and determining whether to trigger the collection of color depth data can effectively reduce the amount of data collected and the amount of data processed subsequently.

[0097] It should be noted that a step refers to a sudden change in the pattern of depth data, which can reflect whether the pattern of depth data conforms to certain action characteristics. In actual execution, the degree to which depth data conforms to the change characteristics when a certain action occurs can be obtained by confidence calculation. The depth change characteristics brought about by a certain action are assigned a certain confidence level, and the action is judged based on the confidence level.

[0098] In some embodiments, the step signal of depth data can refer to the change characteristics of depth data that are consistent with the user picking up or placing items, thereby distinguishing whether there is user activity of picking up or placing items. The depth change characteristics caused by the user picking up or placing items are assigned a high confidence level. Based on the confidence level, it is determined whether the user has picked up or placed items, so as to identify the picking up or placing of items.

[0099] In this embodiment, target depth data is acquired in real time for step detection. If the door 350 of the storage device is open, a corresponding step signal is detected in the target depth data, which may indicate that a user's limbs have entered the storage space to retrieve or place items. This triggers the data acquisition device 320 to collect color image data and depth image data of the storage space. Based on the color image data and depth image data, the user's retrieval of items can be identified, enabling seamless item entry.

[0100] In other embodiments, a step signal in the depth data can indicate a change in depth data consistent with the characteristics of an object moving. This helps to determine whether object movement has occurred. The depth change characteristics caused by object movement are assigned a high confidence level. Based on the confidence level, it is determined whether object movement has occurred, thus identifying the moved object. If the door 350 of the storage device is closed, and a corresponding step signal is detected in the target depth data, it indicates that the object inside the storage device may have moved. This triggers the data acquisition device 320 to collect color image data and depth image data of the storage space. Based on the color and depth data, the movement of the object can be identified, and corresponding prompts are output to alert the user that the object inside the storage device has moved.

[0101] In actual implementation, target depth data of the target detection location of the storage device can be obtained. The target detection location is a small area of ​​the storage device. By obtaining the target depth data of the target detection location and performing step detection, the amount of data collected and the amount of data processed can be further reduced.

[0102] In practice, the target detection location can be adjusted according to different object recognition tasks.

[0103] For example, for the task of recognizing items, the target detection location can be a small area at the opening of the storage device, where the user's limbs enter the storage device through the opening to retrieve or place items.

[0104] For example, in the task of recognizing the movement of objects, the target detection location can be a small area within the storage device where objects are prone to movement.

[0105] In related technologies, cameras are required to continuously acquire and recognize images, resulting in a large amount of data and computational load, which is not conducive to data transmission and real-time output of detection results, and the hardware cost is also high.

[0106] In this embodiment, target depth data is acquired in real time for step detection. When a corresponding step signal appears in the target depth data, the color image data and depth image data of the storage device are triggered to be collected, thereby realizing the identification of the type and location of the item. The amount of depth data is less than that of color data. The step detection data of depth data is simple to process and has a small computational load, which can effectively reduce the amount of data collected and processed, which is beneficial to data transmission and real-time output of detection results, and can also effectively reduce hardware costs.

[0107] In some embodiments, step 120, obtaining the first image data features based on color image data and depth image data, may include:

[0108] The color image data and depth image data are stitched together to obtain the first fused image data;

[0109] Feature extraction is performed on the first fused image data to obtain the first image data features.

[0110] In this embodiment, color image data and depth image data are stitched together and fused to obtain a first fused image data. Then, feature extraction is performed on the first fused image data to obtain the deep feature of the first image data.

[0111] The first fused image data can have four channels, including three color channels (R, G, B) of the color image data, and the depth information of the depth image data as the fourth channel.

[0112] In practice, feature extraction is performed on the first fused image data obtained by stitching together color image data and depth image data. The extracted features of the first image data incorporate the three-dimensional structural features and visible optical features of the object, which helps to improve the target discrimination and target localization capabilities in the object recognition process and enhances robustness.

[0113] In some embodiments, the second image data features include at least one of the item depth detail features and the item region of interest.

[0114] Among them, the object depth detail feature is used to characterize the micro-texture, shape contour, three-dimensional structure and local geometric changes of the object surface. The object depth detail feature can also reflect the physical state and integrity of the object.

[0115] The Region of Interest (ROI) refers to a specific region in an image that may contain an object. It is the key area that the algorithm focuses on. Identifying the ROI can reduce the amount of computation and improve detection efficiency.

[0116] In practice, the region of interest for an object in the image can be determined by the depth heatmap corresponding to the depth image data. The region of interest can be a highly significant area in the depth heatmap, which has high attention value or importance.

[0117] In this embodiment, feature extraction is performed on the depth image data separately, and the resulting second image data features contain a large amount of effective shallow feature information such as object depth detail features and object region of interest.

[0118] In some embodiments, step 140, obtaining the item type information and item location information of the storage device based on the first image data features and the second image data features, may include:

[0119] The first image data features and the second image data features are concatenated to obtain the second fused image data;

[0120] The second fused image data is decoupled and feature-enhanced to obtain target classification information, target location information, and regression of target detection boxes;

[0121] Based on the regression of target classification information, target location information, and target detection boxes, we obtain item type information and item location information.

[0122] In this embodiment, the first image data features and the second image data features are first spliced ​​and fused. The fused second image data is then decoupled and feature-enhanced, and the target classification information, target location information and target detection box regression of each item in the storage device are output respectively.

[0123] Among them, target classification information refers to the category to which the item belongs, target location information refers to the specific coordinates of the target within the item, and target detection box regression refers to predicting the bounding box of the item, which can accurately define the location and range of the item.

[0124] In practice, the second fused image data can be decoupled and feature enhanced by two-way decoupling and feature enhancement. The second fused image data is enhanced and decoupled by one way to output target classification information, and the second fused image data is enhanced and decoupled by another way to output target location information and target detection box regression, respectively.

[0125] In this embodiment, after obtaining the target classification information, target location information, and target detection box regression, the target classification information, target location information, and target detection box regression are summarized, and the item type information and item location information of each item in the storage device are output.

[0126] Understandably, for items within the storage space of a storage device, a target detection box can be marked for each item. Based on the number of target detection boxes and the corresponding target classification information, the quantity information of each type of item can be obtained.

[0127] In this embodiment, category and location prediction are integrated. By utilizing the strong correlation between the item location and the item detection results in the depth image data and color image data, the second fused image data is enhanced and decoupled. The extracted deep and shallow features are fully utilized, which effectively improves the accuracy of classification and location detection, has high data utilization, and also helps to improve the efficiency of item recognition.

[0128] In practice, the identification of items in storage devices can be achieved through an item identification model, which may include a backbone network, a first convolutional network, and a decoupled head structure.

[0129] In some embodiments, step 120, obtaining the first image data features based on color image data and depth image data, may include:

[0130] Color image data and depth image data are input into the backbone network of the object recognition model to obtain the first image data features output by the backbone network.

[0131] Step 130: Based on the depth image data, obtain the second image data features, which may include:

[0132] The depth image data is input into the first convolutional network of the object recognition model to obtain the second image data features output by the first convolutional network;

[0133] Step 140: Based on the features of the first image data and the features of the second image data, obtain the item type information and item location information of the storage device, which may include:

[0134] The first image data features and the second image data features are input into the decoupled head structure of the item recognition model to obtain the item type information and item location information output by the decoupled head structure.

[0135] The object recognition model is trained based on an object image training set.

[0136] Understandably, an object recognition model is a model that performs object detection tasks. It can detect objects in storage space and output information such as the type, location, and quantity of the objects.

[0137] In this embodiment, the backbone network is an architecture used to extract first image data features from the first fused image data (which is obtained by stitching together color image data and depth image data). It can be built based on deep convolutional neural networks such as ResNet. The output first image data features can provide rich semantic and spatial information support for subsequent classification and regression tasks.

[0138] The first convolutional network is a convolutional layer used to extract features from depth image data. It can capture texture, shape, and spatial information, providing basic feature representations for target classification and localization.

[0139] It should be noted that the shallow features of depth image data carry a lot of useful information. The first convolutional network can extract the shallow features such as the depth details of the object and the region of interest of the object. There is no need to design a high-dimensional feature extraction structure for depth image data.

[0140] In this embodiment, the depth image data is isolated and a single-layer convolution is performed on the first convolutional network to extract shallow features (second image data features). These features are then fused with the first image data features extracted from the fusion of depth image data and color image data and input into the decoupled head structure. The independent network branches in the decoupled head structure process the category prediction and detection box regression respectively, which improves the learning efficiency of the object recognition model, reduces the number of parameters required by the object recognition model, and helps maintain the excellent inference performance of the object recognition model.

[0141] Understandably, the object recognition model is trained before deployment. First, the network structure of the object recognition model is built, a training set of object images is obtained (which can be labeled data), the object recognition model is trained and optimized using the object image training set, the parameters of the object recognition model are updated, and a trained object recognition model is obtained.

[0142] In some embodiments, the data in the training set of object images includes positive object samples and negative object samples. Positive object samples represent object samples within the user's limb range of motion, while negative object samples represent object samples outside the user's limb range of motion.

[0143] Positive samples refer to samples that contain the target, while negative samples refer to samples that do not contain the target.

[0144] In practice, training the object recognition model with positive and negative samples helps the model better distinguish between the detected target and the background, enhances the accuracy of target recognition, reduces false detections and false negatives, and improves the generalization performance and robustness of the object recognition model.

[0145] In this embodiment, items within the user's limb movement range are labeled as positive samples, and items outside the user's limb movement range are labeled as negative samples. This allows the item recognition model to focus on items picked up and placed by the user. For the item entry recognition task, items unrelated to the picking and placing state are not detected. This can effectively improve the detection efficiency of the item recognition model while ensuring the accuracy of item entry.

[0146] For example, during the data labeling process, the training set of images of items is obtained by photographing the user's picking up and putting down the items. Positive samples of items cover the items that the user touches during the picking up and putting down the items. Other items, such as stationary items on the shelf 340 and items that are not pulled out of the drawer 330, can be labeled as negative samples of items.

[0147] The following is a specific example.

[0148] The data acquisition device 320 continuously acquires target depth data and performs step detection. When a step signal in the target depth data is detected, the data acquisition device 320 is triggered to acquire color depth data, including color image data and depth image data, and inputs it into the object recognition model for detection.

[0149] like Figure 4 As shown, the input data is divided into depth images and color optical images (i.e. color image data). The two sets of data are first stitched together to obtain the first fused image data. The first fused image data is then input into the backbone network for feature extraction to extract the deep features of the fused data.

[0150] Meanwhile, the depth image data is independently subjected to single-layer convolution operations, and the depth image data is input into the first convolutional network to extract shallow features of the depth data.

[0151] The first image data features output by the backbone network and the second image data features output by the first convolutional network are concatenated by channels. After further fast feature fusion, the resulting second fused image data is enhanced and input into the decoupling head structure for decoupling.

[0152] In the decoupled head structure, one path outputs the target type classification (i.e., target classification information) after feature enhancement, while the other path continues to decouple after feature enhancement, outputting the target location classification and the target detection box regression respectively.

[0153] It should be noted that the object recognition model can focus on the items that the user picks up and puts down. In addition to the classification of items, the object recognition model can also include the classification of the user's limbs.

[0154] The target locations can be categorized according to different areas of the storage space, for example, such as... Figure 9 As shown, the target location is classified into 9 categories: A, B, C, D, E, F, G, H, and I, which indicate different locations in the shelf 340 area. Below the shelf 340 is the drawer 330.

[0155] The regression of the detection frame is used to determine the position and quantity of items and user limbs in the field of view of the data acquisition device 320. It can be used to capture data of the item area for archiving or sharing with the user, as well as to determine the number of items entered into the system.

[0156] The final result (including item type information and item location information) is output by summarizing the target type classification, target location classification, and target detection box regression.

[0157] In this embodiment, a shortcut connection is designed in the network inference process. The depth image data is isolated and a single-layer convolution is performed through the first convolutional network to extract shallow features. Then, it is spliced ​​and fused with the deep features extracted from the first fused image data to improve the learning efficiency of the network, reduce the number of parameters required by the network, and maintain the excellent inference performance of the object recognition model.

[0158] Furthermore, based on the strong correlation between the object location and the object detection results in the depth image data and color image data, the classification prediction and location prediction are integrated, and a decoupled head structure is designed to fully and efficiently utilize the relevant feature information of the second fused image data. The algorithm has a compact structure, high data utilization, and low repetition rate, which can significantly improve the system's operating efficiency.

[0159] To more clearly illustrate the practical application effect of this application, the following will describe the item recognition method of this application embodiment in detail with the specific scenario of item entry for storage devices. It should be noted that the following embodiments are only illustrative examples, and those skilled in the art can make appropriate adjustments or extensions to the technical solution according to actual needs to achieve the same or similar technical effects.

[0160] In some embodiments, the item entry for the storage device can be achieved through the following steps:

[0161] With the door of the storage device open at 350°, acquire the target depth data of the target detection position of the storage device;

[0162] In response to a step signal in the target depth data, color depth data of the storage space of the storage device is collected;

[0163] Based on the color depth data of the storage space, the item storage information of the storage device is determined.

[0164] In this embodiment, color depth data of the storage space is collected. The color depth data includes color image data and depth image data. Based on the color image data and depth image data, a first image data feature is obtained. Based on the depth image data, a second image data feature is obtained. Based on the first image data feature and the second image data feature, the item type information and item location information of the storage device are obtained, thereby determining the item storage information of the storage device.

[0165] The target detection location is set at the opening of the storage device.

[0166] Understandably, when the door of the storage device is 350 degrees open, users can take or put items in through the opening. By obtaining target depth data of the target detection position, the opening of the storage device can be monitored in real time, and the user's actions of taking or putting items in can be detected in a timely manner.

[0167] It should be noted that the target detection location is a small area at the opening, and for the same scene, the amount of depth data is less than the amount of color data. Obtaining the target depth data at the target detection location and determining whether to trigger the collection of color depth data can effectively reduce the amount of data collected and the amount of data processed subsequently.

[0168] In actual implementation, magnetic switches, mechanical switches, and other devices can be installed on the storage equipment to detect the opening and closing status of the door 350. If the door 350 is detected to be open, the data acquisition device 320 collects the target depth data of the target detection position.

[0169] In this embodiment, target depth data at the target detection location is acquired in real time for step detection. When a corresponding step signal appears in the target depth data, it indicates that a user's limb may enter the storage space to retrieve or place items, triggering the data acquisition device 320 to collect the color depth data of the storage space.

[0170] Based on the color depth data collected in response to the step signal, the system detects items that the user's limbs enter and take out of the storage space, identifies the type and location of the items in the storage space, records the information of the items taken out and put in, and obtains the item storage information of the storage device.

[0171] In this embodiment, based on the color depth data of the storage space, the trained item recognition model can identify all the items stored in the storage space and obtain the item storage information of the storage device; alternatively, it can only identify items that the user's limbs enter the storage space to retrieve or place, and update the item storage information of the storage device in real time.

[0172] In practice, an item information database can be set up. The item information database contains item storage information such as the type and location of items stored in the storage device. It collects color depth data in response to the step signal of the target depth data, identifies the items to be taken or placed based on the color depth data, and updates the item storage information in the item information database, which can effectively improve the efficiency of item entry.

[0173] Understandably, when recognizing objects based on color depth data, a detection box can be marked for each object, and the quantity information of each type of object can be obtained based on the quantity and type information of the detection boxes.

[0174] In related technologies, vision-based object entry requires continuous image acquisition and recognition by a camera, resulting in a large amount of data and computational load, which is not conducive to data transmission and real-time output of detection results. The hardware cost is also high. Some technologies that start image acquisition and recognition only under specific conditions, such as starting image acquisition and recognition after a door is opened, or performing human body part detection first and then performing image recognition after successful detection, still have the problem of data redundancy. Most of them require neural networks to recognize each frame of image, and the computational load is still very heavy.

[0175] In this embodiment, a small target detection position is set at the opening of the storage device. When the door 350 is opened, the target depth data of the target detection position is acquired in real time for step detection. When the target depth data shows a corresponding step signal, the color depth data of the storage space is triggered to be collected. Item recognition is performed on the color depth data collected in response to the step signal. There is no need to continuously collect data or recognize every frame of data. The lightweight triggering method can effectively reduce the amount of data collected and processed, which is beneficial for data transmission and real-time output of detection results. It also helps to reduce the hardware cost of the device. By using color depth data for item recognition and comprehensively utilizing color and depth information, the recognition accuracy of item entry can also be improved.

[0176] In some embodiments, the target detection location includes a first step line 410 and a second step line 420 located opposite each other at the opening, with the first step line 410 near the storage space and the second step line 420 near the door 350.

[0177] Understandably, the target detection location can be adjusted according to the structure of the storage device.

[0178] For example, such as Figure 5 As shown, the target detection position is located at the front edge of the shelf 340 of the storage equipment. The first step line 410 and the second step line 420 are arranged opposite to each other and spaced apart. Figure 5 An upward arrow indicates the direction towards storage space. Figure 5 The downward arrow indicates the direction towards door 350.

[0179] like Figure 8 As shown, the data acquisition device 320 is configured with the positions of the first step line 410 and the second step line 420, and the center line of the field of view of the data acquisition device 320 can coincide with the outer edge of the drawer 330.

[0180] The first step line 410 is located 5 pixels from the outer edge of the shelf 340 toward the door 350, and the second step line 420 is located 20 pixels from the outer edge of the shelf 340 toward the door 350. The first step line 410 and the second step line 420 are spaced 15 pixels apart.

[0181] In this embodiment, target depth data of the target detection location is obtained, namely, depth data at the first step line 410 and depth data at the second step line 420. Step detection is performed using the depth data of the first step line 410 and the second step line 420 to determine whether to trigger the collection of color depth data. This allows for lightweight real-time monitoring of the opening of the storage device and identification of the direction of the step signal, enabling timely perception of the user's actions of taking or placing items.

[0182] The first step line 410 is close to the storage space, and the second step line 420 is close to the door 350. When the second step line 420 is triggered before the first step line 410 (i.e., an object passes through the second step line 420 before the first step line 410), it indicates that the user's limbs have entered the storage space, triggering a forward step. When the first step line 410 is triggered before the second step line 420, it indicates that the user's limbs have moved out of the storage space, triggering a reverse step.

[0183] In actual execution, the confidence level corresponding to the first step line 410 can be calculated based on the depth data at the first step line 410, and the confidence level corresponding to the second step line 420 can be calculated based on the depth data at the second step line 420. The step line that is triggered first can be determined based on the confidence level.

[0184] In the step triggering determination, the first step line 410 and the second step line 420 can be determined independently based on whether their confidence levels are greater than a threshold. If the confidence level of the second step line 420 is less than the threshold and the confidence level of the first step line 410 is greater than the threshold, the first step line 410 is triggered first, and it is determined to be a reverse step signal triggering. If the confidence level of the second step line 420 is greater than the threshold and the confidence level of the first step line 410 is less than the threshold, the second step line 420 is triggered first, and it is determined to be a positive step signal triggering.

[0185] In some embodiments, acquiring color depth data of the storage space of the storage device in response to a step signal of target depth data may include:

[0186] In response to the first step signal of the target depth data, the first color depth data of the storage space is acquired;

[0187] In response to a second step signal in response to the target depth data, second color depth data of the storage space is acquired;

[0188] Based on the color depth data of the storage space, the item storage information of the storage device can be determined, which may include:

[0189] Based on the first color depth data and the second color depth data, the item storage information of the storage device is determined.

[0190] In this embodiment, the direction of the second step signal is opposite to that of the first step signal. The first step signal can be a positive step, and the second step signal can be a negative step.

[0191] A first step signal indicating a positive step in the target depth data indicates that a user's limb has entered the storage space. A second step signal indicating a negative step in the target depth data indicates that a user's limb has moved out of the storage space. First color depth data is acquired in response to the positive step, and second color depth data is acquired in response to the negative step. Based on the first and second color depth data, the situation of a user's limb entering and leaving the storage space and taking or putting away items can be identified.

[0192] In related technologies, there is no judgment on the user's storage status. Users may change the storage location immediately after placing the item, resulting in low effectiveness of item location prediction.

[0193] In this embodiment, when the storage device is open, a step signal of the target depth data at the opening is detected in real time. In response to the first step signal, first color depth data is collected, and in response to the second step signal, second color depth data is collected. Based on the first color depth data and the second color depth data, the user's limbs can be identified during the process of entering and leaving the storage space to pick up and put down items, and the user's storage and retrieval status can be determined. Based on the user's storage and retrieval status, the effectiveness of item location prediction can be effectively improved, and the item storage location can be predicted more accurately.

[0194] In some embodiments, determining the item storage information of the storage device based on the first color depth data and the second color depth data may include:

[0195] Based on the first color depth data, determine the detection result of the first item;

[0196] The detection result of the second item is determined based on the second color depth data;

[0197] Based on the results of the first and second item inspections, the item storage information of the storage device is determined.

[0198] In this embodiment, when a first step signal of the target depth data is detected, the data acquisition device 320 is triggered to acquire first color depth data. Based on the first color depth data, item recognition is performed to obtain a first item detection result corresponding to the first color depth data. At the same time, the acquisition of target depth data is continued. When a second step signal of the target depth data is detected, the data acquisition device 320 is triggered to acquire second color depth data. Based on the second color depth data, item recognition is performed to obtain a second item detection result corresponding to the second color depth data. By combining the first item detection result and the second item detection result, the item storage information of the storage device is obtained.

[0199] The first item detection result may include item detection information and user limb information for operating the storage device, and the second item detection result may include item detection information and user limb information for operating the storage device.

[0200] It is understandable that the user's limbs operating the storage device are the user's limbs appearing at the opening of the storage device in the field of view of the data acquisition device 320. The user's limbs enter the storage device from the opening and move inside the storage device to complete the action of taking and putting away items.

[0201] In practice, an object recognition model can be used to detect targets in the first color depth data and identify the corresponding first object detection result. The first object detection result can include object detection information such as object type and object location, and can also include user limb information for operating the storage device.

[0202] An object recognition model can be used to perform target detection on the second color depth data and identify the corresponding second object detection result. The second object detection result can include object detection information such as object type and object location, and can also include user limb information of the user operating the storage device.

[0203] In this embodiment, the same object recognition model or detection network can be used to perform target detection on the first color depth data and the second color depth data respectively, so as to obtain the first object detection result and the second object detection result.

[0204] Understandably, the first and second item detection results can respectively represent the item status when the user's limbs enter the storage space and the item status when the user's limbs are removed from the storage space. By combining the first and second item detection results, the user's storage and retrieval status can be determined, the item location can be predicted more accurately, and highly accurate item storage information can be obtained.

[0205] It should be noted that, based on the first color depth data and the second color depth data, it is possible to detect only items within the range of the user's limb movements, which helps to reduce the computational load, improve recognition efficiency, and output more accurate detection results in real time.

[0206] In practice, the detection information in the first and second item detection results can be compared to detect whether the items within the user's limb movement range have changed, and to determine whether the user has picked up or put in the items.

[0207] It should be noted that the user's limb movement range refers to the area where the user's limbs appear in the item detection results. Items within the area where the user's limbs do not appear can be considered unchanged. By judging whether the item detection information within the user's limb movement range has changed, the user's access status can be accurately identified.

[0208] Understandably, based on the first color depth data and the second color depth data, it is also possible to detect items in the entire area of ​​the storage space. Combined with the judgment of the user's storage and retrieval status, it is possible to comprehensively and accurately identify information such as the type, quantity, and storage location of items stored in the storage device.

[0209] The embodiments of this application will be described in detail below from three different implementation perspectives.

[0210] 1. The user did not pick up or leave the item.

[0211] In some embodiments, determining the item storage information of the storage device based on the first item detection result and the second item detection result may include:

[0212] If the item detection information of the user's limb range of motion in the first item detection result is the same as the item detection information of the user's limb range of motion in the second item detection result, the item storage information of the storage device remains unchanged.

[0213] In this embodiment, when the item detection information within the user's limb movement range is the same in the first item detection result and the second item detection result, that is, the item type, quantity and position are unchanged, it indicates that the user has not picked up, placed or moved the item, and the item storage information of the storage device remains unchanged.

[0214] 2. The user takes out the item and puts it back in.

[0215] In some embodiments, determining the item storage information of the storage device based on the first item detection result and the second item detection result may include:

[0216] If the item detection information of the user's limb range of motion in the first item detection result is different from the item detection information of the user's limb range of motion in the second item detection result, the item held by the user's limb in the first item detection result is entered into the item storage information of the storage device, and the item held by the user's limb in the second item detection result is removed from the item storage information of the storage device.

[0217] In this embodiment, in response to the first color depth data acquired via a forward step, a first item detection result is determined. The item held by the user's limb in the first item detection result can be an item placed by the user in the storage device, and the item can be recorded in the item storage information of the storage device. In response to the second color depth data acquired via a reverse step, a second item detection result is determined. The item held by the user's limb in the second item detection result can be an item taken out by the user from the storage device, and the item can be removed from the item storage information of the storage device, and the item storage information of the storage device is updated.

[0218] Take a refrigerator as an example of a storage device.

[0219] The food database includes information such as the types and storage locations of food items A, B, and C in the refrigerator. If the first item detection result shows that the item held by the user's limbs is food item D, the type and storage location of food item D will be entered into the food database. If the second item detection result shows that the item held by the user's limbs is food item A, the type and storage location of food item A will be removed from the food database.

[0220] 3. The user puts in an item but does not take it out.

[0221] In some embodiments, determining the item storage information of the storage device based on the first item detection result and the second item detection result may include:

[0222] If the item detection information of the user's limb movement range in the first item detection result is different from the item detection information of the user's limb movement range in the second item detection result, and the user's limb is not holding an item in the second item detection result, then the item held by the user's limb in the first item detection result is recorded into the item storage information of the storage device.

[0223] In this embodiment, the item detection information of the user's limb movement range in the first item detection result and the second item detection result are different, and the user's limbs are not holding items in the second item detection result, indicating that the user only puts items into the storage device and does not take items out of the storage device. Therefore, the items held by the user's limbs in the first item detection result can be recorded into the item storage information of the storage device.

[0224] Take a refrigerator as an example of a storage device.

[0225] The food database includes information such as the types and storage locations of food items A, B, and C in the refrigerator. If the first item detection result shows that the user is holding food item D, and the second item detection result shows that the user is not holding any item, the information such as the type and storage location of food item D will be entered into the food database.

[0226] The item entry method for storage devices provided in this application embodiment can achieve seamless item entry with small data volume and computational load, and high recognition accuracy.

[0227] The following is a specific example.

[0228] The refrigerator's computer board stores a food database, which includes information such as the type, quantity, and storage location of the food in the refrigerator. The food database may also include information such as the expiration date and nutritional composition of the food.

[0229] like Figure 3 As shown, after the refrigerator is powered on, it performs status detection. When the refrigerator door 350 is detected to be open (in the open state), the target depth data at the first step line 410 and the second step line 420 at the refrigerator opening is acquired in real time. In response to the step signal of the target depth data, the color depth data of the storage space is collected, the type and location of the items are identified, the information of the items taken out and put in is recorded, and the food database information is updated.

[0230] like Figure 5 As shown, the first step line 410 and the second step line 420 are set on the outer edge of the shelf 340. The first step line 410 is set near the inner liner of the refrigerator, and the second step line 420 is set near the door 350 of the refrigerator.

[0231] When a positive step detection is performed and the first step signal corresponding to the positive step is triggered, the sensor of the data acquisition device 320 is triggered to acquire color depth data. The target detection is performed on the acquired first color depth data through the detection network (i.e., the object recognition model), the food and storage location are identified, and the recognition result 1 (i.e. the first item detection result) is output. Result 1 is temporarily stored.

[0232] At the same time, the data acquisition device 320 maintains the acquisition of target depth data at the first step line 410 and the second step line 420, performs reverse step detection, and when the second step signal corresponding to the reverse step is triggered, the sensor of the data acquisition device 320 is triggered again to acquire color depth data. The target detection is performed on the acquired second color depth data through the detection network, and the recognition result 2 (second item detection result) is output.

[0233] When result 2 is valid, the relationship between result 2 and result 1 is determined. If the ingredients detected by result 1 and result 2 are the same, the ingredient database is not updated, the item storage information remains unchanged, and the data acquisition device 320 continues to acquire the target depth data at the step line.

[0234] If the ingredients in Result 1 and Result 2 are different, the ingredients in Result 1 are entered, and the ingredients in Result 2 are removed from the ingredient library. The data acquisition device 320 continues to acquire the target depth data at the step line.

[0235] If result 2 detects a human upper limb (i.e., the user's limb) but not food, then the food from result 1 is entered, and the data acquisition device 320 continues to acquire target depth data at the step line until the refrigerator door 350 is closed.

[0236] In the foregoing embodiments, the item recognition method provided in this application is applied to the scenario of non-contact item entry, which can effectively improve the accuracy of item entry. It is understood that the item recognition method provided in this application is also applicable to the scenario of item movement detection, which can also improve the accuracy of item movement detection.

[0237] In actual operation, when the door 350 is closed, the data acquisition device 320 can continuously collect the target depth data of the step line (at this time, the step line can be set at any position of the storage device where the movement of food needs to be detected, instead of at the opening). At this time, whether a positive or negative step signal is triggered, the sensor of the data acquisition device 320 will be triggered to collect color depth data and detect it through the item recognition model. If the item recognition model detects food, it will be recorded in the abnormal food database, indicating that the food has moved while the refrigerator door is closed, thus realizing the detection of item movement.

[0238] In this embodiment, users can be notified of abnormal food conditions and reminded of food falling through interactive methods such as APP, voice, and display screen. The above cycle continues in the closed state until the refrigerator door 350 is opened, and the item entry recognition process begins.

[0239] Understandably, users can also set refrigerator modes through interactive methods such as apps, voice commands, and displays. For example, in low-energy mode, the data acquisition device 320 can be stopped when the door is closed, the food movement reminder can be turned off, and only the food entry function when the door is open can be retained. Users can also actively update the food storage information in the food database through interactive methods such as apps, voice commands, and displays.

[0240] The item identification method for storage devices provided in this application can be executed by an item identification device for the storage device. This application uses an item identification device for the storage device executing the item identification method as an example to illustrate the item identification device for the storage device provided in this application.

[0241] This application also provides an item identification device for storage equipment.

[0242] like Figure 2 As shown, the item recognition device of the storage equipment includes:

[0243] The acquisition module 210 is used to acquire color image data and depth image data of the storage space of the storage device;

[0244] The first processing module 220 is used to obtain first image data features based on color image data and depth image data;

[0245] The second processing module 230 is used to obtain second image data features based on depth image data;

[0246] The third processing module 240 is used to obtain the item type information and item location information of the storage device based on the first image data features and the second image data features.

[0247] According to the embodiment of this application, the storage device for identifying items extracts second image data features by independently processing depth image data and fusing color image data and depth image data to extract first image data features. By fusing shallow and deep features, the device effectively utilizes depth information and can accurately identify the type and location of items. The location identification does not require additional gravity sensors or other devices, making the system simple and relatively low in cost.

[0248] In some embodiments, the acquisition module 210 is further configured to acquire target depth data of the storage device;

[0249] In response to a step signal in the target depth data, color image data and depth image data are acquired.

[0250] In some embodiments, the first processing module 220 is used to stitch together color image data and depth image data to obtain first fused image data;

[0251] Feature extraction is performed on the first fused image data to obtain the first image data features.

[0252] In some embodiments, the second image data features include at least one of the item depth detail features and the item region of interest.

[0253] In some embodiments, the third processing module 240 is used to stitch together the first image data features and the second image data features to obtain the second fused image data;

[0254] The second fused image data is decoupled and feature-enhanced to obtain target classification information, target location information, and regression of target detection boxes;

[0255] Based on the regression of target classification information, target location information, and target detection boxes, we obtain item type information and item location information.

[0256] In some embodiments, the first processing module 220 is used to input color image data and depth image data into the backbone network of the object recognition model to obtain the first image data features output by the backbone network.

[0257] The second processing module 230 is used to input depth image data into the first convolutional network of the object recognition model to obtain the second image data features output by the first convolutional network.

[0258] The third processing module 240 is used to input the first image data features and the second image data features into the decoupling head structure of the item recognition model to obtain the item type information and item location information output by the decoupling head structure.

[0259] The object recognition model is trained based on an object image training set.

[0260] In some embodiments, the data in the training set of object images includes positive object samples and negative object samples. Positive object samples represent object samples within the user's limb range of motion, while negative object samples represent object samples outside the user's limb range of motion.

[0261] The item recognition device for storage equipment provided in this application embodiment can realize the various processes implemented in the above-described item recognition method embodiment for storage equipment. To avoid repetition, these processes will not be described again here.

[0262] This application also provides a storage device.

[0263] like Figure 6 As shown, the storage device includes a housing 310 and a data acquisition device 320.

[0264] The housing 310 defines the storage space, and the data acquisition device 320 is disposed in the housing 310. The data acquisition device 320 is used to acquire depth data and color depth data.

[0265] The storage device also includes an item identification device as described above, which is connected to the data acquisition device 320.

[0266] The storage device provided in the embodiments of this application extracts second image data features by independently processing depth image data, and extracts first image data features by fusing color image data and depth image data, fusing shallow features and deep features, and effectively utilizing depth information. This allows for accurate identification of item type and location. Location identification does not require additional gravity sensors or other devices, making the system simple and relatively low in cost.

[0267] In some embodiments, such as Figure 10As shown, this application embodiment also provides an electronic device 1000, including a processor 1001, a memory 1002, and a computer program stored in the memory 1002 and executable on the processor 1001. When the program is executed by the processor 1001, it implements the various processes of the above-described storage device item identification method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0268] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0269] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described embodiment of the item identification method for storage devices and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0270] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0271] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for identifying items in a storage device.

[0272] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0273] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described storage device item identification method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0274] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0275] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0276] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0277] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0278] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0279] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for identifying items in a storage device, characterized in that, include: Acquire color and depth image data of the storage space of the storage device; Based on the color image data and the depth image data, the first image data features are obtained; Based on the depth image data, the second image data features are obtained; Based on the first image data features and the second image data features, the item type information and item location information of the storage device are obtained.

2. The method for identifying items in a storage device according to claim 1, characterized in that, Before acquiring the color image data and depth image data of the storage space of the storage device, the method further includes: Obtain the target depth data of the storage device; In response to a step signal in the target depth data, the color image data and the depth image data are acquired.

3. The method for identifying items in a storage device according to claim 1, characterized in that, The process of obtaining the first image data feature based on the color image data and the depth image data includes: The color image data and the depth image data are stitched together to obtain the first fused image data; Feature extraction is performed on the first fused image data to obtain the features of the first image data.

4. The method for identifying items in a storage device according to claim 1, characterized in that, The second image data features include at least one of the object depth detail features and the object region of interest.

5. The method for identifying items in a storage device according to claim 1, characterized in that, The step of obtaining the item type information and item location information of the storage device based on the first image data features and the second image data features includes: The first image data features and the second image data features are concatenated to obtain the second fused image data; The second fused image data is decoupled and feature-enhanced to obtain target classification information, target location information, and regression of target detection boxes; Based on the target classification information, the target location information, and the regression of the target detection box, the item type information and the item location information are obtained.

6. The method for identifying items in a storage device according to any one of claims 1-5, characterized in that, The process of obtaining the first image data feature based on the color image data and the depth image data includes: The color image data and the depth image data are input into the backbone network of the object recognition model to obtain the first image data features output by the backbone network; Based on the depth image data, second image data features are obtained, including: The depth image data is input into the first convolutional network of the object recognition model to obtain the second image data features output by the first convolutional network; The step of obtaining the item type information and item location information of the storage device based on the first image data features and the second image data features includes: The first image data features and the second image data features are input into the decoupled head structure of the item recognition model to obtain the item type information and the item location information output by the decoupled head structure. The item recognition model is trained based on an item image training set.

7. The method for identifying items in a storage device according to claim 6, characterized in that, The training set of the object images includes positive object samples and negative object samples. The positive object samples represent object samples within the user's limb range of motion, and the negative object samples represent object samples outside the user's limb range of motion.

8. An item identification device for a storage device, characterized in that, include: The acquisition module is used to acquire color image data and depth image data of the storage space of the storage device; The first processing module is used to obtain first image data features based on the color image data and the depth image data; The second processing module is used to obtain second image data features based on the depth image data; The third processing module is used to obtain the item type information and item location information of the storage device based on the first image data features and the second image data features.

9. A storage device, characterized in that, include: A housing that defines a storage space; A data acquisition device is disposed in the housing, and the data acquisition device is used to acquire depth data and color depth data; The item identification device for storage equipment as described in claim 8, wherein the item identification device is connected to the data acquisition device.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the item identification method for the storage device as described in any one of claims 1-7.

Images