Article input method and device of storage device, storage device, and electronic device

Abstract

The application discloses an article entry method and device of a storage device, the storage device and an electronic device, and belongs to the technical field of the storage device. The method comprises the following steps: acquiring target depth data of a target detection position of the storage device under the condition that a door body of the storage device is opened, the target detection position being arranged at an opening of the storage device, and the door body being arranged at the opening in a foldable manner; collecting color depth data of a storage space of the storage device in response to a step signal of the target depth data; and determining article storage information of the storage device based on the color depth data of the storage space. The method can realize article non-inductive entry, has small data quantity and low calculation load, and has high recognition accuracy.

Description

Technical Field

[0001] This application belongs to the field of storage equipment technology, and particularly relates to a method, apparatus, storage equipment and electronic equipment for entering items into a storage device. Background Technology

[0002] With the development of smart technology, more and more storage devices are equipped with item management functions, which can automatically record stored items and provide users with item management suggestions. For example, smart refrigerators have food management functions, which can monitor the food stored in the refrigerator and provide users with recipes and food purchase suggestions. Currently, item management functions mainly rely on manual input, voice, and radio frequency identification (RFID) methods. These methods are not contactless input and are slow and cumbersome. Vision technology can autonomously sense items, but this method has a large data volume and computational load, and the recognition accuracy is relatively low. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a method, apparatus, storage device, and electronic device for inputting items into a storage device, which can achieve seamless input of items with low data volume and computational load, and high recognition accuracy.

[0004] Firstly, this application provides a method for inputting items into a storage device, the method comprising:

[0005] With the door of the storage device open, target depth data of the target detection position of the storage device is acquired. The target detection position is set at the opening of the storage device, and the door is closable at the opening.

[0006] In response to a step signal in the target depth data, color depth data of the storage space of the storage device is acquired;

[0007] Based on the color depth data of the storage space, the item storage information of the storage device is determined.

[0008] According to the item entry method of the storage device in this application, when the storage device is open, the target depth data of the target detection position at the opening is collected for step detection. When the step signal is detected, the color depth data of the storage space is collected for item recognition, the items taken and placed by the user are recorded, and the item storage information of the storage device is obtained. This method can realize seamless item entry, with small data volume and computational load, and high recognition accuracy.

[0009] According to one embodiment of this application, the step signal in response to the target depth data, acquiring the color depth data of the storage space of the storage device, includes:

[0010] In response to a first step signal of the target depth data, first color depth data of the storage space is acquired;

[0011] In response to a second step signal of the target depth data, second color depth data of the storage space is acquired, wherein the direction of the second step signal is opposite to the direction of the first step signal;

[0012] The process of determining the item storage information of the storage device based on the color depth data of the storage space includes:

[0013] Based on the first color depth data and the second color depth data, the item storage information of the storage device is determined.

[0014] According to one embodiment of this application, determining the item storage information of the storage device based on the first color depth data and the second color depth data includes:

[0015] Based on the first color depth data, determine the detection result of the first item;

[0016] Based on the second color depth data, the detection result of the second item is determined;

[0017] Based on the first item detection result and the second item detection result, the item storage information of the storage device is determined.

[0018] According to one embodiment of this application, the first item detection result includes item detection information and user limb information for operating the storage device, and the second item detection result includes item detection information and user limb information for operating the storage device.

[0019] According to one embodiment of this application, determining the item storage information of the storage device based on the first item detection result and the second item detection result includes:

[0020] If the item detection information of the user's limb range of motion in the first item detection result is the same as the item detection information of the user's limb range of motion in the second item detection result, the item storage information of the storage device remains unchanged.

[0021] According to one embodiment of this application, determining the item storage information of the storage device based on the first item detection result and the second item detection result includes:

[0022] If the item detection information of the user's limb range of motion in the first item detection result is different from the item detection information of the user's limb range of motion in the second item detection result, the item held by the user's limb in the first item detection result is entered into the item storage information of the storage device, and the item held by the user's limb in the second item detection result is removed from the item storage information of the storage device.

[0023] According to one embodiment of this application, determining the item storage information of the storage device based on the first item detection result and the second item detection result includes:

[0024] If the item detection information of the user's limb range of motion in the first item detection result is different from the item detection information of the user's limb range of motion in the second item detection result, and the user's limb is not holding an item in the second item detection result, then the item held by the user's limb in the first item detection result is recorded into the item storage information of the storage device.

[0025] According to one embodiment of this application, the target detection position includes a first step line and a second step line located opposite to each other at the opening, the first step line being closer to the storage space and the second step line being closer to the door.

[0026] Secondly, this application provides an item input device for a storage device, the device comprising:

[0027] The acquisition module is used to acquire target depth data of the target detection position of the storage device when the door of the storage device is open. The target detection position is set at the opening of the storage device, and the door is closably set at the opening.

[0028] The response module is used to collect color depth data of the storage space of the storage device in response to a step signal of the target depth data;

[0029] The processing module is used to determine the item storage information of the storage device based on the color depth data of the storage space.

[0030] According to the item entry device for storage equipment of this application, when the storage equipment is in the open state, the target depth data of the target detection position at the opening is collected for step detection. When the step signal is detected, the color depth data of the storage space is collected for item recognition, the items taken and placed by the user are recorded, and the item storage information of the storage equipment is obtained. This can realize seamless item entry, with small data volume and computational load, and high recognition accuracy.

[0031] Thirdly, this application provides a storage device, comprising:

[0032] A housing that defines a storage space and has an opening;

[0033] A door body, which is openable and closable and disposed within the opening;

[0034] A data acquisition device is disposed in the housing, and the data acquisition device is used to acquire depth data and color depth data;

[0035] The item entry device for the storage equipment described in the second aspect above is connected to the data acquisition device.

[0036] Fourthly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the item input method for the storage device as described in the first aspect above.

[0037] Fifthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the item entry method for the storage device as described in the first aspect above.

[0038] Sixthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the item entry method for the storage device as described in the first aspect above.

[0039] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0040] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0041] Figure 1 This is one of the flowcharts illustrating the method for entering items into a storage device provided in this application embodiment;

[0042] Figure 2 This is a schematic diagram of the structure of the item input device for the storage equipment provided in this application embodiment;

[0043] Figure 3 This is a second flowchart illustrating the method for entering items into a storage device provided in this application embodiment;

[0044] Figure 4 This is a schematic diagram of the process for step detection of the storage device provided in the embodiments of this application;

[0045] Figure 5 This is a schematic diagram of the process of item identification using the storage device provided in the embodiments of this application;

[0046] Figure 6 This is a schematic diagram showing the location of the step line setting for the storage device provided in the embodiments of this application;

[0047] Figure 7 This is one of the structural schematic diagrams of the storage device provided in the embodiments of this application;

[0048] Figure 8 This is a second schematic diagram of the structure of the storage device provided in the embodiments of this application;

[0049] Figure 9 This is the third structural schematic diagram of the storage device provided in the embodiments of this application;

[0050] Figure 10 This is the fourth structural schematic diagram of the storage device provided in the embodiments of this application;

[0051] Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0052] Figure label:

[0053] Housing 310, data acquisition device 320, drawer 330, shelf 340, door 350.

[0054] The first jump line is 410, and the second jump line is 420. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0056] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0057] It should be noted that the storage device in this embodiment can be understood as a refrigeration storage device in a broad sense, including but not limited to refrigerators, freezers, display cases, beverage cabinets, wine cabinets, refrigerated display cases, and refrigerated vending machines. The storage devices have diverse structural forms and a wide range of applications.

[0058] The following description, in conjunction with the accompanying drawings, details the item input method for storage devices, item input device for storage devices, storage devices, electronic devices, and readable storage media provided in this application, through specific embodiments and application scenarios.

[0059] The storage device in this application embodiment may include a housing 310, a door 350, and a data acquisition device 320, etc.

[0060] The housing 310 defines a storage space for storing items.

[0061] The housing 310 has an opening, and the door 350 is closable at the opening. When the door is open, the user can enter the storage space through the opening to retrieve or place items.

[0062] The data acquisition device 320 can be installed in the housing 310. The data acquisition device 320 is used to acquire depth data and color depth data.

[0063] Color depth data includes color data (also known as color image data) and depth data (also known as depth image data).

[0064] Understandably, for a given scenario, the data acquisition device 320 can collect both depth data and color depth data.

[0065] In actual implementation, the data acquisition device 320 can be an RGBD camera, which includes an RGB sensor and a depth sensor. The RGB sensor is used to acquire RGB images (i.e., color data), and the depth sensor is used to acquire depth images (i.e., depth data). When the RGB sensor and the depth sensor acquire an RGB image and a depth image simultaneously, the resulting RGB-D image pair is the color depth data.

[0066] The field of view of the data acquisition device 320 refers to the range of the scene that the data acquisition device 320 can capture. The field of view of the data acquisition device 320 can cover the edge of the storage space to fully collect the depth data and color depth data of the storage space.

[0067] For example, such as Figure 7 As shown, the data acquisition device 320 can be installed at the top of the housing 310 of the storage device, and the field of view of the data acquisition device 320 (shown by the dotted line in the figure) can cover the left and right edges of the shelf 340.

[0068] It should be noted that if the storage device has a pull-out drawer 330 or other storage components, the field of view of the data acquisition device 320 can cover the range of movement of the drawer 330 or other storage components in order to acquire relevant images of the items inside the storage components.

[0069] For example, such as Figure 8 As shown, the data acquisition device 320 can be installed on the top of the housing 310 of the storage device near the door 350. The field of view of the data acquisition device 320 (shown by the dotted line in the figure) can cover the rear edge of the shelf 340 near the inner liner, as well as the edge of the lower drawer 330 when it is pulled out.

[0070] This application provides a method for inputting items into a storage device, which enables seamless input of items with minimal data volume and computational load, and high recognition accuracy.

[0071] like Figure 1 As shown, the item entry method for this storage device includes steps 110, 120, and 130.

[0072] Step 110: With the door 350 of the storage device open, acquire the target depth data of the target detection position of the storage device.

[0073] The target detection location is set at the opening of the storage device.

[0074] Understandably, when the door of the storage device is 350 degrees open, users can take or put items in through the opening. By obtaining target depth data of the target detection position, the opening of the storage device can be monitored in real time, and the user's actions of taking or putting items in can be detected in a timely manner.

[0075] It should be noted that the target detection location is a small area at the opening, and for the same scene, the amount of depth data is less than the amount of color data. Obtaining the target depth data at the target detection location and determining whether to trigger the collection of color depth data can effectively reduce the amount of data collected and the amount of data processed subsequently.

[0076] In actual implementation, magnetic switches, mechanical switches, and other devices can be installed on the storage equipment to detect the opening and closing status of the door 350. If the door 350 is detected to be open, the data acquisition device 320 collects the target depth data of the target detection position.

[0077] Step 120: In response to the step signal of the target depth data, acquire the color depth data of the storage space of the storage device.

[0078] It should be noted that a step refers to a sudden change in the pattern of depth data. A step can reflect whether the pattern of depth data conforms to certain action characteristics. In the embodiments of this application, the step signal of depth data can refer to the change characteristics of depth data that conform to the user picking up or putting down items, thereby distinguishing whether there is user picking up or putting down items. In actual implementation, the degree to which the depth data conforms to the change characteristics of the user picking up or putting down items can be obtained by confidence calculation. The depth change characteristics brought about by the user picking up or putting down items are assigned a high confidence level. Based on the confidence level, it is determined whether the user picking up or putting down action has occurred.

[0079] In this embodiment, target depth data at the target detection location is acquired in real time for step detection. When a corresponding step signal appears in the target depth data, it may be that a user's limb enters the storage space to retrieve or place items, triggering the data acquisition device 320 to collect the color depth data of the storage space.

[0080] Step 130: Determine the item storage information of the storage device based on the color depth data of the storage space.

[0081] In this step, based on the color depth data collected in response to the step signal, the system detects items that the user's limbs enter and take out of the storage space, identifies the type and location of the items in the storage space, records the information of the items taken out and put in, and obtains the item storage information of the storage device.

[0082] In this embodiment, based on the color depth data of the storage space, the items stored in the entire storage space can be identified to obtain the item storage information of the storage device; alternatively, only the items retrieved or placed by the user's limbs in the storage space can be identified to update the item storage information of the storage device in real time.

[0083] In practice, an item information database can be set up. The item information database contains item storage information such as the type and location of items stored in the storage device. It collects color depth data in response to the step signal of the target depth data, identifies the items to be taken or placed based on the color depth data, and updates the item storage information in the item information database, which can effectively improve the efficiency of item entry.

[0084] Understandably, when recognizing objects based on color depth data, a detection box can be marked for each object, and the quantity information of each type of object can be obtained based on the quantity and type information of the detection boxes.

[0085] In related technologies, vision-based object entry requires continuous image acquisition and recognition by a camera, resulting in a large amount of data and computational load, which is not conducive to data transmission and real-time output of detection results. The hardware cost is also high. Some technologies that start image acquisition and recognition only under specific conditions, such as starting image acquisition and recognition after a door is opened, or performing human body part detection first and then performing image recognition after successful detection, still have the problem of data redundancy. Most of them require neural networks to recognize each frame of image, and the computational load is still very heavy.

[0086] In this embodiment, a small target detection position is set at the opening of the storage device. When the door 350 is opened, the target depth data of the target detection position is acquired in real time for step detection. When the target depth data shows a corresponding step signal, the color depth data of the storage space is triggered to be collected. Item recognition is performed on the color depth data collected in response to the step signal. There is no need to continuously collect data or recognize every frame of data. The lightweight triggering method can effectively reduce the amount of data collected and processed, which is beneficial for data transmission and real-time output of detection results. It also helps to reduce the hardware cost of the device. By using color depth data for item recognition and comprehensively utilizing color and depth information, the recognition accuracy of item entry can also be improved.

[0087] According to the item entry method for storage devices provided in this application embodiment, by collecting target depth data at the target detection position at the opening of the storage device when the door is open, step detection is performed. When a step signal is detected, the color depth data of the storage space is triggered to identify the item, and the items taken and placed by the user are recorded to obtain the item storage information of the storage device. This method can achieve seamless item entry with a small amount of data and computational load, and a high recognition accuracy.

[0088] In some embodiments, the target detection location includes a first step line 410 and a second step line 420 located opposite each other at the opening, with the first step line 410 near the storage space and the second step line 420 near the door 350.

[0089] Understandably, the target detection location can be adjusted according to the structure of the storage device.

[0090] For example, such as Figure 6 As shown, the target detection position is located at the front edge of the shelf 340 of the storage equipment. The first step line 410 and the second step line 420 are arranged opposite to each other and spaced apart. Figure 6 An upward arrow indicates the direction towards storage space. Figure 6 The downward arrow indicates the direction towards door 350.

[0091] like Figure 9As shown, the data acquisition device 320 is configured with the positions of the first step line 410 and the second step line 420, and the center line of the field of view of the data acquisition device 320 can coincide with the outer edge of the drawer 330.

[0092] The first step line 410 is located 5 pixels from the outer edge of the shelf 340 toward the door 350, and the second step line 420 is located 20 pixels from the outer edge of the shelf 340 toward the door 350. The first step line 410 and the second step line 420 are spaced 15 pixels apart.

[0093] In this embodiment, target depth data of the target detection location is obtained, namely, depth data at the first step line 410 and depth data at the second step line 420. Step detection is performed using the depth data of the first step line 410 and the second step line 420 to determine whether to trigger the collection of color depth data. This allows for lightweight real-time monitoring of the opening of the storage device and identification of the direction of the step signal, enabling timely perception of the user's actions of taking or placing items.

[0094] The first step line 410 is close to the storage space, and the second step line 420 is close to the door 350. When the second step line 420 is triggered before the first step line 410 (i.e., an object passes through the second step line 420 before the first step line 410), it indicates that the user's limbs have entered the storage space, triggering a forward step. When the first step line 410 is triggered before the second step line 420, it indicates that the user's limbs have moved out of the storage space, triggering a reverse step.

[0095] In actual execution, the confidence level corresponding to the first step line 410 can be calculated based on the depth data at the first step line 410, and the confidence level corresponding to the second step line 420 can be calculated based on the depth data at the second step line 420. The confidence level is then used to determine which step line is triggered first.

[0096] In the step triggering determination, the first step line 410 and the second step line 420 can be determined independently based on whether their confidence levels are greater than a threshold. If the confidence level of the second step line 420 is less than the threshold and the confidence level of the first step line 410 is greater than the threshold, the first step line 410 is triggered first, and it is determined to be a reverse step signal triggering. If the confidence level of the second step line 420 is greater than the threshold and the confidence level of the first step line 410 is less than the threshold, the second step line 420 is triggered first, and it is determined to be a positive step signal triggering.

[0097] In some embodiments, step 120, in response to a step signal in the target depth data, acquiring color depth data of the storage space of the storage device, may include:

[0098] In response to the first step signal of the target depth data, the first color depth data of the storage space is acquired;

[0099] In response to a second step signal in response to the target depth data, second color depth data of the storage space is acquired;

[0100] Step 130: Based on the color depth data of the storage space, determine the item storage information of the storage device, which may include:

[0101] Based on the first color depth data and the second color depth data, the item storage information of the storage device is determined.

[0102] In this embodiment, the direction of the second step signal is opposite to that of the first step signal. The first step signal can be a positive step, and the second step signal can be a negative step.

[0103] A first step signal indicating a positive step in the target depth data indicates that a user's limb has entered the storage space. A second step signal indicating a negative step in the target depth data indicates that a user's limb has moved out of the storage space. First color depth data is acquired in response to the positive step, and second color depth data is acquired in response to the negative step. Based on the first and second color depth data, the situation of a user's limb entering and leaving the storage space and taking or putting away items can be identified.

[0104] In related technologies, there is no judgment on the user's storage status. Users may change the storage location immediately after placing the item, resulting in low effectiveness of item location prediction.

[0105] In this embodiment, when the storage device is open, a step signal of the target depth data at the opening is detected in real time. In response to the first step signal, first color depth data is collected, and in response to the second step signal, second color depth data is collected. Based on the first color depth data and the second color depth data, the user's limbs can be identified during the process of entering and leaving the storage space to pick up and put down items, and the user's storage and retrieval status can be determined. Based on the user's storage and retrieval status, the effectiveness of item location prediction can be effectively improved, and the item storage location can be predicted more accurately.

[0106] In some embodiments, determining the item storage information of the storage device based on the first color depth data and the second color depth data may include:

[0107] Based on the first color depth data, determine the detection result of the first item;

[0108] The detection result of the second item is determined based on the second color depth data;

[0109] Based on the results of the first and second item inspections, the item storage information of the storage device is determined.

[0110] In this embodiment, when a first step signal of the target depth data is detected, the data acquisition device 320 is triggered to acquire first color depth data. Based on the first color depth data, item recognition is performed to obtain a first item detection result corresponding to the first color depth data. At the same time, the acquisition of target depth data is continued. When a second step signal of the target depth data is detected, the data acquisition device 320 is triggered to acquire second color depth data. Based on the second color depth data, item recognition is performed to obtain a second item detection result corresponding to the second color depth data. By combining the first item detection result and the second item detection result, the item storage information of the storage device is obtained.

[0111] The first item detection result may include item detection information and user limb information for operating the storage device, and the second item detection result may include item detection information and user limb information for operating the storage device.

[0112] It is understandable that the user's limbs operating the storage device are the user's limbs appearing at the opening of the storage device in the field of view of the data acquisition device 320. The user's limbs enter the storage device from the opening and move inside the storage device to complete the action of taking and putting away items.

[0113] In practice, an object recognition model can be used to detect targets in the first color depth data and identify the corresponding first object detection result. The first object detection result can include object detection information such as object type and object location, and can also include user limb information for operating the storage device.

[0114] An object recognition model can be used to perform target detection on the second color depth data and identify the corresponding second object detection result. The second object detection result can include object detection information such as object type and object location, and can also include user limb information of the user operating the storage device.

[0115] In this embodiment, the same object recognition model or detection network can be used to perform target detection on the first color depth data and the second color depth data respectively, so as to obtain the first object detection result and the second object detection result.

[0116] Understandably, the first and second item detection results can respectively represent the item status when the user's limbs enter the storage space and the item status when the user's limbs are removed from the storage space. By combining the first and second item detection results, the user's storage and retrieval status can be determined, the item location can be predicted more accurately, and highly accurate item storage information can be obtained.

[0117] It should be noted that, based on the first color depth data and the second color depth data, it is possible to detect only items within the range of the user's limb movements, which helps to reduce the computational load, improve recognition efficiency, and output more accurate detection results in real time.

[0118] In practice, the detection information in the first and second item detection results can be compared to detect whether the items within the user's limb movement range have changed, and to determine whether the user has picked up or put in the items.

[0119] It should be noted that the user's limb movement range refers to the area where the user's limbs appear in the item detection results. Items within the area where the user's limbs do not appear can be considered unchanged. By judging whether the item detection information within the user's limb movement range has changed, the user's access status can be accurately identified.

[0120] Understandably, based on the first color depth data and the second color depth data, it is also possible to detect items in the entire area of ​​the storage space. Combined with the judgment of the user's storage and retrieval status, it is possible to comprehensively and accurately identify information such as the type, quantity, and storage location of items stored in the storage device.

[0121] The embodiments of this application will be described in detail below from three different implementation perspectives.

[0122] 1. The user did not pick up or leave the item.

[0123] In some embodiments, determining the item storage information of the storage device based on the first item detection result and the second item detection result may include:

[0124] If the item detection information of the user's limb range of motion in the first item detection result is the same as the item detection information of the user's limb range of motion in the second item detection result, the item storage information of the storage device remains unchanged.

[0125] In this embodiment, when the item detection information within the user's limb movement range is the same in the first item detection result and the second item detection result, that is, the item type, quantity and position are unchanged, it indicates that the user has not picked up, placed or moved the item, and the item storage information of the storage device remains unchanged.

[0126] 2. The user takes out the item and puts it back in.

[0127] In some embodiments, determining the item storage information of the storage device based on the first item detection result and the second item detection result may include:

[0128] If the item detection information of the user's limb range of motion in the first item detection result is different from the item detection information of the user's limb range of motion in the second item detection result, the item held by the user's limb in the first item detection result is entered into the item storage information of the storage device, and the item held by the user's limb in the second item detection result is removed from the item storage information of the storage device.

[0129] In this embodiment, in response to the first color depth data acquired via a forward step, a first item detection result is determined. The item held by the user's limb in the first item detection result can be an item placed by the user in the storage device, and the item can be recorded in the item storage information of the storage device. In response to the second color depth data acquired via a reverse step, a second item detection result is determined. The item held by the user's limb in the second item detection result can be an item taken out by the user from the storage device, and the item can be removed from the item storage information of the storage device, and the item storage information of the storage device is updated.

[0130] Take a refrigerator as an example of a storage device.

[0131] The food database includes information such as the types and storage locations of food items A, B, and C in the refrigerator. If the first item detection result shows that the item held by the user's limbs is food item D, the type and storage location of food item D will be entered into the food database. If the second item detection result shows that the item held by the user's limbs is food item A, the type and storage location of food item A will be removed from the food database.

[0132] 3. The user puts in an item but does not take it out.

[0133] In some embodiments, determining the item storage information of the storage device based on the first item detection result and the second item detection result may include:

[0134] If the item detection information of the user's limb movement range in the first item detection result is different from the item detection information of the user's limb movement range in the second item detection result, and the user's limb is not holding an item in the second item detection result, then the item held by the user's limb in the first item detection result is recorded into the item storage information of the storage device.

[0135] In this embodiment, the item detection information of the user's limb movement range in the first item detection result and the second item detection result are different, and the user's limbs are not holding items in the second item detection result, indicating that the user only puts items into the storage device and does not take items out of the storage device. Therefore, the items held by the user's limbs in the first item detection result can be recorded into the item storage information of the storage device.

[0136] Take a refrigerator as an example of a storage device.

[0137] The food database includes information such as the types and storage locations of food items A, B, and C in the refrigerator. If the first item detection result shows that the user is holding food item D, and the second item detection result shows that the user is not holding any item, the information such as the type and storage location of food item D will be entered into the food database.

[0138] The item entry method for storage devices provided in this application embodiment can achieve seamless item entry with small data volume and computational load, and high recognition accuracy.

[0139] The following is a specific example.

[0140] The refrigerator's computer board stores a food database, which includes information such as the type, quantity, and storage location of the food in the refrigerator. The food database may also include information such as the expiration date and nutritional composition of the food.

[0141] like Figure 3 As shown, after the refrigerator is powered on, it performs status detection. When the refrigerator door 350 is detected to be open (in the open state), the target depth data at the first step line 410 and the second step line 420 at the refrigerator opening is acquired in real time. In response to the step signal of the target depth data, the color depth data of the storage space is collected, the type and location of the items are identified, the information of the items taken out and put in is recorded, and the food database information is updated.

[0142] like Figure 6 As shown, the first step line 410 and the second step line 420 are set on the outer edge of the shelf 340. The first step line 410 is set near the inner liner of the refrigerator, and the second step line 420 is set near the door 350 of the refrigerator.

[0143] When a positive step detection is performed and the first step signal corresponding to the positive step is triggered, the sensor of the data acquisition device 320 is triggered to acquire color depth data. The target detection is performed on the acquired first color depth data through the detection network (i.e., the object recognition model), the food and storage location are identified, and the recognition result 1 (i.e. the first item detection result) is output. Result 1 is temporarily stored.

[0144] At the same time, the data acquisition device 320 maintains the acquisition of target depth data at the first step line 410 and the second step line 420, performs reverse step detection, and when the second step signal corresponding to the reverse step is triggered, the sensor of the data acquisition device 320 is triggered again to acquire color depth data. The target detection is performed on the acquired second color depth data through the detection network, and the recognition result 2 (second item detection result) is output.

[0145] When result 2 is valid, the relationship between result 2 and result 1 is determined. If the ingredients detected by result 1 and result 2 are the same, the ingredient database is not updated, the item storage information remains unchanged, and the data acquisition device 320 continues to acquire the target depth data at the step line.

[0146] If the ingredients in Result 1 and Result 2 are different, the ingredients in Result 1 are entered, and the ingredients in Result 2 are removed from the ingredient library. The data acquisition device 320 continues to acquire the target depth data at the step line.

[0147] If result 2 detects a human upper limb (i.e., the user's limb) but not food, then the food from result 1 is entered, and the data acquisition device 320 continues to acquire target depth data at the step line until the refrigerator door 350 is closed.

[0148] In actual operation, when the door 350 is closed, the data acquisition device 320 can continuously collect step line data (at this time, the step line can be set at any position of the storage device where the movement of food needs to be detected, instead of at the opening). At this time, whether a positive step signal or a reverse step signal is triggered, the sensor of the data acquisition device 320 will be triggered to collect color depth data and detect it through the detection network. If the detection network detects food, it will be recorded in the abnormal food database, indicating that the food has moved while the refrigerator door is closed.

[0149] In this embodiment, users can be notified of abnormal food conditions and reminded of food falling through interactive methods such as APP, voice, and display screen. The above cycle continues in the closed state until the refrigerator door 350 is opened, and the item entry recognition process begins.

[0150] Understandably, users can also set refrigerator modes through interactive methods such as apps, voice commands, and displays. For example, in low-energy mode, the data acquisition device 320 can be stopped when the door is closed, the food movement reminder can be turned off, and only the food entry function when the door is open can be retained. Users can also actively update the food storage information in the food database through interactive methods such as apps, voice commands, and displays.

[0151] The above describes the overall technical solution of this application. By performing step detection on the target depth data at the target detection location, the color depth data of the storage space is triggered for item recognition, realizing seamless item entry, reducing data volume and computational load, and improving recognition accuracy.

[0152] To better understand the specific implementation methods of this application, the implementation steps will be described in detail below with reference to some embodiments. It should be noted that the specific steps described below are only one achievable embodiment, and their purpose is to more clearly illustrate the technical content of this application. Within the spirit and scope of this application, those skilled in the art can adopt other equivalent or alternative methods to achieve the same technical effect according to actual needs.

[0153] I. Step detection.

[0154] In some embodiments, after acquiring the target depth data but before responding to a step signal in response to the target depth data, the method may further include:

[0155] Based on target depth data from two consecutive frames, determine the step trigger condition.

[0156] Under the condition that the step triggering condition is met, a step signal is obtained to obtain the target depth data.

[0157] In this embodiment, a step trigger condition is determined based on two consecutive frames of target depth data. If the step trigger condition is met, a step signal of the target depth data is generated to trigger the acquisition of color depth data of the storage device. At the same time, the target depth data is continuously acquired for step detection. When the generation of the next step signal is detected, the acquisition of color depth data of the storage device is triggered again.

[0158] It should be noted that, based on two consecutive frames of target depth data, it is possible to analyze whether the target depth data undergoes sudden and significant changes within the two consecutive frames, with a relatively small amount of data collected and processed.

[0159] In some embodiments, after determining the step trigger condition based on target depth data from two consecutive frames, the following may be included:

[0160] If the target depth data of two consecutive frames does not meet the step trigger condition, clear one frame of target depth data, reacquire one frame of target depth data, and return to the step trigger condition judgment based on the target depth data of two consecutive frames.

[0161] In this embodiment, a step trigger condition is determined for two consecutive frames of target depth data. If the step trigger condition is not met, target depth data is collected again for step trigger condition determination.

[0162] It should be noted that for two consecutive frames of target depth data that have already undergone step trigger condition judgment, one frame of target depth data can be cleared, the other frame of target depth data can be retained, and a new frame of target depth data can be collected to obtain two new consecutive frames of target depth data, and the step trigger condition judgment can be performed again.

[0163] For example, each time one frame of target depth data is collected, two consecutive frames of target depth data are retained for step trigger condition judgment. The step trigger condition judgment is performed on the target depth data of the nth frame and the (n+1)th frame. If the step trigger condition is not met, the data of the nth frame is cleared, the data of the (n+2)th frame is collected, and the step trigger condition judgment is performed on the target depth data of the (n+1)th frame and the (n+2)th frame again.

[0164] Among them, the nth frame and the (n+1)th frame are two consecutive frames, and the (n+1)th frame and the (n+2)th frame are two consecutive frames.

[0165] In actual execution, a step trigger condition is determined for the target depth data of the nth frame and the (n+1)th frame. If the step trigger condition is met, the acquisition of color depth data is triggered, and the acquisition of the (n+2)th frame can continue. Then, the step trigger condition is determined for the target depth data of the (n+1)th frame and the (n+2)th frame.

[0166] It should be noted that when two consecutive time series meet the triggering conditions, the triggering of the latter time series can be ignored. For example, if the step triggering condition is met based on the nth frame and the (n+1)th frame, and the step triggering condition is also met on the (n+1)th frame and the (n+2)th frame, the step signals triggered by the (n+1)th frame and the (n+2)th frame can be ignored to eliminate logical errors and additional system consumption caused by multiple data collections due to a single item handling operation.

[0167] In some embodiments, determining the step trigger condition based on target depth data from two consecutive frames may include:

[0168] The target confidence score is obtained based on the target depth data of two consecutive frames.

[0169] The target confidence level is compared with the confidence threshold in the step trigger condition, and the step trigger condition is determined based on the comparison result.

[0170] It should be noted that confidence level can characterize the reliability and consistency of depth data over time. High confidence level regions may indicate a sudden change in depth values ​​that match the pattern of a user picking up or placing items. Based on the confidence level, step detection can be implemented to determine whether the step triggering conditions are met.

[0171] In this embodiment, the target confidence is calculated based on the target depth data of two consecutive frames. The target confidence is compared with the preset confidence threshold in the step trigger condition. Based on the level of confidence, it is detected whether there is a sudden change in the depth value pattern, and it is determined whether the step trigger condition is met.

[0172] The embodiments of this application will be specifically described below from the perspective of implementing step detection based on target depth data of the first step line 410 and the second step line 420.

[0173] It should be noted that the first step line 410 and the second step line 420 are set relative to each other.

[0174] In some embodiments, obtaining the target confidence score based on target depth data from two consecutive frames may include:

[0175] Based on the target depth data of two consecutive frames at the first step line 410, the first target confidence corresponding to the first step line 410 is obtained, and based on the target depth data of two consecutive frames at the second step line 420, the second target confidence corresponding to the second step line 420 is obtained.

[0176] Comparing the target confidence level with the confidence threshold in the step trigger condition, and determining whether the step trigger condition is met based on the comparison result, may include:

[0177] The confidence level of the first target is compared with the confidence threshold, and the confidence level of the second target is compared with the confidence threshold. The result of the comparison determines whether the step triggering condition is met.

[0178] In this embodiment, based on the target depth data of two consecutive frames at the first step line 410, the first target confidence level corresponding to the first step line 410 is calculated, and the first target confidence level is compared with the confidence level threshold to determine the level of the first step line 410 handling confidence.

[0179] Meanwhile, based on the target depth data of two consecutive frames at the second step line 420, the second target confidence level corresponding to the second step line 420 is calculated. The second target confidence level is compared with the confidence level threshold to determine the level of the second step line 420 handling confidence.

[0180] For the first step line 410 and the second step line 420 that are set relative to each other, the target confidence level corresponding to the first step line 410 and the second step line 420 is considered to determine whether the target depth data of two consecutive frames meets the step triggering condition, and the direction of the step signal can also be determined.

[0181] In some embodiments, comparing a first target confidence level with a confidence threshold and comparing a second target confidence level with a confidence threshold, and determining whether a step trigger condition is met based on the comparison result, may include:

[0182] If the confidence level of the first target is less than the confidence threshold and the confidence level of the second target is greater than the confidence threshold, the step triggering condition is met, and the first step signal is triggered.

[0183] In this embodiment, when the first target confidence level of the first step line 410 is less than the confidence level threshold and the second target confidence level of the second step line 420 is greater than the confidence level threshold, it indicates that the second step line 420 is triggered before the first step line 410, thus triggering the first step signal.

[0184] For example, such as Figure 6 As shown, the first step line 410 and the second step line 420, which are set opposite to each other, are located on the outer edge of the shelf 340 of the storage device. The first step line 410 is close to the storage space, and the second step line 420 is close to the door 350.

[0185] The confidence level of the first target of the first jump line 410 is less than the confidence threshold, so the first jump line 410 is a region with low confidence. The confidence level of the second target is greater than the confidence threshold, so the second jump line 420 is a region with high confidence. The second jump line 420 is triggered before the first jump line 410, indicating that the user's limb may have entered the storage space.

[0186] In other embodiments, comparing a first target confidence level with a confidence threshold and comparing a second target confidence level with a confidence threshold, and determining whether a step trigger condition is met based on the comparison result, may include:

[0187] When the confidence level of the first target is greater than the confidence threshold and the confidence level of the second target is less than the confidence threshold, the step triggering condition is met, and the second step signal is triggered. The direction of the second step signal is opposite to the direction of the first step signal.

[0188] In this embodiment, when the first target confidence of the first step line 410 is greater than the confidence threshold and the second target confidence of the second step line 420 is less than the confidence threshold, it indicates that the first step line 410 is triggered before the second step line 420, thus triggering the second step signal.

[0189] For example, such as Figure 6 As shown, the first step line 410 and the second step line 420, which are set opposite to each other, are located on the outer edge of the shelf 340 of the storage device. The first step line 410 is close to the storage space, and the second step line 420 is close to the door 350.

[0190] The confidence level of the first target of the first jump line 410 is greater than the confidence threshold, and the first jump line 410 is a region with high confidence. The confidence level of the second target is less than the confidence threshold, and the second jump line 420 is a region with low confidence. The first jump line 410 is triggered before the second jump line 420, indicating that the user's limb may have been moved out of the storage space.

[0191] The following is a specific example.

[0192] like Figure 4As shown, by setting the position of the step line and the confidence threshold T, a first step line 410 and a second step line 420 can be set on the outer edge of the storage equipment shelf 340. The first step line 410 and the second step line 420 are set opposite to each other, with the first step line 410 close to the storage space and the second step line 420 close to the door 350.

[0193] Start collecting step line data (target depth data of the step line). Read the step line data Dn of frame n and the step line data Dn+1 of frame n+1. The system retains only two frames of depth data at the same time. Perform step trigger condition judgment on the obtained two consecutive frames (frame n and frame n+1) of depth data.

[0194] If the condition is not met, the data of frame n is cleared and the data of frame n+2 is collected. The step trigger condition is judged based on the data of frame n+1 and frame n+2, and the loop continues until the condition is met.

[0195] If the condition is met, a step signal is triggered, and the loop continues to detect the triggering of the next step signal.

[0196] The step trigger condition judgment process is as follows:

[0197] Obtain the target depth data Dn in frame n and the target depth data Dn+1 in frame (n+1). Calculate the absolute value of the difference between the two frames to obtain the depth data difference Ds = |Dn+1 - Dn|, where Ds 上线 Ds represents the depth data difference corresponding to the first step line 410. 下线 This represents the depth data difference corresponding to the second step line 420.

[0198] For Ds 上线 and Ds 下线 Pattern recognition is performed separately, and the confidence level C is calculated. When the confidence level C of the first step line 410 is... Ds上线 The C value is less than the confidence threshold T, and the second step line is 420. Ds下线 When the value exceeds the confidence threshold T, the first step signal is triggered, indicating that a user's limb may have entered the storage space.

[0199] When the confidence level C of the first step line 410 Ds上线 C is greater than the confidence threshold T, and the second step line is 420. Ds下线 When the value is less than the confidence threshold T, a second step signal is triggered, indicating that a user's limb may have been removed from the storage space.

[0200] In this embodiment, step detection can be achieved by calculating the target depth data of two frames to determine whether the acquisition of color depth data is triggered. It has low memory usage, simple and efficient algorithm, and extremely low computational load. It can accurately and efficiently determine the process of picking up and putting down items. It does not require continuous reading of color depth data of each frame, nor does it require a large amount of computation based on neural networks on the obtained data, thus ensuring the compactness and efficiency of the system.

[0201] II. Item Identification.

[0202] In some embodiments, color depth data can be divided into color image data and depth image data. Determining the item storage information of the storage device based on the color depth data of the storage space may include:

[0203] Based on color image data and depth image data, the first image data features are obtained;

[0204] Based on the depth image data, the second image data features are obtained;

[0205] Based on the features of the first image data and the features of the second image data, the item type information and item location information of the storage device are obtained. The item type information and item location information are used to determine the item storage information.

[0206] Among them, color image data represents the color information in the scene. It can record the color and texture of the object through three channels: red (R), green (G), and blue (B), reflecting the visible optical characteristics of the object.

[0207] Depth image data represents the distance information of each pixel in the scene, records the spatial positional relationship between the object and the sensor, and reflects the three-dimensional structure of the object and its surrounding scene.

[0208] Based on color image data and depth image data, feature extraction is performed to obtain the first image data feature. The first image data feature may include the visible optical features and three-dimensional structural features of the object. The first image data feature belongs to the deep feature map and can be used to capture the global semantics of the scene.

[0209] In this embodiment, feature extraction is performed on the depth image data separately to obtain the second image data features, wherein the second image data features include the three-dimensional structural features of the object.

[0210] It should be noted that the second image data feature is a shallow feature map extracted from depth image data, which can capture subtle differences in depth values ​​and local structures in the scene.

[0211] In this embodiment, the first image data features and the second image data features can be fused. The first image data features are used to capture global semantics, and the second image data features are used to capture subtle differences in depth values ​​and local structures. This allows for accurate identification of items within the storage space, obtaining information on the type and location of items stored in the storage device.

[0212] In related technologies, there are two main methods for vision-based species and location detection. One method is to use visible light images to predict species and location, and depth images to help confirm the location. The other method is to extract RGB-D features by stitching together RGB images and depth images to predict species and location. Neither of these methods effectively utilizes depth information, resulting in low detection accuracy.

[0213] In this embodiment, a shortcut connection is designed in the network inference process. The depth image data is processed independently, and the shallow features (second image data features) of the depth image data are extracted separately. Then, these features are combined with the deep features (first image data features) extracted from the fused color image data and depth image data. This effectively utilizes the information contained in the depth image data and fully captures the depth, color, texture and other features of the items in the storage space. This can achieve higher detection and recognition accuracy while supporting the detection of the item's location.

[0214] By independently processing depth image data and extracting features from second image data, and by fusing color image data and depth image data to extract features from first image data, and by fusing shallow and deep features, depth information can be effectively utilized to accurately identify the type and location of objects. Location identification does not require additional gravity sensors or other devices, making the system simple and relatively low in cost.

[0215] In some embodiments, obtaining the first image data features based on color image data and depth image data may include:

[0216] The color image data and depth image data are stitched together to obtain the first fused image data;

[0217] Feature extraction is performed on the first fused image data to obtain the first image data features.

[0218] In this embodiment, color image data and depth image data are stitched together and fused to obtain a first fused image data. Then, feature extraction is performed on the first fused image data to obtain the deep feature of the first image data.

[0219] The first fused image data can have four channels, including three color channels (R, G, B) of the color image data, and the depth information of the depth image data as the fourth channel.

[0220] In practice, feature extraction is performed on the first fused image data obtained by stitching together color image data and depth image data. The extracted features of the first image data incorporate the three-dimensional structural features and visible optical features of the object, which helps to improve the target discrimination and target localization capabilities in the object recognition process and enhances robustness.

[0221] In some embodiments, the second image data features include at least one of the item depth detail features and the item region of interest.

[0222] Among them, the object depth detail feature is used to characterize the micro-texture, shape contour, three-dimensional structure and local geometric changes of the object surface. The object depth detail feature can also reflect the physical state and integrity of the object.

[0223] The Region of Interest (ROI) refers to a specific region in an image that may contain an object. It is the key area that the algorithm focuses on. Identifying the ROI can reduce the amount of computation and improve detection efficiency.

[0224] In practice, the region of interest for an object in the image can be determined by the depth heatmap corresponding to the depth image data. The region of interest can be a highly significant area in the depth heatmap, which has high attention value or importance.

[0225] In this embodiment, feature extraction is performed on the depth image data separately, and the resulting second image data features contain a large amount of effective shallow feature information such as object depth detail features and object region of interest.

[0226] In some embodiments, obtaining the item type information and item location information of the storage device based on the first image data features and the second image data features may include:

[0227] The first image data features and the second image data features are concatenated to obtain the second fused image data;

[0228] The second fused image data is decoupled and feature-enhanced to obtain target classification information, target location information, and regression of target detection boxes;

[0229] Based on the regression of target classification information, target location information, and target detection boxes, we obtain item type information and item location information.

[0230] In this embodiment, the first image data features and the second image data features are first spliced ​​and fused. The fused second image data is then decoupled and feature-enhanced, and the target classification information, target location information and target detection box regression of each item in the storage device are output respectively.

[0231] Among them, target classification information refers to the category to which the item belongs, target location information refers to the specific coordinates of the target within the item, and target detection box regression refers to predicting the bounding box of the item, which can accurately define the location and range of the item.

[0232] In practice, the second fused image data can be decoupled and feature enhanced by two-way decoupling and feature enhancement. The second fused image data is enhanced and decoupled by one way to output target classification information, and the second fused image data is enhanced and decoupled by another way to output target location information and target detection box regression, respectively.

[0233] In this embodiment, after obtaining the target classification information, target location information, and target detection box regression, the target classification information, target location information, and target detection box regression are summarized, and the item type information and item location information of each item in the storage device are output.

[0234] Understandably, for items within the storage space of a storage device, a target detection box can be marked for each item. Based on the number of target detection boxes and the corresponding target classification information, the quantity information of each type of item can be obtained.

[0235] In this embodiment, category and location prediction are integrated. By utilizing the strong correlation between the item location and the item detection results in the depth image data and color image data, the second fused image data is enhanced and decoupled. The extracted deep and shallow features are fully utilized, which effectively improves the accuracy of classification and location detection, has high data utilization, and also helps to improve the efficiency of item recognition.

[0236] In practice, the identification of items in storage devices can be achieved through an item identification model, which may include a backbone network, a first convolutional network, and a decoupled head structure.

[0237] In some embodiments, obtaining the first image data features based on color image data and depth image data may include:

[0238] Color image data and depth image data are input into the backbone network of the object recognition model to obtain the first image data features output by the backbone network.

[0239] Based on depth image data, the second image data features can be obtained, which may include:

[0240] The depth image data is input into the first convolutional network of the object recognition model to obtain the second image data features output by the first convolutional network;

[0241] Based on the features of the first image data and the features of the second image data, the information on the type and location of items in the storage device can be obtained, which may include:

[0242] The first image data features and the second image data features are input into the decoupled head structure of the item recognition model to obtain the item type information and item location information output by the decoupled head structure.

[0243] The object recognition model is trained based on an object image training set.

[0244] Understandably, an object recognition model is a model that performs object detection tasks. It can detect objects in storage space and output information such as the type, location, and quantity of the objects.

[0245] In this embodiment, the backbone network is an architecture used to extract first image data features from the first fused image data (which is obtained by stitching together color image data and depth image data). It can be built based on deep convolutional neural networks such as ResNet. The output first image data features can provide rich semantic and spatial information support for subsequent classification and regression tasks.

[0246] The first convolutional network is a convolutional layer used to extract features from depth image data. It can capture texture, shape, and spatial information, providing basic feature representations for target classification and localization.

[0247] It should be noted that the shallow features of depth image data carry a lot of useful information. The first convolutional network can extract the shallow features such as the depth details of the object and the region of interest of the object. There is no need to design a high-dimensional feature extraction structure for depth image data.

[0248] In this embodiment, the depth image data is isolated and a single-layer convolution is performed on the first convolutional network to extract shallow features (second image data features). These features are then fused with the first image data features extracted from the fusion of depth image data and color image data and input into the decoupled head structure. The independent network branches in the decoupled head structure process the category prediction and detection box regression respectively, which improves the learning efficiency of the object recognition model, reduces the number of parameters required by the object recognition model, and helps maintain the excellent inference performance of the object recognition model.

[0249] Understandably, the object recognition model is trained before deployment. First, the network structure of the object recognition model is built, a training set of object images is obtained (which can be labeled data), the object recognition model is trained and optimized using the object image training set, the parameters of the object recognition model are updated, and a trained object recognition model is obtained.

[0250] In some embodiments, the data in the training set of object images includes positive object samples and negative object samples. Positive object samples represent object samples within the user's limb range of motion, while negative object samples represent object samples outside the user's limb range of motion.

[0251] Positive samples refer to samples that contain the target, while negative samples refer to samples that do not contain the target.

[0252] In practice, training the object recognition model with positive and negative samples helps the model better distinguish between the detected target and the background, enhances the accuracy of target recognition, reduces false detections and false negatives, and improves the generalization performance and robustness of the object recognition model.

[0253] In this embodiment, items within the user's limb movement range are labeled as positive samples, and items outside the user's limb movement range are labeled as negative samples. This allows the item recognition model to focus on items picked up and placed by the user. For the item entry recognition task, items unrelated to the picking and placing state are not detected. This can effectively improve the detection efficiency of the item recognition model while ensuring the accuracy of item entry.

[0254] For example, during the data labeling process, the training set of images of items is obtained by photographing the user's picking up and putting down the items. Positive samples of items cover the items that the user touches during the picking up and putting down the items. Other items, such as stationary items on the shelf 340 and items that are not pulled out of the drawer 330, can be labeled as negative samples of items.

[0255] The following is a specific example.

[0256] The data acquisition device 320 continuously acquires target depth data and performs step detection. When a step signal in the target depth data is detected, the data acquisition device 320 is triggered to acquire color depth data, including color image data and depth image data, and inputs it into the object recognition model for detection.

[0257] like Figure 5 As shown, the input data is divided into depth images and color optical images (i.e. color image data). The two sets of data are first stitched together to obtain the first fused image data. The first fused image data is then input into the backbone network for feature extraction to extract the deep features of the fused data.

[0258] Meanwhile, the depth image data is independently subjected to single-layer convolution operations, and the depth image data is input into the first convolutional network to extract shallow features of the depth data.

[0259] The first image data features output by the backbone network and the second image data features output by the first convolutional network are concatenated by channels. After further fast feature fusion, the resulting second fused image data is enhanced and input into the decoupling head structure for decoupling.

[0260] In the decoupled head structure, one path outputs the target type classification (i.e., target classification information) after feature enhancement, while the other path continues to decouple after feature enhancement, outputting the target location classification and the target detection box regression respectively.

[0261] It should be noted that the object recognition model can focus on the items that the user picks up and puts down. In addition to the classification of items, the object recognition model can also include the classification of the user's limbs.

[0262] The target locations can be categorized according to different areas of the storage space, for example, such as... Figure 10 As shown, the target location is classified into 9 categories: A, B, C, D, E, F, G, H, and I, which indicate different locations in the shelf 340 area. Below the shelf 340 is the drawer 330.

[0263] The regression of the detection frame is used to determine the position and quantity of items and user limbs in the field of view of the data acquisition device 320. It can be used to capture data of the item area for archiving or sharing with the user, as well as to determine the number of items entered into the system.

[0264] The final result (including item type information and item location information) is output by summarizing the target type classification, target location classification, and target detection box regression.

[0265] In this embodiment, a shortcut connection is designed in the network inference process. The depth image data is isolated and a single-layer convolution is performed through the first convolutional network to extract shallow features. Then, it is spliced ​​and fused with the deep features extracted from the first fused image data to improve the learning efficiency of the network, reduce the number of parameters required by the network, and maintain the excellent inference performance of the object recognition model.

[0266] Furthermore, based on the strong correlation between the object location and the object detection results in the depth image data and color image data, the classification prediction and location prediction are integrated, and a decoupled head structure is designed to fully and efficiently utilize the relevant feature information of the second fused image data. The algorithm has a compact structure, high data utilization, and low repetition rate, which can significantly improve the system's operating efficiency.

[0267] It should be noted again that the step detection and object recognition embodiments described above are merely feasible embodiments, intended to more clearly illustrate the technical content of this application. Within the spirit and scope of this application, those skilled in the art can adopt other equivalent or alternative methods to achieve the same technical effect according to actual needs.

[0268] The method for entering items into a storage device provided in this application can be executed by an item entry device for the storage device. This application uses an item entry device for the storage device as an example to illustrate the item entry device for the storage device provided in this application.

[0269] This application also provides an item input device for storage equipment.

[0270] like Figure 2 As shown, the item input device of the storage equipment includes:

[0271] The acquisition module 210 is used to acquire target depth data of the target detection position of the storage device when the door 350 of the storage device is open. The target detection position is set at the opening of the storage device, and the door 350 is closable at the opening.

[0272] Response module 220 is used to collect color depth data of the storage space of the storage device in response to a step signal of the target depth data;

[0273] The processing module 230 is used to determine the item storage information of the storage device based on the color depth data of the storage space.

[0274] According to the embodiment of this application, the storage device for recording items can perform step detection by collecting target depth data at the target detection position at the opening when the storage device is open. When a step signal is detected, it triggers the collection of color depth data of the storage space for item recognition, records the items taken and placed by the user, and obtains the item storage information of the storage device. This can achieve seamless item recording with a small amount of data and computational load, and high recognition accuracy.

[0275] In some embodiments, the response module 220 is configured to acquire first color depth data of the storage space in response to a first step signal of the target depth data;

[0276] The response module 220 is used to collect the second color depth data of the storage space in response to the second step signal of the target depth data. The direction of the second step signal is opposite to the direction of the first step signal.

[0277] The processing module 230 is used to determine the item storage information of the storage device based on the first color depth data and the second color depth data.

[0278] In some embodiments, the processing module 230 is configured to determine the detection result of the first item based on the first color depth data;

[0279] The detection result of the second item is determined based on the second color depth data;

[0280] Based on the results of the first and second item inspections, the item storage information of the storage device is determined.

[0281] In some embodiments, the first item detection result includes item detection information and user limb information for operating the storage device, and the second item detection result includes item detection information and user limb information for operating the storage device.

[0282] In some embodiments, the processing module 230 is configured to keep the item storage information of the storage device unchanged when the item detection information of the user's limb range of motion in the first item detection result is the same as the item detection information of the user's limb range of motion in the second item detection result.

[0283] In some embodiments, the processing module 230 is configured to, when the item detection information of the user's limb range of motion in the first item detection result is different from the item detection information of the user's limb range of motion in the second item detection result, record the item held by the user's limb in the first item detection result into the item storage information of the storage device, and remove the item held by the user's limb in the second item detection result from the item storage information of the storage device.

[0284] In some embodiments, the processing module 230 is used to record the item held by the user's limbs in the first item detection result into the item storage information of the storage device when it is determined that the item detection information of the user's limbs' range of motion in the first item detection result is different from the item detection information of the user's limbs' range of motion in the second item detection result, and the user's limbs are not holding an item in the second item detection result.

[0285] In some embodiments, the target detection location includes a first step line 410 and a second step line 420 located opposite each other at the opening, with the first step line 410 near the storage space and the second step line 420 near the door 350.

[0286] The item entry device for storage equipment provided in this application embodiment can realize all the processes implemented in the above-described item entry method embodiment for storage equipment. To avoid repetition, it will not be described again here.

[0287] This application also provides a storage device.

[0288] like Figure 7 As shown, the storage device includes a housing 310, a door 350, and a data acquisition device 320.

[0289] The housing 310 defines a storage space and has an opening. A door 350 is closable at the opening. A data acquisition device 320 is disposed in the housing 310 and is used to acquire depth data and color depth data.

[0290] The storage device also includes an item entry device as described above, which is connected to the data acquisition device 320.

[0291] According to the storage device provided in the embodiments of this application, by collecting target depth data at the target detection position at the opening when the storage device is in the open state, step detection is performed. When a step signal is detected, the color depth data of the storage space is triggered to identify the item, and the items taken and placed by the user are recorded to obtain the item storage information of the storage device. This can achieve seamless item entry, with a small amount of data and computational load, and a high recognition accuracy.

[0292] In some embodiments, such as Figure 11 As shown, this application embodiment also provides an electronic device 1100, including a processor 1101, a memory 1102, and a computer program stored on the memory 1102 and executable on the processor 1101. When the program is executed by the processor 1101, it implements the various processes of the above-described storage device item entry method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0293] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0294] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described method for entering items into the storage device and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0295] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0296] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for inputting items into a storage device.

[0297] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0298] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described storage device item entry method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0299] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0300] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0301] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0302] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0303] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for inputting items into a storage device, characterized in that, include: With the door of the storage device open, target depth data of the target detection position of the storage device is acquired. The target detection position is set at the opening of the storage device, and the door is closable at the opening. In response to a step signal in the target depth data, color depth data of the storage space of the storage device is acquired; Based on the color depth data of the storage space, the item storage information of the storage device is determined.

2. The method for entering items into a storage device according to claim 1, characterized in that, The step signal in response to the target depth data, acquiring color depth data of the storage space of the storage device, includes: In response to a first step signal of the target depth data, first color depth data of the storage space is acquired; In response to a second step signal of the target depth data, second color depth data of the storage space is acquired, wherein the direction of the second step signal is opposite to the direction of the first step signal; The process of determining the item storage information of the storage device based on the color depth data of the storage space includes: Based on the first color depth data and the second color depth data, the item storage information of the storage device is determined.

3. The method for inputting items into a storage device according to claim 2, characterized in that, The step of determining the item storage information of the storage device based on the first color depth data and the second color depth data includes: Based on the first color depth data, determine the detection result of the first item; Based on the second color depth data, the detection result of the second item is determined; Based on the first item detection result and the second item detection result, the item storage information of the storage device is determined.

4. The method for inputting items into a storage device according to claim 3, characterized in that, The first item detection result includes item detection information and user limb information for operating the storage device; the second item detection result includes item detection information and user limb information for operating the storage device.

5. The method for inputting items into a storage device according to claim 4, characterized in that, The step of determining the item storage information of the storage device based on the first item detection result and the second item detection result includes: If the item detection information of the user's limb range of motion in the first item detection result is the same as the item detection information of the user's limb range of motion in the second item detection result, the item storage information of the storage device remains unchanged.

6. The method for inputting items into a storage device according to claim 4, characterized in that, The step of determining the item storage information of the storage device based on the first item detection result and the second item detection result includes: If the item detection information of the user's limb range of motion in the first item detection result is different from the item detection information of the user's limb range of motion in the second item detection result, the item held by the user's limb in the first item detection result is entered into the item storage information of the storage device, and the item held by the user's limb in the second item detection result is removed from the item storage information of the storage device.

7. The method for inputting items into a storage device according to claim 4, characterized in that, The step of determining the item storage information of the storage device based on the first item detection result and the second item detection result includes: If the item detection information of the user's limb range of motion in the first item detection result is different from the item detection information of the user's limb range of motion in the second item detection result, and the user's limb is not holding an item in the second item detection result, then the item held by the user's limb in the first item detection result is recorded into the item storage information of the storage device.

8. The method for inputting items into a storage device according to any one of claims 1-7, characterized in that, The target detection position includes a first step line and a second step line located opposite each other at the opening, with the first step line closer to the storage space and the second step line closer to the door.

9. An item input device for a storage device, characterized in that, include: The acquisition module is used to acquire target depth data of the target detection position of the storage device when the door of the storage device is open. The target detection position is set at the opening of the storage device, and the door is closably set at the opening. The response module is used to collect color depth data of the storage space of the storage device in response to a step signal of the target depth data; The processing module is used to determine the item storage information of the storage device based on the color depth data of the storage space.

10. A storage device, characterized in that, include: A housing that defines a storage space and has an opening; A door body, which is openable and closable and disposed within the opening; A data acquisition device is disposed in the housing, and the data acquisition device is used to acquire depth data and color depth data; The item entry device for the storage equipment as described in claim 9, wherein the item entry device is connected to the data acquisition device.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the item entry method for the storage device as described in any one of claims 1-8.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the item entry method for the storage device as described in any one of claims 1-8.

Images