Data acquisition method, data acquisition device and storage apparatus

Abstract

The application discloses a data acquisition method, a data acquisition device and a storage equipment, and belongs to the technical field of the storage equipment. The method comprises the following steps: acquiring target depth data of the storage equipment; and in response to a step signal of the target depth data, color depth data of a storage space of the storage equipment is acquired, and the color depth data is used for object identification of the storage equipment. The method can reduce the amount of data required for identifying objects and the computing load by acquiring the target depth data to perform step detection, triggering the acquisition of the color depth data of the storage equipment for object identification when the target depth data has a corresponding step signal, and helping to reduce the hardware cost of the equipment.

Description

Technical Field

[0001] This application belongs to the field of storage equipment technology, and particularly relates to a data acquisition method, a data acquisition device, and a storage device. Background Technology

[0002] With the development of vision technology, more and more storage devices are equipped with item recognition functions, automatically identifying items and providing intelligent services to users. However, existing vision solutions for storage devices involve large amounts of data collection and computational loads, which is detrimental to data transmission and real-time result output, increasing the cost of item recognition in storage devices. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a data acquisition method, a data acquisition device, and a storage device, which can reduce the amount of data and the computational load, thereby helping to reduce costs.

[0004] Firstly, this application provides a data acquisition method, which includes:

[0005] Acquire target depth data for storage devices;

[0006] In response to a step signal in the target depth data, color depth data of the storage space of the storage device is acquired, and the color depth data is used for item identification of the storage device.

[0007] According to the data acquisition method of this application, step detection is performed by acquiring target depth data. When a corresponding step signal appears in the target depth data, the color depth data of the storage device is triggered for item recognition. This can reduce the amount of data and computational load required for item recognition, and help reduce the hardware cost of the device.

[0008] According to one embodiment of this application, prior to the step signal in response to the target depth data, the method includes:

[0009] Based on the target depth data of two consecutive frames, a step trigger condition is determined;

[0010] Under the condition that the step triggering condition is met, a step signal of the target depth data is obtained.

[0011] According to one embodiment of this application, the step trigger condition determination based on the target depth data of two consecutive frames includes:

[0012] The target confidence level is obtained based on the target depth data of two consecutive frames;

[0013] The target confidence level is compared with the confidence threshold in the step trigger condition, and the step trigger condition is determined based on the comparison result.

[0014] According to one embodiment of this application, obtaining the target confidence score based on the target depth data of two consecutive frames includes:

[0015] Based on the target depth data of two consecutive frames at the first step line, a first target confidence level corresponding to the first step line is obtained, and based on the target depth data of two consecutive frames at the second step line, a second target confidence level corresponding to the second step line is obtained, wherein the first step line and the second step line are set relative to each other.

[0016] The step of comparing the target confidence level with the confidence threshold in the step trigger condition, and determining whether the step trigger condition is met based on the comparison result, includes:

[0017] The first target confidence level is compared with the confidence threshold, and the second target confidence level is compared with the confidence threshold. Based on the comparison result, it is determined whether the step triggering condition is met.

[0018] According to one embodiment of this application, comparing the first target confidence level with the confidence threshold, and comparing the second target confidence level with the confidence threshold, and determining whether the step triggering condition is met based on the comparison result, includes:

[0019] If the confidence level of the first target is less than the confidence threshold and the confidence level of the second target is greater than the confidence threshold, the step triggering condition is met, and the first step signal is triggered.

[0020] Alternatively, if the confidence level of the first target is greater than the confidence threshold and the confidence level of the second target is less than the confidence threshold, the step triggering condition is met, and a second step signal is triggered, wherein the direction of the second step signal is opposite to the direction of the first step signal.

[0021] According to one embodiment of this application, the target confidence level is determined through the following steps:

[0022] The depth data difference is obtained based on the target depth data of two consecutive frames;

[0023] Based on the depth data difference, the step coefficient and the accumulation coefficient are determined;

[0024] The target confidence level is determined based on the step coefficient and the accumulation coefficient.

[0025] According to one embodiment of this application, after determining the step trigger condition based on the target depth data of two consecutive frames, the method further includes:

[0026] If the target depth data in two consecutive frames does not meet the step trigger condition, clear one frame of target depth data, reacquire one frame of target depth data, and return to the step trigger condition judgment step based on the target depth data in two consecutive frames.

[0027] According to one embodiment of this application, obtaining the target depth data of the storage device includes:

[0028] Obtain the target depth data at the target detection location of the storage device.

[0029] Secondly, this application provides a data acquisition device, which includes:

[0030] The acquisition module is used to acquire the target depth data of the storage device;

[0031] A response module is used to collect color depth data of the storage space of the storage device in response to a step signal of the target depth data, and the color depth data is used for item identification of the storage device.

[0032] According to the data acquisition device of this application, step detection is performed by acquiring target depth data. When a corresponding step signal appears in the target depth data, the color depth data of the storage device is triggered for item recognition. This can reduce the amount of data and computational load required for item recognition and help reduce the hardware cost of the device.

[0033] Thirdly, this application provides a storage device, comprising:

[0034] A housing that defines a storage space;

[0035] A data acquisition device is disposed in the housing, the data acquisition device is used to acquire depth data and color depth data, and the data acquisition device is used to perform the data acquisition method as described in the first aspect above.

[0036] Fourthly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data acquisition method as described in the first aspect above.

[0037] Fifthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data acquisition method as described in the first aspect above.

[0038] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the data acquisition method as described in the first aspect above.

[0039] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0040] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0041] Figure 1 This is one of the flowcharts illustrating the data acquisition method provided in the embodiments of this application;

[0042] Figure 2 This is a schematic diagram of the structure of the data acquisition device provided in the embodiments of this application;

[0043] Figure 3 This is a schematic diagram of the process of entering items into the storage device provided in this application embodiment;

[0044] Figure 4 This is a second schematic flowchart of the data acquisition method provided in the embodiments of this application;

[0045] Figure 5 This is the third flowchart illustrating the data acquisition method provided in the embodiments of this application;

[0046] Figure 6 This is a schematic diagram showing the location of the step line setting for the storage device provided in the embodiments of this application;

[0047] Figure 7 This is one of the structural schematic diagrams of the storage device provided in the embodiments of this application;

[0048] Figure 8 This is a second schematic diagram of the structure of the storage device provided in the embodiments of this application;

[0049] Figure 9 This is the third structural schematic diagram of the storage device provided in the embodiments of this application;

[0050] Figure 10 This is the fourth structural schematic diagram of the storage device provided in the embodiments of this application;

[0051] Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0052] Figure label:

[0053] Housing 310, data acquisition device 320, drawer 330, shelf 340, door 350.

[0054] The first jump line is 410, and the second jump line is 420. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0056] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0057] It should be noted that the storage device in this embodiment can be understood as a refrigeration storage device in a broad sense, including but not limited to refrigerators, freezers, display cases, beverage cabinets, wine cabinets, refrigerated display cases, and refrigerated vending machines. The storage devices have diverse structural forms and a wide range of applications.

[0058] The data acquisition method, data acquisition device, storage device, electronic device, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0059] The storage device in this application embodiment may include a housing 310, a door 350, and a data acquisition device 320.

[0060] The housing 310 defines a storage space for storing items.

[0061] The housing 310 has an opening, and the door 350 is closable at the opening. When the door is open, the user can enter the storage space through the opening to retrieve or place items.

[0062] The data acquisition device 320 can be installed in the housing 310. The data acquisition device 320 is used to acquire depth data and color depth data.

[0063] Color depth data includes color data (also known as color image data) and depth data (also known as depth image data).

[0064] Understandably, for a given scenario, the data acquisition device 320 can collect both depth data and color depth data.

[0065] In actual implementation, the data acquisition device 320 can be an RGBD camera, which includes an RGB sensor and a depth sensor. The RGB sensor is used to acquire RGB images (i.e., color data), and the depth sensor is used to acquire depth images (i.e., depth data). When the RGB sensor and the depth sensor acquire an RGB image and a depth image simultaneously, the resulting RGB-D image pair is the color depth data.

[0066] The field of view of the data acquisition device 320 refers to the range of the scene that the data acquisition device 320 can capture. The field of view of the data acquisition device 320 can cover the edge of the storage space to fully collect the depth data and color depth data of the storage space.

[0067] For example, such as Figure 7 As shown, the data acquisition device 320 can be installed at the top of the housing 310 of the storage device, and the field of view of the data acquisition device 320 (shown by the dotted line in the figure) can cover the left and right edges of the shelf 340.

[0068] It should be noted that if the storage device has a pull-out drawer 330 or other storage components, the field of view of the data acquisition device 320 can cover the range of movement of the drawer 330 or other storage components in order to acquire relevant images of the items inside the storage components.

[0069] For example, such as Figure 8 As shown, the data acquisition device 320 can be installed on the top of the housing 310 of the storage device near the door 350. The field of view of the data acquisition device 320 (shown by the dotted line in the figure) can cover the rear edge of the shelf 340 near the inner liner, as well as the edge of the lower drawer 330 when it is pulled out.

[0070] This application provides a data acquisition method applied to a storage device. The method triggers the data acquisition device 320 of the storage device to collect data for identifying items in the storage device. This method can reduce the amount of data and computational load required to identify items, and helps to reduce the hardware cost of the device.

[0071] like Figure 1 As shown, the data acquisition method includes steps 110 and 120.

[0072] Step 110: Obtain the target depth data of the storage device.

[0073] In this step, target depth data is collected by data acquisition device 320. The target depth data at the opening of the storage device can be collected when the door 350 is open, or the target depth data inside the storage device can be collected when the door 350 is closed.

[0074] Understandably, for the same scene, the amount of depth data is less than the amount of color data. Collecting target depth data and determining whether to trigger the collection of color depth data can effectively reduce the amount of data collected and the amount of data processed subsequently.

[0075] Step 120: In response to the step signal of the target depth data, acquire the color depth data of the storage space of the storage device.

[0076] Among them, color depth data is used for item identification in storage devices. It can identify the type and location of items stored in the storage space of the storage device.

[0077] It should be noted that a step refers to a sudden change in the pattern of depth data, which can reflect whether the pattern of depth data conforms to certain action characteristics. In actual execution, the degree to which depth data conforms to the change characteristics when a certain action occurs can be obtained by confidence calculation. The depth change characteristics brought about by a certain action are assigned a certain confidence level, and the action is judged based on the confidence level.

[0078] In some embodiments, the step signal of depth data can refer to the change characteristics of depth data that are consistent with the user picking up or placing items, thereby distinguishing whether there is user picking up or placing items. The depth change characteristics caused by the user picking up or placing items are assigned a high confidence level, and the user picking up or placing actions are determined based on the confidence level.

[0079] In this embodiment, target depth data is acquired in real time for step detection. If the door 350 of the storage device is open, a corresponding step signal is detected in the target depth data at the opening of the storage device. This may indicate that a user's limbs have entered the storage space to retrieve or place items. This triggers the data acquisition device 320 to collect the color depth data of the storage space. Based on the color depth data, the user's retrieval of items can be identified, enabling seamless item entry.

[0080] In other embodiments, the step signal of the depth data can indicate that the depth data has changed in a way that is consistent with the characteristics of an object moving, thereby distinguishing whether there is an activity of object movement. The depth change characteristics caused by the object movement are assigned a high confidence level, and the object movement is determined based on the confidence level.

[0081] In this embodiment, if the door 350 of the storage device is closed, and a step signal is detected corresponding to the target depth data at a certain point in the storage space, the items in the storage space may have moved. This triggers the data acquisition device 320 to collect the color depth data of the storage space. Based on the color depth data, the movement of the items can be identified, and corresponding prompts can be output to alert the user that the items in the storage device have moved. Related technologies require continuous image acquisition and recognition by a camera, resulting in a large data volume and computational load, which is not conducive to data transmission and real-time output of detection results, and also incurs high hardware costs.

[0082] In this embodiment, target depth data is acquired in real time for step detection. When a corresponding step signal appears in the target depth data, the color depth data of the storage device is triggered to be collected. The amount of depth data is less than that of color data. The step detection data of depth data is simple to process and has a small computational load, which can effectively reduce the amount of data collected and processed, which is beneficial for data transmission and real-time output of detection results, and can also effectively reduce hardware costs.

[0083] According to the data acquisition method provided in the embodiments of this application, step detection is performed by acquiring target depth data. When a corresponding step signal appears in the target depth data, the color depth data of the storage device is triggered for item recognition. This can reduce the amount of data and computational load required to recognize items, and help reduce the hardware cost of the device.

[0084] In some embodiments, step 110, obtaining target depth data of the storage device, may include:

[0085] Obtain target depth data of the target detection location of the storage device.

[0086] It should be noted that the target detection location is a small area of ​​the storage device. Obtaining the target depth data at the target detection location and performing step detection can further reduce the amount of data collected and the amount of data processed subsequently.

[0087] In practice, the target detection location can be adjusted according to different object recognition tasks.

[0088] For example, in the task of recognizing items, the target detection location can be a small area at the opening of the storage device. The user's limbs enter the storage device through the opening to take or put away items. By using the target depth data of the target detection location, the user's taking or putting away of items can be monitored in a lightweight and real-time manner.

[0089] For example, in the task of recognizing the movement of objects, the target detection location can be a small area within the storage space of the storage device where objects are prone to movement. By using the target depth data of the target detection location, the movement or falling of objects within the storage space can be monitored in a lightweight and real-time manner.

[0090] In some embodiments, the target detection location may include a first step line 410 and a second step line 420 that are set relative to each other.

[0091] In this embodiment, the direction of the step signal can be identified using the target depth data of the first step line 410 and the second step line 420.

[0092] Understandably, the target detection location can be adjusted based on the structure of the storage device and the item recognition task.

[0093] In some embodiments, for the object entry recognition task, the target detection position may include a first step line 410 and a second step line 420 located opposite each other at the opening, with the first step line 410 close to the storage space and the second step line 420 close to the door 350.

[0094] For example, such as Figure 6 As shown, the target detection position is located at the front edge of the shelf 340 of the storage equipment. The first step line 410 and the second step line 420 are arranged opposite to each other and spaced apart. Figure 6 An upward arrow indicates the direction towards storage space. Figure 6 The downward arrow indicates the direction towards door 350.

[0095] like Figure 9 As shown, the data acquisition device 320 is configured with the positions of the first step line 410 and the second step line 420, and the center line of the field of view of the data acquisition device 320 can coincide with the outer edge of the drawer 330.

[0096] The first step line 410 is located 5 pixels from the outer edge of the shelf 340 toward the door 350, and the second step line 420 is located 20 pixels from the outer edge of the shelf 340 toward the door 350. The first step line 410 and the second step line 420 are spaced 15 pixels apart.

[0097] In this embodiment, target depth data of the target detection location is obtained, namely, depth data at the first step line 410 and depth data at the second step line 420. Step detection is performed using the depth data of the first step line 410 and the second step line 420 to determine whether to trigger the collection of color depth data. This allows for lightweight real-time monitoring of the opening of the storage device and identification of the direction of the step signal, enabling timely perception of the user's actions of taking or placing items.

[0098] The first step line 410 is close to the storage space, and the second step line 420 is close to the door 350. When the second step line 420 is triggered before the first step line 410 (i.e., an object passes through the second step line 420 before the first step line 410), it indicates that the user's limbs have entered the storage space, triggering a forward step. When the first step line 410 is triggered before the second step line 420, it indicates that the user's limbs have moved out of the storage space, triggering a reverse step.

[0099] In actual execution, the confidence level corresponding to the first step line 410 can be calculated based on the depth data at the first step line 410, and the confidence level corresponding to the second step line 420 can be calculated based on the depth data at the second step line 420. The step line that is triggered first can be determined based on the confidence level.

[0100] In the step triggering determination, the first step line 410 and the second step line 420 can be determined independently based on whether their confidence levels are greater than a threshold. If the confidence level of the second step line 420 is less than the threshold and the confidence level of the first step line 410 is greater than the threshold, the first step line 410 is triggered first, and it is determined to be a reverse step signal triggering. If the confidence level of the second step line 420 is greater than the threshold and the confidence level of the first step line 410 is less than the threshold, the second step line 420 is triggered first, and it is determined to be a positive step signal triggering.

[0101] In some embodiments, after step 110, acquiring the target depth data of the storage device, and before step 120, responding to a step signal in response to the target depth data, the data acquisition method may further include:

[0102] Based on target depth data from two consecutive frames, determine the step trigger condition.

[0103] Under the condition that the step triggering condition is met, a step signal is obtained to obtain the target depth data.

[0104] In this embodiment, a step trigger condition is determined based on two consecutive frames of target depth data. If the step trigger condition is met, a step signal of the target depth data is generated to trigger the acquisition of color depth data of the storage device. At the same time, the target depth data is continuously acquired for step detection. When the generation of the next step signal is detected, the acquisition of color depth data of the storage device is triggered again.

[0105] It should be noted that, based on two consecutive frames of target depth data, it is possible to analyze whether the target depth data undergoes sudden and significant changes within the two consecutive frames, with a relatively small amount of data collected and processed.

[0106] In some embodiments, after determining the step trigger condition based on target depth data from two consecutive frames, the data acquisition method may include:

[0107] If the target depth data of two consecutive frames does not meet the step trigger condition, clear one frame of target depth data, reacquire one frame of target depth data, and return to the step trigger condition judgment based on the target depth data of two consecutive frames.

[0108] In this embodiment, a step trigger condition is determined for two consecutive frames of target depth data. If the step trigger condition is not met, target depth data is collected again for step trigger condition determination.

[0109] It should be noted that for two consecutive frames of target depth data that have already undergone step trigger condition judgment, one frame of target depth data can be cleared, the other frame of target depth data can be retained, and a new frame of target depth data can be collected to obtain two new consecutive frames of target depth data, and the step trigger condition judgment can be performed again.

[0110] For example, each time one frame of target depth data is collected, two consecutive frames of target depth data are retained for step trigger condition judgment. The step trigger condition judgment is performed on the target depth data of the nth frame and the (n+1)th frame. If the step trigger condition is not met, the data of the nth frame is cleared, the data of the (n+2)th frame is collected, and the step trigger condition judgment is performed on the target depth data of the (n+1)th frame and the (n+2)th frame again.

[0111] Among them, the nth frame and the (n+1)th frame are two consecutive frames, and the (n+1)th frame and the (n+2)th frame are two consecutive frames.

[0112] In actual execution, a step trigger condition is determined for the target depth data of the nth frame and the (n+1)th frame. If the step trigger condition is met, the acquisition of color depth data is triggered, and the acquisition of the (n+2)th frame can continue. Then, the step trigger condition is determined for the target depth data of the (n+1)th frame and the (n+2)th frame.

[0113] It should be noted that when two consecutive time series meet the triggering conditions, the triggering of the latter time series can be ignored. For example, if the step triggering condition is met based on the nth frame and the (n+1)th frame, and the step triggering condition is also met on the (n+1)th frame and the (n+2)th frame, the step signals triggered by the (n+1)th frame and the (n+2)th frame can be ignored to eliminate logical errors and additional system consumption caused by multiple data collections due to a single item handling operation.

[0114] In some embodiments, determining the step trigger condition based on target depth data from two consecutive frames may include:

[0115] The target confidence score is obtained based on the target depth data of two consecutive frames.

[0116] The target confidence level is compared with the confidence threshold in the step trigger condition, and the step trigger condition is determined based on the comparison result.

[0117] It should be noted that confidence level can characterize the reliability and consistency of depth data over time. High confidence level regions may indicate abrupt changes in depth value patterns. Step detection can be implemented based on confidence level to determine whether the step triggering conditions are met.

[0118] In this embodiment, the target confidence is calculated based on the target depth data of two consecutive frames. The target confidence is compared with the preset confidence threshold in the step trigger condition. Based on the level of confidence, it is detected whether there is a sudden change in the depth value pattern, and it is determined whether the step trigger condition is met.

[0119] The embodiments of this application will be specifically described below from the perspective of implementing step detection based on target depth data of the first step line 410 and the second step line 420.

[0120] It should be noted that the first step line 410 and the second step line 420 are set relative to each other.

[0121] In some embodiments, obtaining the target confidence score based on target depth data from two consecutive frames may include:

[0122] Based on the target depth data of two consecutive frames at the first step line 410, the first target confidence corresponding to the first step line 410 is obtained, and based on the target depth data of two consecutive frames at the second step line 420, the second target confidence corresponding to the second step line 420 is obtained.

[0123] Comparing the target confidence level with the confidence threshold in the step trigger condition, and determining whether the step trigger condition is met based on the comparison result, may include:

[0124] The confidence level of the first target is compared with the confidence threshold, and the confidence level of the second target is compared with the confidence threshold. The result of the comparison determines whether the step triggering condition is met.

[0125] In this embodiment, based on the target depth data of two consecutive frames at the first step line 410, the first target confidence level corresponding to the first step line 410 is calculated, and the first target confidence level is compared with the confidence level threshold to determine the level of the first step line 410 handling confidence.

[0126] Meanwhile, based on the target depth data of two consecutive frames at the second step line 420, the second target confidence level corresponding to the second step line 420 is calculated. The second target confidence level is compared with the confidence level threshold to determine the level of the second step line 420 handling confidence.

[0127] For the first step line 410 and the second step line 420 that are set relative to each other, the target confidence level corresponding to the first step line 410 and the second step line 420 is considered to determine whether the target depth data of two consecutive frames meets the step triggering condition, and the direction of the step signal can also be determined.

[0128] In some embodiments, comparing a first target confidence level with a confidence threshold and comparing a second target confidence level with a confidence threshold, and determining whether a step trigger condition is met based on the comparison result, may include:

[0129] If the confidence level of the first target is less than the confidence threshold and the confidence level of the second target is greater than the confidence threshold, the step triggering condition is met, and the first step signal is triggered.

[0130] In this embodiment, when the first target confidence level of the first step line 410 is less than the confidence level threshold and the second target confidence level of the second step line 420 is greater than the confidence level threshold, it indicates that the second step line 420 is triggered before the first step line 410, thus triggering the first step signal.

[0131] For example, such as Figure 6 As shown, the first step line 410 and the second step line 420, which are set opposite to each other, are located on the outer edge of the shelf 340 of the storage device. The first step line 410 is close to the storage space, and the second step line 420 is close to the door 350.

[0132] The confidence level of the first target of the first jump line 410 is less than the confidence threshold, so the first jump line 410 is a region with low confidence. The confidence level of the second target is greater than the confidence threshold, so the second jump line 420 is a region with high confidence. The second jump line 420 is triggered before the first jump line 410, indicating that the user's limb may have entered the storage space.

[0133] In other embodiments, comparing a first target confidence level with a confidence threshold and comparing a second target confidence level with a confidence threshold, and determining whether a step trigger condition is met based on the comparison result, may include:

[0134] When the confidence level of the first target is greater than the confidence threshold and the confidence level of the second target is less than the confidence threshold, the step triggering condition is met, and the second step signal is triggered. The direction of the second step signal is opposite to the direction of the first step signal.

[0135] In this embodiment, when the first target confidence of the first step line 410 is greater than the confidence threshold and the second target confidence of the second step line 420 is less than the confidence threshold, it indicates that the first step line 410 is triggered before the second step line 420, thus triggering the second step signal.

[0136] For example, such as Figure 6As shown, the first step line 410 and the second step line 420, which are set opposite to each other, are located on the outer edge of the shelf 340 of the storage device. The first step line 410 is close to the storage space, and the second step line 420 is close to the door 350.

[0137] The confidence level of the first target of the first jump line 410 is greater than the confidence threshold, and the first jump line 410 is a region with high confidence. The confidence level of the second target is less than the confidence threshold, and the second jump line 420 is a region with low confidence. The first jump line 410 is triggered before the second jump line 420, indicating that the user's limb may have been moved out of the storage space.

[0138] In some embodiments, the target confidence level is determined by the following steps:

[0139] The depth data difference is obtained based on the target depth data of two consecutive frames.

[0140] Based on the depth data difference, determine the step coefficient and the accumulation coefficient;

[0141] The target confidence level is determined based on the step coefficient and the cumulative coefficient.

[0142] It should be noted that target confidence can be used to measure whether depth data conforms to the step characteristics formed when a user picks up or puts down an item. The step coefficient and the cumulative coefficient reflect the target confidence from two aspects.

[0143] The step coefficient is used to characterize the change in depth data in the horizontal direction. The greater the change in depth value, the more likely there is a step signal.

[0144] The cumulative coefficient is used to characterize whether the depth data in the horizontal direction matches the pulse characteristics formed when a person picks up or puts down an object. The larger the pulse duration and the smaller the number of pulses, the more likely there is a step signal, and the greater the confidence level of the target.

[0145] In this embodiment, the depth data difference corresponding to the first step line 410 is obtained by subtracting the target depth data of two consecutive frames at the first step line 410. Based on the depth data difference corresponding to the first step line 410, the step coefficient and accumulation coefficient corresponding to the first step line 410 can be calculated.

[0146] The depth data difference corresponding to the second step line 420 is obtained by subtracting the target depth data of two consecutive frames at the second step line 420. Based on the depth data difference corresponding to the second step line 420, the step coefficient and accumulation coefficient corresponding to the second step line 420 can be calculated.

[0147] For example, obtain the target depth data Dn in frame n and the target depth data Dn+1 in frame n+1, calculate the absolute value of the difference between the two frames, and obtain the depth data difference Ds = |Dn+1 - Dn|.

[0148] For the relatively set first step line 410 and second step line 420, the depth data difference Ds corresponding to the first step line 410 is calculated based on the target depth data of the nth frame and the (n+1th frame) at the first step line 410. 上线 Based on the target depth data at frame n and frame (n+1) at the second step line 420, the depth data difference Ds corresponding to the second step line 420 is calculated. 下线 .

[0149] In practice, the step coefficient can be calculated using the following formula:

[0150]

[0151] Among them, c s D is the step coefficient, k is the number of pixel values ​​in a row of target depth data at the step line, and i is the pixel index of the target depth data at the step line, i.e., the 1st, 2nd, ith, up to the kth pixel in a row. s [i] represents the depth data difference at the i-th pixel position, D s [i+1] represents the depth data difference at the (i+1)th pixel position.

[0152] In actual implementation, such as Figure 5 As shown, the accumulated depth data difference D can be used as a basis. s To calculate the cumulative coefficient, first input the data D to be judged. s Then iterate through D. s For each of the k pixels, we first determine whether the value of a single pixel is greater than a threshold t (t is a custom threshold used to characterize the triggering condition of a single pixel step).

[0153] If the value is less than or equal to the threshold t, then continue to check if parameter s is 0. If parameter s is 0, do nothing and continue to check the next pixel data. If parameter s is not 0, store s in list L and set s to 0 (s is used to determine the number of consecutive pixels greater than the threshold, i.e., the pulse span; storing s in L allows us to determine the number of pulses based on the length of list L), and continue to check the next pixel data. If the current pixel value is greater than the threshold t, then increment s and the pixel value D. s [i].

[0154] After iterating through all k data points, return L and a, where max(L) represents the maximum pulse span in this data set, and a represents the depth data difference D. s The accumulation of.

[0155] When the maximum pulse span of max(L) is greater than 15, ca is calculated based on the number of pulses len(L) and a. When the span is less than 15, ca is set to zero. Finally, ca is returned to calculate the target confidence level C.

[0156] Where L is a list that stores the length of each consecutive single-pixel step, max(L) refers to the maximum value in L, and len(L) refers to the length of the list L.

[0157] In this embodiment, the formula can be applied.

[0158]

[0159] Determine the target confidence level C;

[0160] Where e is the base of the natural logarithm, w s w is the weight of the step coefficient. a c represents the weight of the cumulative coefficient. s c is the step coefficient. a This is the cumulative coefficient.

[0161] In practice, the weights of the step coefficient and the cumulative coefficient can be adjusted according to the detection requirements. For example, the weight of the step coefficient is 0.15 and the weight of the cumulative coefficient is 0.85.

[0162] The following is a specific example.

[0163] like Figure 4 As shown, by setting the position of the step line and the confidence threshold T, a first step line 410 and a second step line 420 can be set on the outer edge of the storage equipment shelf 340. The first step line 410 and the second step line 420 are set opposite to each other, with the first step line 410 close to the storage space and the second step line 420 close to the door 350.

[0164] Start collecting step line data (target depth data of the step line). Read the step line data Dn of frame n and the step line data Dn+1 of frame n+1. The system retains only two frames of depth data at the same time. Perform step trigger condition judgment on the obtained two consecutive frames (frame n and frame n+1) of depth data.

[0165] If the condition is not met, the data of frame n is cleared and the data of frame n+2 is collected. The step trigger condition is judged based on the data of frame n+1 and frame n+2, and the loop continues until the condition is met.

[0166] If the condition is met, a step signal is triggered, and the loop continues to detect the triggering of the next step signal.

[0167] The step trigger condition judgment process is as follows:

[0168] Obtain the target depth data Dn in frame n and the target depth data Dn+1 in frame (n+1). Calculate the absolute value of the difference between the two frames to obtain the depth data difference Ds = |Dn+1 - Dn|, where Ds 上线 Ds represents the depth data difference corresponding to the first step line 410. 下线 This represents the depth data difference corresponding to the second step line 420.

[0169] For Ds 上线 and Ds 下线 Pattern recognition is performed separately, and the confidence level C is calculated based on the step coefficient and the cumulative coefficient. When the confidence level C of the first step line 410 is reached... Ds上线 The C value is less than the confidence threshold T, and the second step line is 420. Ds下线 When the value exceeds the confidence threshold T, the first step signal is triggered, indicating that a user's limb may have entered the storage space.

[0170] When the confidence level C of the first step line 410 Ds上线 C is greater than the confidence threshold T, and the second step line is 420. Ds下线 When the value is less than the confidence threshold T, a second step signal is triggered, indicating that a user's limb may have been removed from the storage space.

[0171] In this embodiment, step detection can be achieved by calculating the target depth data of two frames to determine whether the acquisition of color depth data is triggered. It has low memory usage, simple and efficient algorithm, and extremely low computational load. It can accurately and efficiently determine the process of picking up and putting down items. It does not require continuous reading of color depth data of each frame, nor does it require a large amount of computation based on neural networks on the obtained data, thus ensuring the compactness and efficiency of the system.

[0172] To more clearly illustrate the practical application effect of this application, the following will describe the embodiments of this application in detail with specific scenarios such as item entry and item movement detection of storage devices. It should be noted that the following embodiments are only illustrative examples, and those skilled in the art can make appropriate adjustments or extensions to the technical solutions according to actual needs to achieve the same or similar technical effects.

[0173] In some embodiments, the item entry for the storage device can be achieved through the following steps:

[0174] With the door of the storage device open at 350°, acquire the target depth data of the target detection position of the storage device;

[0175] In response to a step signal in the target depth data, color depth data of the storage space of the storage device is collected;

[0176] Based on the color depth data of the storage space, the item storage information of the storage device is determined.

[0177] The target detection location is set at the opening of the storage device.

[0178] Understandably, when the door of the storage device is 350 degrees open, users can take or put items in through the opening. By obtaining target depth data of the target detection position, the opening of the storage device can be monitored in real time, and the user's actions of taking or putting items in can be detected in a timely manner.

[0179] It should be noted that the target detection location is a small area at the opening, and for the same scene, the amount of depth data is less than the amount of color data. Obtaining the target depth data at the target detection location and determining whether to trigger the collection of color depth data can effectively reduce the amount of data collected and the amount of data processed subsequently.

[0180] In actual implementation, magnetic switches, mechanical switches, and other devices can be installed on the storage equipment to detect the opening and closing status of the door 350. If the door 350 is detected to be open, the data acquisition device 320 collects the target depth data of the target detection position.

[0181] In this embodiment, target depth data at the target detection location is acquired in real time for step detection. When a corresponding step signal appears in the target depth data, it indicates that a user's limb may enter the storage space to retrieve or place items, triggering the data acquisition device 320 to collect the color depth data of the storage space.

[0182] Based on the color depth data collected in response to the step signal, the system detects items that the user's limbs enter and take out of the storage space, identifies the type and location of the items in the storage space, records the information of the items taken out and put in, and obtains the item storage information of the storage device.

[0183] In this embodiment, based on the color depth data of the storage space, the items stored in the entire storage space can be identified to obtain the item storage information of the storage device; alternatively, only the items retrieved or placed by the user's limbs in the storage space can be identified to update the item storage information of the storage device in real time.

[0184] In practice, an item information database can be set up. The item information database contains item storage information such as the type and location of items stored in the storage device. It collects color depth data in response to the step signal of the target depth data, identifies the items to be taken or placed based on the color depth data, and updates the item storage information in the item information database, which can effectively improve the efficiency of item entry.

[0185] Understandably, when recognizing objects based on color depth data, a detection box can be marked for each object, and the quantity information of each type of object can be obtained based on the quantity and type information of the detection boxes.

[0186] In related technologies, vision-based object entry requires continuous image acquisition and recognition by a camera, resulting in a large amount of data and computational load, which is not conducive to data transmission and real-time output of detection results. The hardware cost is also high. Some technologies that start image acquisition and recognition only under specific conditions, such as starting image acquisition and recognition after a door is opened, or performing human body part detection first and then performing image recognition after successful detection, still have the problem of data redundancy. Most of them require neural networks to recognize each frame of image, and the computational load is still very heavy.

[0187] In this embodiment, a small target detection position is set at the opening of the storage device. When the door 350 is opened, the target depth data of the target detection position is acquired in real time for step detection. When the target depth data shows a corresponding step signal, the color depth data of the storage space is triggered to be collected. Item recognition is performed on the color depth data collected in response to the step signal. There is no need to continuously collect data or recognize every frame of data. The lightweight triggering method can effectively reduce the amount of data collected and processed, which is beneficial for data transmission and real-time output of detection results. It also helps to reduce the hardware cost of the device. By using color depth data for item recognition and comprehensively utilizing color and depth information, the recognition accuracy of item entry can also be improved.

[0188] In some embodiments, the target detection location includes a first step line 410 and a second step line 420 located opposite each other at the opening, with the first step line 410 near the storage space and the second step line 420 near the door 350.

[0189] Understandably, the target detection location can be adjusted according to the structure of the storage device.

[0190] For example, such as Figure 6 As shown, the target detection position is located at the front edge of the shelf 340 of the storage equipment. The first step line 410 and the second step line 420 are arranged opposite to each other and spaced apart. Figure 6 An upward arrow indicates the direction towards storage space. Figure 6 The downward arrow indicates the direction towards door 350.

[0191] like Figure 9 As shown, the data acquisition device 320 is configured with the positions of the first step line 410 and the second step line 420, and the center line of the field of view of the data acquisition device 320 can coincide with the outer edge of the drawer 330.

[0192] The first step line 410 is located 5 pixels from the outer edge of the shelf 340 toward the door 350, and the second step line 420 is located 20 pixels from the outer edge of the shelf 340 toward the door 350. The first step line 410 and the second step line 420 are spaced 15 pixels apart.

[0193] In this embodiment, target depth data of the target detection location is obtained, namely, depth data at the first step line 410 and depth data at the second step line 420. Step detection is performed using the depth data of the first step line 410 and the second step line 420 to determine whether to trigger the collection of color depth data. This allows for lightweight real-time monitoring of the opening of the storage device and identification of the direction of the step signal, enabling timely perception of the user's actions of taking or placing items.

[0194] In some embodiments, acquiring color depth data of the storage space of the storage device in response to a step signal of target depth data may include:

[0195] In response to the first step signal of the target depth data, the first color depth data of the storage space is acquired;

[0196] In response to a second step signal in response to the target depth data, second color depth data of the storage space is acquired;

[0197] Based on the color depth data of the storage space, the item storage information of the storage device can be determined, which may include:

[0198] Based on the first color depth data and the second color depth data, the item storage information of the storage device is determined.

[0199] In this embodiment, the direction of the second step signal is opposite to that of the first step signal. The first step signal can be a positive step, and the second step signal can be a negative step.

[0200] A first step signal indicating a positive step in the target depth data indicates that a user's limb has entered the storage space. A second step signal indicating a negative step in the target depth data indicates that a user's limb has moved out of the storage space. First color depth data is acquired in response to the positive step, and second color depth data is acquired in response to the negative step. Based on the first and second color depth data, the situation of a user's limb entering and leaving the storage space and taking or putting away items can be identified.

[0201] In related technologies, there is no judgment on the user's storage status. Users may change the storage location immediately after placing the item, resulting in low effectiveness of item location prediction.

[0202] In this embodiment, when the storage device is open, a step signal of the target depth data at the opening is detected in real time. In response to the first step signal, first color depth data is collected, and in response to the second step signal, second color depth data is collected. Based on the first color depth data and the second color depth data, the user's limbs can be identified during the process of entering and leaving the storage space to pick up and put down items, and the user's storage and retrieval status can be determined. Based on the user's storage and retrieval status, the effectiveness of item location prediction can be effectively improved, and the item storage location can be predicted more accurately.

[0203] In some embodiments, determining the item storage information of the storage device based on the first color depth data and the second color depth data may include:

[0204] Based on the first color depth data, determine the detection result of the first item;

[0205] The detection result of the second item is determined based on the second color depth data;

[0206] Based on the results of the first and second item inspections, the item storage information of the storage device is determined.

[0207] In this embodiment, when a first step signal of the target depth data is detected, the data acquisition device 320 is triggered to acquire first color depth data. Based on the first color depth data, item recognition is performed to obtain a first item detection result corresponding to the first color depth data. At the same time, the acquisition of target depth data is continued. When a second step signal of the target depth data is detected, the data acquisition device 320 is triggered to acquire second color depth data. Based on the second color depth data, item recognition is performed to obtain a second item detection result corresponding to the second color depth data. By combining the first item detection result and the second item detection result, the item storage information of the storage device is obtained.

[0208] The first item detection result may include item detection information and user limb information for operating the storage device, and the second item detection result may include item detection information and user limb information for operating the storage device.

[0209] It is understandable that the user's limbs operating the storage device are the user's limbs appearing at the opening of the storage device in the field of view of the data acquisition device 320. The user's limbs enter the storage device from the opening and move inside the storage device to complete the action of taking and putting away items.

[0210] In practice, an object recognition model can be used to detect targets in the first color depth data and identify the corresponding first object detection result. The first object detection result can include object detection information such as object type and object location, and can also include user limb information for operating the storage device.

[0211] An object recognition model can be used to perform target detection on the second color depth data and identify the corresponding second object detection result. The second object detection result can include object detection information such as object type and object location, and can also include user limb information of the user operating the storage device.

[0212] In this embodiment, the same object recognition model or detection network can be used to perform target detection on the first color depth data and the second color depth data respectively, so as to obtain the first object detection result and the second object detection result.

[0213] Understandably, the first and second item detection results can respectively represent the item status when the user's limbs enter the storage space and the item status when the user's limbs are removed from the storage space. By combining the first and second item detection results, the user's storage and retrieval status can be determined, the item location can be predicted more accurately, and highly accurate item storage information can be obtained.

[0214] It should be noted that, based on the first color depth data and the second color depth data, it is possible to detect only items within the range of the user's limb movements, which helps to reduce the computational load, improve recognition efficiency, and output more accurate detection results in real time.

[0215] In practice, the detection information in the first and second item detection results can be compared to detect whether the items within the user's limb movement range have changed, and to determine whether the user has picked up or put in the items.

[0216] It should be noted that the user's limb movement range refers to the area where the user's limbs appear in the item detection results. Items within the area where the user's limbs do not appear can be considered unchanged. By judging whether the item detection information within the user's limb movement range has changed, the user's access status can be accurately identified.

[0217] Understandably, based on the first color depth data and the second color depth data, it is also possible to detect items in the entire area of ​​the storage space. Combined with the judgment of the user's storage and retrieval status, it is possible to comprehensively and accurately identify information such as the type, quantity, and storage location of items stored in the storage device.

[0218] The embodiments of this application will be described in detail below from three different implementation perspectives.

[0219] 1. The user did not pick up or leave the item.

[0220] In some embodiments, determining the item storage information of the storage device based on the first item detection result and the second item detection result may include:

[0221] If the item detection information of the user's limb range of motion in the first item detection result is the same as the item detection information of the user's limb range of motion in the second item detection result, the item storage information of the storage device remains unchanged.

[0222] In this embodiment, when the item detection information within the user's limb movement range is the same in the first item detection result and the second item detection result, that is, the item type, quantity and position are unchanged, it indicates that the user has not picked up, placed or moved the item, and the item storage information of the storage device remains unchanged.

[0223] 2. The user takes out the item and puts it back in.

[0224] In some embodiments, determining the item storage information of the storage device based on the first item detection result and the second item detection result may include:

[0225] If the item detection information of the user's limb range of motion in the first item detection result is different from the item detection information of the user's limb range of motion in the second item detection result, the item held by the user's limb in the first item detection result is entered into the item storage information of the storage device, and the item held by the user's limb in the second item detection result is removed from the item storage information of the storage device.

[0226] In this embodiment, in response to the first color depth data acquired via a forward step, a first item detection result is determined. The item held by the user's limb in the first item detection result can be an item placed by the user in the storage device, and the item can be recorded in the item storage information of the storage device. In response to the second color depth data acquired via a reverse step, a second item detection result is determined. The item held by the user's limb in the second item detection result can be an item taken out by the user from the storage device, and the item can be removed from the item storage information of the storage device, and the item storage information of the storage device is updated.

[0227] Take a refrigerator as an example of a storage device.

[0228] The food database includes information such as the types and storage locations of food items A, B, and C in the refrigerator. If the first item detection result shows that the item held by the user's limbs is food item D, the type and storage location of food item D will be entered into the food database. If the second item detection result shows that the item held by the user's limbs is food item A, the type and storage location of food item A will be removed from the food database.

[0229] 3. The user puts in an item but does not take it out.

[0230] In some embodiments, determining the item storage information of the storage device based on the first item detection result and the second item detection result may include:

[0231] If the item detection information of the user's limb movement range in the first item detection result is different from the item detection information of the user's limb movement range in the second item detection result, and the user's limb is not holding an item in the second item detection result, then the item held by the user's limb in the first item detection result is recorded into the item storage information of the storage device.

[0232] In this embodiment, the item detection information of the user's limb movement range in the first item detection result and the second item detection result are different, and the user's limbs are not holding items in the second item detection result, indicating that the user only puts items into the storage device and does not take items out of the storage device. Therefore, the items held by the user's limbs in the first item detection result can be recorded into the item storage information of the storage device.

[0233] Take a refrigerator as an example of a storage device.

[0234] The food database includes information such as the types and storage locations of food items A, B, and C in the refrigerator. If the first item detection result shows that the user is holding food item D, and the second item detection result shows that the user is not holding any item, the information such as the type and storage location of food item D will be entered into the food database.

[0235] The following is a specific example.

[0236] The refrigerator's computer board stores a food database, which includes information such as the type, quantity, and storage location of the food in the refrigerator. The food database may also include information such as the expiration date and nutritional composition of the food.

[0237] like Figure 3 As shown, after the refrigerator is powered on, it performs status detection. When it detects that the refrigerator door 350 is open (in the open state), it acquires the target depth data at the first step line 410 and the second step line 420 at the refrigerator opening in real time, and then... Figure 4 The step detection process shown determines whether the target depth data triggers a step signal.

[0238] In response to a step signal in the target depth data, the system collects color depth data of the storage space, identifies the type and location of items, records information on items taken out and put in, and updates the food inventory information.

[0239] like Figure 6 As shown, the first step line 410 and the second step line 420 are set on the outer edge of the shelf 340. The first step line 410 is set near the inner liner of the refrigerator, and the second step line 420 is set near the door 350 of the refrigerator.

[0240] When a positive step detection is performed and the first step signal corresponding to the positive step is triggered, the sensor of the data acquisition device 320 is triggered to acquire color depth data. The target detection is performed on the acquired first color depth data through the detection network (i.e., the object recognition model), the food and storage location are identified, and the recognition result 1 (i.e. the first item detection result) is output. Result 1 is temporarily stored.

[0241] The storage locations can be divided according to different areas of the storage space, for example, Figure 10 As shown, the storage location can be classified into 9 categories: A, B, C, D, E, F, G, H, and I, which respectively indicate different locations in the shelf 340 area. Below the shelf 340 is the drawer 330.

[0242] At the same time, the data acquisition device 320 maintains the acquisition of target depth data at the first step line 410 and the second step line 420, performs reverse step detection, and when the second step signal corresponding to the reverse step is triggered, the sensor of the data acquisition device 320 is triggered again to acquire color depth data. The target detection is performed on the acquired second color depth data through the detection network, and the recognition result 2 (second item detection result) is output.

[0243] When result 2 is valid, the relationship between result 2 and result 1 is determined. If the ingredients detected by result 1 and result 2 are the same, the ingredient database is not updated, the item storage information remains unchanged, and the data acquisition device 320 continues to acquire the target depth data at the step line.

[0244] If the ingredients in Result 1 and Result 2 are different, the ingredients in Result 1 are entered, and the ingredients in Result 2 are removed from the ingredient library. The data acquisition device 320 continues to acquire the target depth data at the step line.

[0245] If result 2 detects a human upper limb (i.e., the user's limb) but not food, then the food from result 1 is entered, and the data acquisition device 320 continues to acquire target depth data at the step line until the refrigerator door 350 is closed.

[0246] In the foregoing embodiments, the data acquisition method provided by this application can effectively reduce the amount of data collected and has a low computational load in the scenario of non-contact item entry. It can also accurately and efficiently determine the process of item retrieval and placement, thereby improving the accuracy of item entry. It is understood that the data acquisition method provided by this application is also applicable to the scenario of item movement detection, and can similarly reduce the amount of data collected and processed.

[0247] In actual operation, when the door 350 is closed, the data acquisition device 320 can continuously collect the target depth data of the step line (at this time, the step line can be set at any position of the storage device where the movement of food needs to be detected, instead of at the opening). At this time, whether a positive step signal or a negative step signal is triggered, the sensor of the data acquisition device 320 will be triggered to collect color depth data and detect it through the detection network. If the detection network detects food, it will be recorded in the abnormal food database, indicating that the food has moved while the refrigerator door is closed, thus realizing the detection of item movement.

[0248] In this embodiment, users can be notified of abnormal food conditions and reminded of food falling through interactive methods such as APP, voice, and display screen. The above cycle continues in the closed state until the refrigerator door 350 is opened, and the item entry recognition process begins.

[0249] Understandably, users can also set refrigerator modes through interactive methods such as apps, voice commands, and displays. For example, in low-energy mode, the data acquisition device 320 can be stopped when the door is closed, the food movement reminder can be turned off, and only the food entry function when the door is open can be retained. Users can also actively update the food storage information in the food database through interactive methods such as apps, voice commands, and displays.

[0250] The data acquisition method provided in this application can be executed by a data acquisition device. This application uses an example of a data acquisition device executing the data acquisition method to illustrate the data acquisition device provided in this application.

[0251] This application also provides a data acquisition device.

[0252] like Figure 2 As shown, the data acquisition device includes:

[0253] The acquisition module 210 is used to acquire the target depth data of the storage device;

[0254] The response module 220 is used to collect color depth data of the storage space of the storage device in response to the step signal of the target depth data. The color depth data is used for item recognition of the storage device.

[0255] According to the data acquisition device provided in the embodiments of this application, step detection is performed by acquiring target depth data. When a corresponding step signal appears in the target depth data, the color depth data of the storage device is triggered for item recognition. This can reduce the amount of data and computational load required for item recognition, and help reduce the hardware cost of the device.

[0256] In some embodiments, the response module 220 is further configured to determine the step trigger condition based on the target depth data of two consecutive frames;

[0257] Under the condition that the step triggering condition is met, a step signal is obtained to obtain the target depth data.

[0258] In some embodiments, the response module 220 is further configured to obtain a target confidence level based on target depth data from two consecutive frames;

[0259] The target confidence level is compared with the confidence threshold in the step trigger condition, and the step trigger condition is determined based on the comparison result.

[0260] In some embodiments, the response module 220 is further configured to obtain a first target confidence level corresponding to the first step line 410 based on the target depth data of two consecutive frames at the first step line 410, and to obtain a second target confidence level corresponding to the second step line 420 based on the target depth data of two consecutive frames at the second step line 420, wherein the first step line 410 and the second step line 420 are set relative to each other.

[0261] The target confidence level is compared with the confidence threshold in the step trigger condition, and the result of the comparison determines whether the step trigger condition is met, including:

[0262] The confidence level of the first target is compared with the confidence threshold, and the confidence level of the second target is compared with the confidence threshold. The result of the comparison determines whether the step triggering condition is met.

[0263] In some embodiments, the response module 220 is further configured to trigger a first step signal when the first target confidence level is less than the confidence level threshold and the second target confidence level is greater than the confidence level threshold, thereby satisfying the step triggering condition.

[0264] Alternatively, if the confidence level of the first target is greater than the confidence threshold and the confidence level of the second target is less than the confidence threshold, the step triggering condition is met, and a second step signal is triggered, the direction of which is opposite to that of the first step signal.

[0265] In some embodiments, the target confidence level is determined by the following steps:

[0266] The depth data difference is obtained based on the target depth data of two consecutive frames.

[0267] Based on the depth data difference, determine the step coefficient and the accumulation coefficient;

[0268] The target confidence level is determined based on the step coefficient and the cumulative coefficient.

[0269] In some embodiments, the response module 220 is further configured to clear one frame of target depth data and reacquire one frame of target depth data if the target depth data of two consecutive frames does not meet the step triggering condition, and return the step triggering condition judgment based on the target depth data of two consecutive frames.

[0270] In some embodiments, the acquisition module 210 is further configured to acquire target depth data of the target detection location of the storage device.

[0271] The data acquisition device provided in this application embodiment can realize the various processes implemented in the above data acquisition method embodiment, and will not be described again here to avoid repetition.

[0272] This application also provides a storage device.

[0273] like Figure 7 As shown, the storage device may include a housing 310 and a data acquisition device 320.

[0274] The housing 310 defines a storage space, and the data acquisition device 320 is disposed in the housing 310. The data acquisition device 320 is used to acquire depth data and color depth data, and is used to perform the above-mentioned data acquisition method.

[0275] According to the storage device provided in the embodiments of this application, step detection is performed by acquiring target depth data. When a corresponding step signal appears in the target depth data, the color depth data of the storage device is triggered for item recognition. This can reduce the amount of data and computational load required to recognize items, and help reduce the hardware cost of the device.

[0276] In some embodiments, such as Figure 11 As shown, this application embodiment also provides an electronic device 1100, including a processor 1101, a memory 1102, and a computer program stored in the memory 1102 and executable on the processor 1101. When the program is executed by the processor 1101, it implements the various processes of the above-described data acquisition method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0277] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0278] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described data acquisition method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0279] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0280] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described data acquisition method.

[0281] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0282] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above data acquisition method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0283] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0284] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0285] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0286] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0287] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0288] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A data acquisition method, characterized in that, include: Acquire target depth data for storage devices; In response to a step signal in the target depth data, color depth data of the storage space of the storage device is acquired, and the color depth data is used for item identification of the storage device.

2. The data acquisition method according to claim 1, characterized in that, Prior to the step signal in response to the target depth data, the method includes: Based on the target depth data of two consecutive frames, a step trigger condition is determined; Under the condition that the step triggering condition is met, a step signal of the target depth data is obtained.

3. The data acquisition method according to claim 2, characterized in that, The step trigger condition determination based on the target depth data of two consecutive frames includes: The target confidence level is obtained based on the target depth data of two consecutive frames; The target confidence level is compared with the confidence threshold in the step trigger condition, and the step trigger condition is determined based on the comparison result.

4. The data acquisition method according to claim 3, characterized in that, The target confidence score is obtained based on the target depth data from two consecutive frames, including: Based on the target depth data of two consecutive frames at the first step line, a first target confidence level corresponding to the first step line is obtained, and based on the target depth data of two consecutive frames at the second step line, a second target confidence level corresponding to the second step line is obtained, wherein the first step line and the second step line are set relative to each other. The step of comparing the target confidence level with the confidence threshold in the step trigger condition, and determining whether the step trigger condition is met based on the comparison result, includes: The first target confidence level is compared with the confidence threshold, and the second target confidence level is compared with the confidence threshold. Based on the comparison result, it is determined whether the step triggering condition is met.

5. The data acquisition method according to claim 4, characterized in that, The step of comparing the first target confidence level with the confidence threshold and comparing the second target confidence level with the confidence threshold, and determining whether the step triggering condition is met based on the comparison result, includes: If the confidence level of the first target is less than the confidence level threshold and the confidence level of the second target is greater than the confidence level threshold, the step triggering condition is met, and the first step signal is triggered. Alternatively, if the confidence level of the first target is greater than the confidence threshold and the confidence level of the second target is less than the confidence threshold, the step triggering condition is met, and a second step signal is triggered, wherein the direction of the second step signal is opposite to the direction of the first step signal.

6. The data acquisition method according to claim 3, characterized in that, The target confidence level is determined through the following steps: The depth data difference is obtained based on the target depth data of two consecutive frames; Based on the depth data difference, the step coefficient and the accumulation coefficient are determined; The target confidence level is determined based on the step coefficient and the accumulation coefficient.

7. The data acquisition method according to any one of claims 2-6, characterized in that, After determining the step trigger condition based on the target depth data from two consecutive frames, the method further includes: If the target depth data in two consecutive frames does not meet the step trigger condition, clear one frame of target depth data, reacquire one frame of target depth data, and return to the step trigger condition judgment step based on the target depth data in two consecutive frames.

8. The data acquisition method according to any one of claims 1-6, characterized in that, The acquisition of target depth data for the storage device includes: Obtain the target depth data at the target detection location of the storage device.

9. A data acquisition device, characterized in that, include: The acquisition module is used to acquire the target depth data of the storage device; A response module is used to collect color depth data of the storage space of the storage device in response to a step signal of the target depth data, and the color depth data is used for item identification of the storage device.

10. A storage device, characterized in that, include: A housing that defines a storage space; A data acquisition device is disposed in the housing, the data acquisition device is used to acquire depth data and color depth data, and the data acquisition device is used to perform the data acquisition method as described in any one of claims 1-8.

Images