Intelligent refrigerator, access action recognition method, device and medium

Abstract

This application provides a smart refrigerator, a method, device, and medium for recognizing access actions. In this application, a first target ROI region containing a hand area is determined in a first target RGB image, and a second target ROI region containing a hand area is determined in a second target RGB image acquired after the acquisition time of the first target RGB image. If the first target ROI region is identified as being located in a food storage area, a first area of ​​the first target ROI region in the first target RGB image and a second area of ​​the second target ROI region in the second target RGB image are determined. Based on the size of the first area and the second area, the access action is determined, thereby improving the accuracy of access action recognition.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a smart refrigerator, a method, device and medium for recognizing access actions. Background Technology

[0002] With the advancement of science and technology, smart refrigerators have appeared in various households, becoming one of the most common white goods in family life. Among these features, food management has become a core function of smart refrigerators, encompassing the management of food types, quantities, shelf lives, and storage locations. The food storage location information is determined based on the actions taken to access the food.

[0003] In some implementations of refrigerator food storage and retrieval actions, a depth camera is typically used to acquire the spatial position of the item, and then the position of the hand is determined. Based on the spatial position of the item and the position of the hand, the user's current action is identified as either a storage or retrieval action. However, when determining the spatial position of an item based on a depth camera, if the item is far from the camera, the calculated spatial position of the item will have a large error compared to the actual spatial position, leading to inaccurate or unrecognizable storage and retrieval actions. Summary of the Invention

[0004] This application provides a smart refrigerator, a method for recognizing access actions, a device, and a medium to solve the problems of inaccurate or unrecognizable access actions in the prior art.

[0005] In a first aspect, this application provides a smart refrigerator, the smart refrigerator comprising:

[0006] Acquisition unit, configured to acquire a first target RGB image and acquire a second target RGB image;

[0007] Control unit, the control unit being configured to:

[0008] The first target ROI region containing the hand region is determined in the first target RGB image acquired, and the second target ROI region containing the hand region is determined in the second target RGB image acquired after the acquisition time of the first target RGB image.

[0009] If the first target ROI region is identified as being located in the food storage area, then the first area of ​​the first target ROI region in the first target RGB image and the second area of ​​the second target ROI region in the second target RGB image are determined; based on the size of the first area and the second area, the access action is determined.

[0010] Secondly, this application also provides a method for recognizing access actions, the method comprising:

[0011] The first target ROI region containing the hand region is determined in the first target RGB image acquired, and the second target ROI region containing the hand region is determined in the second target RGB image acquired after the acquisition time of the first target RGB image.

[0012] If the first target ROI region is identified as being located in the food storage area, then the first area of ​​the first target ROI region in the first target RGB image and the second area of ​​the second target ROI region in the second target RGB image are determined.

[0013] The access action is determined based on the size of the first area and the second area.

[0014] Thirdly, this application also provides an electronic device, which includes at least a processor and a memory, wherein the processor is used to implement the steps of the access action recognition method described above when executing a computer program stored in the memory.

[0015] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the access action recognition method described above.

[0016] In this application, a first ROI containing a hand region is identified in the first target RGB image, and a second ROI containing a hand region is identified in the second target RGB image acquired after the acquisition of the first target RGB image. If the first target ROI is identified as being located in a food storage area, a first area of ​​the first target ROI in the first target RGB image and a second area of ​​the second target ROI in the second target RGB image are determined. Based on the size of the first and second areas, an access action is determined. In this application, by identifying a first target ROI containing a hand region in the first target RGB image and a second target ROI containing a hand region in the first target RGB image, when the first target ROI is identified as being located in a food storage area, the access action is determined based on the size of the areas of the first and second target ROI regions, thus improving the accuracy of access action recognition. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This application provides a structural schematic diagram of an intelligent refrigerator;

[0019] Figure 2a Schematic diagram of ROI regions provided for some embodiments of this application;

[0020] Figure 2b Schematic diagram of ROI regions provided for some embodiments of this application;

[0021] Figure 3a A schematic diagram of the first region of a drawer provided in some embodiments of this application;

[0022] Figure 3b A schematic diagram of the first region of a drawer provided in some embodiments of this application;

[0023] Figure 4 This application provides schematic diagrams of the internal space of a smart refrigerator according to some embodiments.

[0024] Figure 5a A schematic diagram illustrating a first target ROI region not located within a first region, provided for some embodiments of this application;

[0025] Figure 5b A schematic diagram showing the first target ROI region located within a first region, provided for some embodiments of this application;

[0026] Figure 6 A schematic diagram illustrating the access action recognition process provided for some embodiments of this application;

[0027] Figure 7 This is a schematic diagram of the access action recognition process provided in some embodiments of this application;

[0028] Figure 8 This is a schematic diagram of an electronic device structure provided in this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] In this application, after receiving a first target RGB image and a second target RGB image acquired after the acquisition time of the first target RGB image, it is determined that the first target RGB image contains a first target ROI region containing a hand region, and the second target RGB image contains a second target ROI region containing a hand region. If the first target ROI region is identified as being located in the food storage area, a first area of ​​the first target ROI region in the first target RGB image and a second area of ​​the second target ROI region in the second target RGB image are determined. Based on the size of the first area and the second area, an access operation is determined.

[0031] To improve the accuracy of access action recognition, this application provides a smart refrigerator, access action recognition method, device and medium.

[0032] Figure 1 This application provides a schematic diagram of the structure of a smart refrigerator, which includes:

[0033] Acquisition unit 101, the acquisition unit is configured to acquire a first target RGB image and acquire a second target RGB image;

[0034] Control unit 102, the control unit is configured to determine a first target ROI region containing a hand region in a first target RGB image acquired, and a second target ROI region containing a hand region in a second target RGB image acquired after the acquisition time of the first target RGB image;

[0035] If the first target ROI region is identified as being located in the food storage area, then the first target is determined. The first area of ​​the first target ROI region in the RGB image and the second area of ​​the second target ROI region in the second target RGB image;

[0036] The access action is determined based on the size of the first area and the second area.

[0037] In this application, a data acquisition unit is installed in the top area inside the smart refrigerator. When the refrigerator door is opened, the data acquisition unit acquires RGB images in real time. Specifically, the data acquisition unit may include an image acquisition device. To reduce the blurriness of the captured RGB images due to hand movements, this application preferentially uses an image acquisition device with a short exposure time.

[0038] After acquiring the first target RGB image and the second target RGB image, in order to recognize the access action, this application first needs to track the position of the hand in the target RGB image. The second target RGB image can be the first RGB image acquired after the acquisition time of the first target RGB image, or it can be an RGB image acquired after a preset time interval from the acquisition time of the first target RGB image.

[0039] Specifically, a first region of interest (ROI) containing the hand area is determined in the first target RGB image, and a second target RGB image containing the hand area is obtained from the second target RGB image. In this application, the hand area includes the user's hand and the food held in the user's hand.

[0040] Specifically, the hand region can be determined based on traditional vision methods, such as hand skin color detection or moving object detection.

[0041] In this application, when recognizing access actions based on the target RGB image, in order to save resources, the smart refrigerator performs access action recognition when the target ROI region in the target RGB image is located in the food storage area. If the target ROI region in the target RGB image is not located in the food storage area, access action recognition is not performed until a target RGB image with the target ROI region located in the food storage area appears.

[0042] In this application, if the first target ROI region is identified as being located in the food storage area, the first area of ​​the first target ROI region in the first target RGB image and the second area of ​​the second target ROI region in the second target RGB image are determined.

[0043] After determining the first area and the second area, the storage or retrieval action is determined based on their sizes. Specifically, if the first area is larger than the second area, a storage action is determined; if the first area is smaller than the second area, a retrieval action is determined.

[0044] In this application, by determining a first target ROI region and a second target ROI region in a first target RGB image, when the first target ROI region is identified as being located in the food storage area, the access action is determined based on the size of the areas of the first target ROI region and the second target ROI region, thereby improving the accuracy of access action recognition.

[0045] In order to determine the Region of Interest (ROI) containing the hand area in the target RGB image, based on the above embodiments, in this application, the control unit 102 is specifically configured as follows:

[0046] The target RGB image is input into the trained network model, and the network model outputs an RGB image that identifies the target ROI region.

[0047] In this application, the determination of the target ROI region can be based on a deep learning method. Specifically, when determining the ROI region of the hand area contained in the target RGB image, the target RGB image is input into a trained network model, and the network model outputs an RGB image that identifies the target ROI region. Specifically, the information of the target ROI region can be a shaded area selected in the RGB image, or it can be the coordinate information of the target ROI region in the RGB image.

[0048] Figure 2a A schematic diagram of the ROI region provided for some embodiments of this application, as shown below. Figure 2a As shown, the Figure 2a The area enclosed in the box is the ROI region.

[0049] Figure 2b A schematic diagram of the ROI region provided for some embodiments of this application, as shown below. Figure 2b As shown, the Figure 2b The area enclosed in the box is the ROI region.

[0050] In order to obtain a trained network model for acquiring the target ROI region, based on the above embodiments, in this application, the control unit 102 is specifically configured as follows:

[0051] Obtain any sample RGB image from the training set, wherein the sample RGB image is pre-labeled with the first location information of the ROI region;

[0052] The sample RGB image is input into the network model, and the second location information of the ROI region in the sample RGB image is output.

[0053] The network model is trained based on the first and second location information.

[0054] When determining the target Region of Interest (ROI) using deep learning-based methods, it is necessary to pre-train the network model. To train the network model, a training set is pre-configured, which stores sample RGB images, each of which is pre-annotated with the first information of the ROI region.

[0055] During training, any sample RGB image in the training set is obtained, wherein the ROI region is pre-annotated in the sample RGB image with first information. The sample RGB image is input into the network model, and the second information of the ROI region in the sample RGB image is output. The network model is trained based on the first and second information.

[0056] Specifically, an RGB image with labeled Regions of Interest (ROIs) is input into the network model for training. The model's loss value is calculated based on the network output and the pre-labeled ROIs, and the model's parameters are adjusted accordingly. Training is considered complete when the loss value reaches the minimum or the number of iterations reaches the preset maximum.

[0057] In order to identify whether the first target ROI area is located in the food storage area, based on the above embodiments, in this application, the control unit 102 is specifically configured as follows:

[0058] If it is determined that the drawer of the smart refrigerator is in the open state, then the first region of the drawer in the first target RGB image is determined;

[0059] If the first target ROI region is located within the first region, then the first target ROI region is determined to be located in the food storage area.

[0060] In this application, if it is determined that the drawer of the smart refrigerator is in an open state, the user may be storing or retrieving food in the drawer. Therefore, in order to determine whether the user is storing or retrieving food in the drawer, the first region where the drawer is located in the first target RGB image is determined, and it is determined whether the first target ROI region is located within the first region. If the first target ROI region is located within the first region, it is determined that the first target ROI region is located in the food storage area, and the drawer is located in the food storage area.

[0061] After confirming that the drawer is open, in the first target RGB image in the constructed coordinate system, determine each coordinate of the edge of the first target ROI region. If all coordinates of the edge of the first target ROI region are within the range of the first region, then the first target ROI region is determined to be located within the first region, that is, the first target ROI region is determined to be located in the food storage area.

[0062] Figure 3a This is a schematic diagram of the first region of a drawer provided in some embodiments of this application, as shown below. Figure 3a As shown, the area outlined in the box is the first section of the drawer.

[0063] Figure 3bThis is a schematic diagram of the first region of a drawer provided in some embodiments of this application, as shown below. Figure 3b As shown, the area outlined in the box is the first section of the drawer.

[0064] In order to identify that the first target ROI region is located in the food storage area, based on the above embodiments, in this application, the control unit 102 is specifically configured as follows:

[0065] Based on the second area of ​​the food storage area of ​​the smart refrigerator (excluding drawers) that is saved in advance, determine whether the first target ROI area is located within the second area;

[0066] If so, then the first target ROI region is determined to be located in the food storage area.

[0067] The smart refrigerator also includes other food storage areas besides drawers, such as shelves and door side storage. In this application, a second area for other food storage areas besides drawers of the smart refrigerator is pre-stored. Each coordinate of the edge of the first target ROI area is determined. If all coordinates of the edge of the first target ROI area are within the range of the second area, then the first target ROI area is determined to be located in the second area, and the first target ROI area is determined to be located in the food storage area.

[0068] For example, the coordinates of the top-left corner of the drawer in the first target RGB image are determined to be (105, 90), and the coordinates of the bottom-right corner are determined to be (200, 75). The coordinates of the top-left corner of the drawer are pre-saved as (105, 75), and the coordinates of the bottom-right corner are also pre-saved as (200, 75). This indicates that the drawer is in an open state, and the first region of the drawer is determined to be a rectangle with vertices (150, 90), (105, 75), (200, 75), and (200, 90). The first target ROI region is identified as a rectangle with vertices (160, 82), (160, 70), (180, 70), and (180, 82). Since the coordinates indicate that the first target ROI region is located within this first region, it is determined that the first target ROI region is located in the food storage area.

[0069] Figure 4 This is a schematic diagram of the internal space of a smart refrigerator provided in some embodiments of this application, such as... Figure 4 As shown, the data collection unit of the smart refrigerator is located at the top center of the interior space, and the food storage area of ​​the smart refrigerator includes drawers and shelves.

[0070] Figure 5a This is a schematic diagram illustrating how the first target ROI region is not located within the first region, as provided in some embodiments of this application. Figure 5a As shown, the first target ROI region is not located in the first region of the drawer.

[0071] Figure 5b This application provides a schematic diagram showing the first target ROI region located within a first region, as illustrated in some embodiments. Figure 5b As shown, the first target ROI region is located in the first region of the drawer.

[0072] Figure 6 This is a schematic diagram of the access action recognition process provided in some embodiments of this application, such as... Figure 6 As shown, the process includes:

[0073] S601: Determine the first target ROI region containing the hand region in the first target RGB image acquired, and the second target ROI region containing the hand region in the second target RGB image acquired after the acquisition time of the first target RGB image.

[0074] S602: Identify whether the first target ROI area is located in the food storage area. If yes, proceed to S603; otherwise, end.

[0075] S603: Determine the first area of ​​the first target ROI region in the first target RGB image and the second area of ​​the second target ROI region in the second target RGB image.

[0076] S604: Determine the access action based on the size of the first area and the second area.

[0077] To improve the accuracy of access action recognition, based on the above embodiments, in this application, the control unit 102 is specifically configured as follows:

[0078] If the first area is larger than the second area, then the current action is determined to be a storage action;

[0079] If the first area is smaller than the second area, then the current action is determined to be a take-out action.

[0080] In this application, when determining the access action, the access action can be determined based on the first area of ​​the first target ROI region in the first target RGB image and the second area of ​​the second target ROI region in the second target RGB image acquired after the acquisition time of the first target RGB image.

[0081] Specifically, in this application, the position of the acquisition unit for acquiring RGB images in the smart refrigerator is fixed and located at the top of the smart refrigerator. When a storage action occurs, as the action occurs, the hand area, which is the target ROI area, gradually moves away from the image acquisition device. Therefore, in the RGB image acquired by the acquisition unit, the area of ​​the target ROI area gradually decreases. When a retrieval action occurs, as the action occurs, the hand area, which is the target ROI area, gradually moves closer to the image acquisition device. Therefore, in the RGB image acquired by the acquisition unit, the area of ​​the target ROI area gradually increases.

[0082] Therefore, in this application, if the first area of ​​the first target ROI region of the first target RGB image is smaller than the second area of ​​the second ROI region of the second RGB image, it indicates that the current action is approaching the image acquisition device, and the current action is determined to be a retrieval action; if the first area of ​​the first target ROI region of the first target RGB image is larger than the second area of ​​the second ROI region of the second RGB image, it indicates that the current action is moving away from the image acquisition device, and the current action is determined to be a storage action.

[0083] In order to identify the open / closed state of the drawer, based on the above embodiments, in this application, the control unit 102 is specifically configured as follows:

[0084] Determine the first position information of the drawer edge in the first target RGB image;

[0085] If the first position information is inconsistent with the second position information of the drawer edge when the drawer is closed (pre-saved), then the drawer is determined to be in the open state.

[0086] When determining whether a drawer is open, first position information of the drawer edge is determined in the first target RGB image. Based on this first position information and pre-saved second position information of the drawer edge when the drawer is closed, the state information of the drawer in the first target RGB image is determined. If the first position information and the second position information are consistent, the drawer is determined to be closed; if the first position information and the second position information are inconsistent, the drawer is determined to be open.

[0087] Specifically, in this application, a coordinate system is constructed in the first target RGB image according to a pre-set method for constructing a coordinate system. The coordinates of the drawer edge in the first target RGB image are used to determine whether the drawer is in an open state based on the pre-saved coordinates of the drawer edge when the drawer is closed. If the coordinates of the drawer edge in the first RGB image are inconsistent with the pre-saved coordinates of the drawer edge, the drawer is determined to be in an open state.

[0088] Figure 7This is a schematic diagram of the access action recognition process provided in some embodiments of this application, such as... Figure 7 As shown, the process includes:

[0089] S701: Determine that the first target RGB image acquired contains a first target ROI region containing a hand region, and the second target RGB image acquired after the acquisition time of the first target RGB image contains a second target ROI region containing a hand region.

[0090] S702: If the first target ROI region is identified as being located in the food storage area, then the first area of ​​the first target ROI region in the first target RGB image and the second area of ​​the second target ROI region in the second target RGB image are determined.

[0091] S703: Determine the access action based on the size of the first area and the second area.

[0092] In one possible implementation, determining the target ROI region containing the hand region in the acquired target RGB image includes:

[0093] The target RGB image is input into the trained network model, and the network model outputs an RGB image that identifies the target ROI region.

[0094] In one possible implementation, identifying that the first target ROI region is located at the food storage location includes:

[0095] If it is determined that the drawer of the smart refrigerator is in the open state, then the first region of the drawer in the first target RGB image is determined;

[0096] If the first target ROI region is located within the first region, then the first target ROI region is determined to be located in the food storage area.

[0097] In one possible implementation, identifying that the first target ROI region is located at the food storage location includes:

[0098] Based on the second area of ​​the food storage area of ​​the smart refrigerator (excluding drawers) that is saved in advance, determine whether the first target ROI area is located within the second area;

[0099] If so, then the first target ROI region is determined to be located in the food storage area.

[0100] In one possible implementation, determining the access action based on the size of the first area and the second area includes:

[0101] If the first area is larger than the second area, then the current action is determined to be a storage action;

[0102] If the first area is smaller than the second area, then the current action is determined to be a take-out action.

[0103] In one possible implementation, the training process of the network model includes:

[0104] Obtain any sample RGB image from the training set, wherein the sample RGB image is pre-labeled with first information of the ROI region;

[0105] The sample RGB image is input into the network model, and the second information of the ROI region in the sample RGB image is output.

[0106] The network model is trained based on the first and second information.

[0107] In one possible implementation, determining that the drawer is in the open state includes:

[0108] Determine the first position information of the drawer edge in the first target RGB image;

[0109] If the first position information is inconsistent with the second position information of the drawer edge when the drawer is closed, then the drawer is determined to be in the open state.

[0110] Figure 8 This application provides a schematic diagram of an electronic device structure. Based on the above embodiments, this application also provides an electronic device, such as... Figure 8 As shown, it includes: processor 801, communication interface 802, memory 803 and communication bus 804, wherein processor 801, communication interface 802 and memory 803 communicate with each other through communication bus 804.

[0111] The memory 803 stores a computer program, which, when executed by the processor 801, causes the processor 801 to perform the following steps:

[0112] The first target ROI region containing the hand region is determined in the first target RGB image acquired, and the second target ROI region containing the hand region is determined in the second target RGB image acquired after the acquisition time of the first target RGB image.

[0113] If the first target ROI region is identified as being located in the food storage area, then the first area of ​​the first target ROI region in the first target RGB image and the second area of ​​the second target ROI region in the second target RGB image are determined.

[0114] The access action is determined based on the size of the first area and the second area.

[0115] In one possible implementation, determining the target ROI region containing the hand region in the acquired target RGB image includes:

[0116] The target RGB image is input into the trained network model, and the network model outputs an RGB image that identifies the target ROI region.

[0117] In one possible implementation, identifying that the first target ROI region is located at the food storage location includes:

[0118] If it is determined that the drawer of the smart refrigerator is in the open state, then the first region of the drawer in the first target RGB image is determined;

[0119] If the first target ROI region is located within the first region, then the first target ROI region is determined to be located in the food storage area.

[0120] In one possible implementation, identifying that the first target ROI region is located at the food storage location includes:

[0121] Based on the second area of ​​the food storage area of ​​the smart refrigerator (excluding drawers) that is saved in advance, determine whether the first target ROI area is located within the second area;

[0122] If so, then the first target ROI region is determined to be located in the food storage area.

[0123] In one possible implementation, determining the access action based on the size of the first area and the second area includes:

[0124] If the first area is larger than the second area, then the current action is determined to be a storage action;

[0125] If the first area is smaller than the second area, then the current action is determined to be a take-out action.

[0126] In one possible implementation, the training process of the network model includes:

[0127] Obtain any sample RGB image from the training set, wherein the sample RGB image is pre-labeled with first information of the ROI region;

[0128] The sample RGB image is input into the network model, and the second information of the ROI region in the sample RGB image is output.

[0129] The network model is trained based on the first and second information.

[0130] In one possible implementation, determining that the drawer is in the open state includes:

[0131] Determine the first position information of the drawer edge in the first target RGB image;

[0132] If the first position information is inconsistent with the second position information of the drawer edge when the drawer is closed, then the drawer is determined to be in the open state.

[0133] Since the principle of the above-mentioned electronic device in solving the problem is similar to that of the access action recognition method, the implementation of the above-mentioned electronic device can refer to the above embodiments, and the repeated parts will not be described again.

[0134] The communication bus mentioned in the above-mentioned electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 802 is used for communication between the above-mentioned electronic device and other devices. The memory can include random access memory (RAM), or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor. The aforementioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processing unit (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0135] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program executable by a processor. When the program is run on the processor, the processor executes the following steps:

[0136] The first target ROI region containing the hand region is determined in the first target RGB image acquired, and the second target ROI region containing the hand region is determined in the second target RGB image acquired after the acquisition time of the first target RGB image.

[0137] If the first target ROI region is identified as being located in the food storage area, then the first area of ​​the first target ROI region in the first target RGB image and the second area of ​​the second target ROI region in the second target RGB image are determined.

[0138] The access action is determined based on the size of the first area and the second area.

[0139] In one possible implementation, determining the target ROI region containing the hand region in the acquired target RGB image includes:

[0140] The target RGB image is input into the trained network model, and the network model outputs an RGB image that identifies the target ROI region.

[0141] In one possible implementation, identifying that the first target ROI region is located at the food storage location includes:

[0142] If it is determined that the drawer of the smart refrigerator is in the open state, then the first region of the drawer in the first target RGB image is determined;

[0143] If the first target ROI region is located within the first region, then the first target ROI region is determined to be located in the food storage area.

[0144] In one possible implementation, identifying that the first target ROI region is located at the food storage location includes:

[0145] Based on the second area of ​​the food storage area of ​​the smart refrigerator (excluding drawers) that is saved in advance, determine whether the first target ROI area is located within the second area;

[0146] If so, then the first target ROI region is determined to be located in the food storage area.

[0147] In one possible implementation, determining the access action based on the size of the first area and the second area includes:

[0148] If the first area is larger than the second area, then the current action is determined to be a storage action;

[0149] If the first area is smaller than the second area, then the current action is determined to be a take-out action.

[0150] In one possible implementation, the training process of the network model includes:

[0151] Obtain any sample RGB image from the training set, wherein the sample RGB image is pre-labeled with first information of the ROI region;

[0152] The sample RGB image is input into the network model, and the second information of the ROI region in the sample RGB image is output.

[0153] The network model is trained based on the first and second information.

[0154] In one possible implementation, determining that the drawer is in the open state includes:

[0155] Determine the first position information of the drawer edge in the first target RGB image;

[0156] If the first position information is inconsistent with the second position information of the drawer edge when the drawer is closed, then the drawer is determined to be in the open state.

[0157] Since the principle of solving the problem using the computer-readable medium provided above is similar to the access action recognition method, the steps implemented after the processor executes the computer program in the computer-readable medium can be referred to the above embodiments, and repeated parts will not be described again.

[0158] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0160] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A smart refrigerator, characterized in that, The smart refrigerator includes: Acquisition unit, configured to acquire a first target RGB image and acquire a second target RGB image; Control unit, the control unit being configured to: The first target ROI region containing the hand region is determined in the first target RGB image acquired, and the second target ROI region containing the hand region is determined in the second target RGB image acquired after the acquisition time of the first target RGB image. If the first target ROI region is identified as being located in the food storage area, then the first area of ​​the first target ROI region in the first target RGB image and the second area of ​​the second target ROI region in the second target RGB image are determined; based on the size of the first area and the second area, the access action is determined. The data acquisition unit is located at the top of the smart refrigerator; Specifically, the control unit is configured as follows: If the first area is larger than the second area, it is determined that the current action is moving away from the acquisition device, and the current action is determined to be a storage action; If the first area is smaller than the second area, it is determined that the current action is approaching the acquisition device, and the current action is determined to be a retrieval action.

2. The intelligent refrigerator according to claim 1, characterized in that, The control unit is specifically configured as follows: The target RGB image is input into the trained network model, and the network model outputs an RGB image that identifies the target ROI region.

3. The intelligent refrigerator according to claim 1, characterized in that, The control unit is specifically configured as follows: If it is determined that the drawer of the smart refrigerator is in the open state, then the first region of the drawer in the first target RGB image is determined; If the first target ROI region is located within the first region, then the first target ROI region is determined to be located in the food storage area.

4. The intelligent refrigerator according to claim 1, characterized in that, The control unit is specifically configured as follows: Based on the second area of ​​the food storage area of ​​the smart refrigerator (excluding drawers) that is saved in advance, determine whether the first target ROI area is located within the second area; If so, then the first target ROI region is determined to be located in the food storage area.

5. The intelligent refrigerator according to claim 2, characterized in that, The control unit is specifically configured as follows: Obtain any sample RGB image from the training set, wherein the sample RGB image is pre-labeled with first information of the ROI region; The sample RGB image is input into the network model, and the second information of the ROI region in the sample RGB image is output. The network model is trained based on the first and second information.

6. The intelligent refrigerator according to claim 1, characterized in that, The control unit is specifically configured as follows: Determine the first position information of the drawer edge in the first target RGB image; If the first position information is inconsistent with the second position information of the drawer edge when the drawer is closed, then the drawer is determined to be in the open state.

7. A method for recognizing access actions, characterized in that, The method includes: The first target ROI region containing the hand region is determined in the first target RGB image acquired, and the second target ROI region containing the hand region is determined in the second target RGB image acquired after the acquisition time of the first target RGB image. If the first target ROI region is identified as being located in the food storage area, then the first area of ​​the first target ROI region in the first target RGB image and the second area of ​​the second target ROI region in the second target RGB image are determined. The access action is determined based on the size of the first area and the second area; The acquisition unit for acquiring the first target RGB image and the second target RGB image is located at the top of the smart refrigerator. The step of determining the access action based on the size of the first area and the second area includes: If the first area is larger than the second area, it is determined that the current action is moving away from the acquisition device, and the current action is determined to be a storage action; If the first area is smaller than the second area, it is determined that the current action is approaching the acquisition device, and the current action is determined to be a retrieval action.

8. An electronic device, characterized in that, The electronic device includes at least a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the steps of the access action recognition method as described in claim 7.

9. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the steps of the access action recognition method as described in claim 7.

Images