Food storage equipment

Abstract

The present application discloses a food storage device. When a preset trigger condition is met, a camera is used to collect an interior picture of a storage room, so as to display the interior picture to a user. The validity of the collected interior picture is determined by judging whether a detector detects an opening signal of a box door during the process of the camera collecting the interior picture. If the detector detects an opening signal of a box door during the process of the camera collecting the interior picture, the collected interior picture is determined to be an invalid picture.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the priority of the Chinese patent application filed on June 23, 2021, with application number 202110699305.6, and invention name “Refrigerator and image acquisition method”, as well as the priority of the Chinese patent application filed on June 23, 2021, with application number 202110697463.8, and invention name “Refrigerator and storage room interior scene picture display method”, and the priority of the Chinese patent application filed on June 23, 2021, with application number 202110699331.9, and invention name “Refrigerator and food entry method”, and the priority of the Chinese patent application filed on September 13, 2021, with application number 202111070757.4, and invention name “Food storage device and image processing method”, and all of their contents are incorporated by reference into this application. Technical Field

[0003] The present application relates to the technical field of food storage equipment, and in particular to a food storage equipment. Background Art

[0004] At present, for food storage equipment products, users will check the food stored in the storage room of the food storage equipment from time to time. Each time the food is checked, the user is required to open and close the door of the food storage equipment, which makes the process of checking the food in the storage room of the food storage equipment cumbersome. In addition, the frequent opening and closing of the door of the food storage equipment also increases the power consumption of the door of the food storage equipment, which in turn increases the working power consumption of the food storage equipment, which does not conform to the mainstream development trend of energy saving of electrical appliances. Summary of the invention

[0005] In a first aspect, the food storage device provided by the present application includes:

[0006] The box body has a storage room inside for storing food;

[0007] A box door, arranged at the opening of the storage room;

[0008] A camera, disposed on one side of the door located inside the storage room, for collecting pictures of the interior of the storage room;

[0009] A detector, used to detect a switch signal of the box door;

[0010] A controller connected to the camera and the detector is configured as follows:

[0011] When a preset trigger condition is met, the camera is controlled to collect an interior picture of the storage room;

[0012] Determine whether the detector detects an opening signal of the door during the process of the camera collecting the interior scene picture;

[0013] If the detector detects an opening signal of the door during the process of the camera collecting the interior picture, it is determined that the collected interior picture is an invalid picture.

[0014] In a second aspect, the present application also provides a food storage device, including:

[0015] A box body, wherein a storage chamber is provided inside, and the storage chamber includes a plurality of storage compartments for storing food;

[0016] A box door, arranged at the opening of the storage room;

[0017] A camera, disposed on one side of the door located inside the storage room, for collecting pictures of the interior of the storage room;

[0018] A display, arranged on the door, for displaying a user interface;

[0019] A controller connected to the camera and the display is configured as follows:

[0020] In response to an input instruction for viewing interior scenes of a storage room, acquiring at least one group of interior scene pictures, each group of interior scene pictures including interior scene pictures of each of the storage compartments acquired at the same time;

[0021] The at least one group of interior scene pictures is displayed in the user interface, wherein the display position relationship between the interior scene pictures in the same group is consistent with the position relationship between the containing grids corresponding to the interior scene pictures.

[0022] In a third aspect, the food storage device provided by the present application includes:

[0023] The box body has a storage room inside for storing food;

[0024] A box door, arranged at the opening of the storage room;

[0025] A camera, disposed on one side of the door located inside the storage room, for collecting pictures of the interior of the storage room;

[0026] A display, arranged on the door, for displaying a user interface;

[0027] A controller connected to the camera and the display is configured as follows:

[0028] In response to an input interior view instruction, displaying an interior view picture of the storage room, wherein the interior view picture includes a food object;

[0029] In response to the input food identification instruction, displaying food icons matching the food objects in the interior picture;

[0030] According to the target ingredient icon selected by the user, the ingredient information corresponding to the target ingredient icon is determined as the ingredient information of the existing ingredients in the storage room.

[0031] In a fourth aspect, the food storage device provided by the present application includes:

[0032] A box body, wherein a storage chamber is provided inside for storing food, wherein the storage chamber is provided with a storage partition and a fixed reference object, wherein the storage partition divides the storage chamber into at least two layers of storage areas, and the fixed reference object vertically spans each layer of storage areas in the storage chamber;

[0033] A box door, arranged at the opening of the storage room;

[0034] A plurality of cameras are arranged on one side of the door located inside the storage compartment and are arranged one-to-one corresponding to the storage areas, and are used to collect interior images of storage areas at different levels in the storage compartment when the door is closed;

[0035] The controller connected to the camera is configured as follows:

[0036] Acquire interior images of storage areas at different levels;

[0037] Acquire reference alignment positions of the fixed reference objects in interior images of different layered object areas;

[0038] Performing alignment correction on interior images of different storage areas according to the reference alignment position;

[0039] A display is arranged on the outside of the box door and is used to present the interior images of the storage areas on different levels after the alignment correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] Figure 1A This is a three-dimensional diagram of a food storage device according to an exemplary embodiment of the present application;

[0042] Figure 1B This is a schematic diagram of the appearance of a food storage device according to an exemplary embodiment of the present application;

[0043] Figure 2 This is a schematic diagram of the hardware configuration of a food storage device according to an exemplary embodiment of the present application;

[0044] Figure 3 A flowchart of a method that a food storage device controller is configured to execute according to an exemplary embodiment of the present application;

[0045] Figure 4 An example is shown Figure 1A The interior view of the storage compartment in the food storage device shown;

[0046] Figure 5a to 5c This is an interior picture shown according to an exemplary embodiment of the present application;

[0047] Figure 6 A flowchart of a method that a food storage device controller is configured to execute according to an exemplary embodiment of the present application;

[0048] Figure 7 A template view is shown according to an exemplary embodiment of the present application;

[0049] Figure 8 This is a flowchart of an image processing method according to an exemplary embodiment of the present application;

[0050] Fig. 9 This is a schematic diagram of an interior image of an area with different layers of objects after grayscale processing according to an exemplary embodiment of the present application;

[0051] Fig.10 A schematic diagram of an interior scene image width adjustment process corresponding to left alignment correction according to an exemplary embodiment of the present application;

[0052] Fig.11 A schematic diagram of an interior scene image width adjustment process corresponding to right alignment correction according to an exemplary embodiment of the present application;

[0053] Fig.12 This is a schematic diagram of an interior image of different layers of storage areas after alignment correction according to an exemplary embodiment of the present application;

[0054] Fig.13 An interior scene display interface shown in this application according to an exemplary embodiment;

[0055] Fig.14 Another interior scene display interface shown in this application according to an exemplary embodiment;

[0056] Fig.15 This is a food management interface displayed on a terminal device according to an exemplary embodiment of the present application;

[0057] Fig.16 An interior scene display interface displayed on a terminal device according to an exemplary embodiment of the present application;

[0058] Fig.17 A flowchart of a method that a food storage device controller is configured to execute according to an exemplary embodiment of the present application;

[0059] Fig.18 An interior scene display interface shown in this application according to an exemplary embodiment;

[0060] Fig.19 An interior scene display interface shown in this application according to an exemplary embodiment;

[0061] Fig. 20 This is a schematic diagram of an interaction process between a food storage device and a server according to an exemplary embodiment of the present application;

[0062] Fig.21 This is another schematic diagram of the interaction process between a food storage device and a server according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0064] In the description of the present invention, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0065] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0066] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0067] Figure 1A This is a three-dimensional diagram of a food storage device according to an exemplary embodiment of the present application. The shape of the food storage device provided by the embodiment of the present invention is approximately a rectangular parallelepiped. Generally, the food storage device can be a refrigerator, a freezer, a beverage cabinet, a food display cabinet, a wine cabinet, etc. that can store food. The food storage device is defined by a storage chamber 110 that provides limited storage space and a door 120 disposed at the opening of the storage chamber. The storage chamber 110 is a box with an opening. Figure 1A In the example shown, the storage chamber 110 includes a freezer chamber 111 located at the bottom and a refrigerator chamber 112 located at the top. The freezer chamber 111 and the refrigerator chamber 112 each have one or more independent storage spaces, such as a drawer-type compartment included in the freezer chamber 111 and a plurality of storage compartments 1121-1123 included in the refrigerator chamber 112, and a top lighting 140 is located inside the storage compartment 1121.

[0068] It should be noted that Figure 1A As an example of an embodiment of the present application, the food storage device shown therein does not constitute a limitation on the food storage device provided by the present application. For example, in another embodiment of the present application, the food storage device box is divided into a left side and a right side, the left side box is defined as a freezer, and the right side box is defined as a refrigerator, and both the freezer and refrigerator can be covered by a door pivotally mounted on the box. For another example, in another embodiment of the present application, the storage room includes a refrigerator, a freezer, and a temperature-changing room.

[0069] Based on the food storage device provided in the above embodiment, one or more cameras are arranged inside the box of the food storage device to collect pictures of the interior of the storage room. In some embodiments, the camera can be arranged on the side of the door located inside the storage room, and the lens is arranged facing the storage room. In this way, when the door is closed, the camera will face the inside of the storage room, so as to capture pictures of the interior of the storage room. It should be understood that a plurality of lighting lamps are arranged on the inner wall of the storage room, including but not limited to top lighting lamps arranged on the top wall of the storage room and side backlights arranged on the side walls of the storage room. In order to provide sufficient lighting conditions for the camera to take pictures of the interior of the storage room, during the operation of the camera, the top lighting lamp is kept on and the side backlight is turned off.

[0070] like Figure 1A As shown, a camera mounting plate 130 is provided at one side edge of the refrigerator door 112B, and cameras 131, 132 and 133 are embeddedly arranged on the camera mounting plate 130. The cameras 131, 132 and 133 are all connected to the controller of the food storage device (see Figure 2 When the refrigerator door 112B is closed, the cameras 131-133 face the storage compartments 1121-1123 included in the refrigerator 112, so the cameras 131-133 can be used to collect interior pictures of the storage compartments 1121-1123. It should be noted that Figure 1A As an example of an embodiment of the present application, the number of cameras, installation positions and installation methods shown therein do not constitute a limitation on the food storage device provided by the present application.

[0071] Based on the food storage device provided in the above embodiment, by arranging a display on the outside of the food storage device box, the food storage device can have a display function.

[0072] Figure 1B This is a schematic diagram of the appearance of a food storage device according to an exemplary embodiment of the present application, such as Figure 1B As shown, a display is provided on the refrigerating chamber door 112B, and the display is embedded in the door body.

[0073] In some embodiments, the plane where the display screen is located is flush with the outer surface of the cabinet door.

[0074] like Figure 1A and Figure 1B As shown, the food storage device provided in the embodiment of the present application has at least two storage chambers inside its box, such as a freezer, a refrigerator, a temperature-changing chamber, etc. Each storage chamber may have multiple independent storage spaces, such as drawer-type compartments and storage compartments.

[0075] Figure 2 FIG. 1 is a schematic diagram of a hardware configuration of a food storage device according to an exemplary embodiment of the present application. Figure 2 As shown, the food storage device 200 may include a display 210, a controller 220, an antenna 230 for detecting an RFID tag, a detector 240, a camera 250, a speaker 260, a memory 270, and a user input interface. The display 210, the antenna 230, the detector 240, the camera 250, the speaker 260, and the memory 270 are coupled to the controller 220 via a communication interface.

[0076] Among them, the display 210 is used to receive the image signal output by the controller 220, display the video content and images and the components of the menu control interface, and display the user control UI interface for controlling the food storage device 200.

[0077] The controller 220 may include one or more processing units, such as a system on a chip (SoC), a central processing unit (CPU), a microcontroller (MCU), a storage controller, etc. Different processing units may be independent devices or integrated into one or more processors.

[0078] In some embodiments, the controller 220 communicates with the antenna 230 via a serial port.

[0079] The memory 270 may include one or more storage units, for example, a volatile memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.; and a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, etc. Different storage units may be independent devices, or may be integrated or packaged in one or more processors or communication interfaces to become a part of the processor or communication interface.

[0080] Program instructions and applications, such as voice applications and food management applications, are stored in the memory 270. The controller 220 can call program instructions or run applications in the memory 270 to enable the food storage device to execute related methods, such as the method provided in the embodiment of the present application.

[0081] Camera 250 is used to capture images of the interior of the storage room, including but not limited to capturing pictures of the interior, for example, it can also capture videos of the interior. Optionally, camera 250 can use a wide-angle lens. Since the focal length of the wide-angle lens is short, a larger area of ​​the interior of the storage room can be captured within a shorter shooting distance. Optionally, camera 250 can use an ordinary wide-angle lens, whose focal length is generally 38 to 24 mm and the viewing angle is generally 60 to 84 degrees, or an ultra-wide-angle lens, whose focal length is generally 20 to 13 mm and the viewing angle is generally 94 to 118 degrees.

[0082] The detector 240 includes at least a sound collector, such as a microphone, which can be used to receive the user's voice. Exemplarily, the microphone collects the user's voice command for controlling the food storage device 200. The detector 240 may also include a door switch state sensor, such as a magnetic switch or a mechanical switch, for detecting the switch signal of each door, and when the switch state of the door is recorded according to the detected switch signal, the switch state information is sent to the controller 220. Exemplarily, when the detector 240 detects an opening signal or a closing signal of any door, the latest switch state information is sent to the controller 220.

[0083] The user input interface includes at least one of a microphone, a touch panel, a sensor, a button, and other input interfaces. For example, the user can implement the user command input function through voice, touch, gesture, pressing, and other actions. The input interface converts the received analog signal into a digital signal, and the digital signal into a corresponding command signal, and sends it to the controller 220.

[0084] In some embodiments, the controller 220 implements the food management function by running the food management application. The application interface provided by the food management application includes: a food management interface that can be used as the main interface of the application, a label editing interface for establishing an association between labels and food information, a food library interface for displaying food information in the preset food library, an food adding interface for adding ingredients, and an interior display interface for displaying the interior scene of the storage room, etc.

[0085] In some embodiments, in response to the start-up or user input of the food management application start-up instruction, the controller 220 starts the food management application. After the food management application is started, the user can enter a label editing instruction in the food management application. In response to the label editing instruction, the food management application scans the label placed by the user on the label reading area through the first antenna to obtain the label information of the label, such as "A", and displays the label editing interface. The user can enter the food information in the label editing interface for association with "A". The food management application receives the food information input by the user and establishes an association relationship between the food information and "A". When the food management application detects that there is a label "A" in the storage room through the second antenna, the real food in the storage room can be determined according to the food information associated with "A", thereby realizing intelligent management of food. It should be understood that when the food management application detects that there is a label "A" in the storage room through the second antenna, the food associated with "A" is the existing food in the storage room.

[0086] In other embodiments, the user can input an ingredient addition instruction in the ingredient management application. In response to the ingredient addition instruction, the ingredient management application displays an ingredient library interface, which includes a large number of optional ingredient icons, each ingredient icon corresponding to an ingredient information. The user can select the ingredients (icons) to be added in the ingredient library interface. The ingredient management application determines the ingredient information corresponding to the ingredient icon selected by the user as the ingredient information of the existing ingredients in the storage room. For example, the ingredient information corresponding to the ingredient icon selected by the user is added to the local ingredient information collection, which is a collection of ingredient information of the existing ingredients in the storage room. The aforementioned ingredient addition operation enters the ingredient information of the real ingredients into the ingredient management application without affixing labels on the real ingredients, thereby realizing intelligent management of ingredients in the ingredient storage device through the ingredient management application.

[0087] In some embodiments, the ingredient information includes one or more of ingredient ID, ingredient name, storage location identifier, ingredient quantity, ingredient shelf life, start time, etc. The storage room where the ingredient is stored can be determined based on the storage location identifier. For example, if the storage location identifier contained in the ingredient information "apple" is "1", it is determined that "apple" is stored in the refrigerator, and if the storage location identifier contained in the ingredient information "apple" is "2", it is determined that "apple" is stored in the freezer. The freshness of the ingredient can be determined based on the current time, the shelf life of the ingredient, and the start time. Among them, the start time can be the actual production date of the ingredient entered by the user, such as the production date of bread on May 1, 2021, or it can be the time when the ingredient is placed in the storage room.

[0088] In some embodiments, the food information corresponding to the existing food in the storage room is stored in the local food information set. The existing food refers to the food currently stored in the food storage device.

[0089] In other embodiments, the above-mentioned local food information set may also store food information corresponding to historical food. The historical food refers to food that was stored in the food storage device in the past and has been removed from the food storage device. In specific implementation, the existing food and the historical food may be distinguished by storage status marking, for example, the food information of the historical food is marked as removed, and the food information of the existing food is marked as put in.

[0090] In other embodiments, the ingredient information of the current ingredients and the ingredient information of the historical ingredients are stored in different ingredient information sets.

[0091] In some embodiments, after the food management application is started, the food information of the existing food in the local food information set is read, for example, the food information of the food whose storage state is the put-in state is read. Then, the food management interface is displayed according to the food information of the existing food, and the food management interface includes food icons drawn according to the food information, so that the existing food in the food storage device is fully displayed in the form of icons, graphics, etc. In addition, various functional controls can also be displayed in the food management interface, such as label editing controls, food adding controls, recipe recommendation controls, interior view controls, etc., for users to operate to trigger the controller 220 to implement corresponding functions.

[0092] In some embodiments, the food management interface may include multiple food display areas, one food display area corresponding to one storage room. For example, food icons of food in the refrigerating room are displayed in the food display area corresponding to the refrigerating room.

[0093] In some embodiments, the user can input an interior viewing instruction by operating the interior viewing control in the food management interface. The food management application collects interior pictures of the storage room through a camera, and displays the interior pictures to the user. Also, the food object in the interior picture is identified based on image recognition technology to obtain food information of the existing food in the storage room. It is worth noting that when the food object in the interior picture is a new food, the function of automatically adding food can be realized based on the food information obtained by image recognition, thereby omitting the user operation of manually entering food information; in addition, the existing food in the storage room can be displayed to the user based on the food information obtained by image recognition, thereby improving the user experience.

[0094] It should be understood that a storage room usually includes a plurality of independent storage spaces, the interior view of each storage space is the partial interior view of the storage room, and the interior view image of each storage space is the partial interior view image of the storage room. Therefore, the partial interior view image of the storage room can be obtained by taking the interior view image of each storage space, and the partial interior view or the global interior view of the storage room can be reflected by the interior view images of part or all of the storage spaces.

[0095] by Figure 1A Taking the food storage device shown as an example, cameras 131-133 are installed on the refrigerator door. When the refrigerator door is closed, the lenses of the cameras 131-133 are respectively facing the storage compartments 1121-1123. The food management application can collect the interior pictures corresponding to the storage compartments 1121-1123 through the cameras 131-133, and use the group of interior pictures as the interior pictures of the refrigerator.

[0096] In some embodiments, the controller 220 executes the food management application. Figure 3 The following steps are shown in:

[0097] S310, when a preset trigger condition is met, the camera is controlled to collect an interior picture of the storage room.

[0098] In some embodiments, when the closing signal of the box door is detected, the camera is controlled to collect the interior view of the storage room. It is worth noting that the opening and closing signal of the box door can be selectively monitored according to the setting position of the camera. Figure 1A In the food storage device shown, since the camera is set on the refrigerator door, the food management application can detect the switch signal of the refrigerator door without detecting the switch signals of other doors. When the closing signal of the refrigerator door is detected, the cameras 131-133 are controlled in sequence to collect the interior pictures corresponding to the storage compartments 1121-1123 to obtain a group of interior pictures. It should be understood that a group of interior pictures are interior pictures of different parts of the storage room collected in the same time period. For example, the interior pictures corresponding to the storage compartments 1121-1123 collected in the same time period constitute a group of interior pictures.

[0099] In specific implementation, the controller 220 detects the switch signals of each door through the detector 240. Optionally, when the detector 240 detects the opening signal or closing signal of any door, the switch state of the door is recorded. The switch state information is sent to the controller, and the switch state information includes the switch states of each door for the last two times, which are the previous state and the subsequent state. The controller determines whether the opening signal or closing signal of the door is detected according to the switch state information. For example, when the previous state of a door is the opening state and the subsequent state is the closing state, it is determined that the closing signal of the door is detected; when the previous state of a door is the closing state and the subsequent state is the opening state, it is determined that the opening signal of the door is detected; when the previous state and the subsequent state of a door are the same state, it is determined that the switch signal of the door is not detected. Exemplarily, the contents of each field in the switch state information sent by the detector to the controller are as follows, wherein the controller can distinguish the doors according to the field "Index".

[0100]

[0101] In other embodiments, when the current time reaches a preset time point or the interval time reaches a preset time, the camera is controlled to collect pictures of the interior of the storage room. For example, the food management application controls the camera to collect pictures of the interior of the storage room at 8:00, 12:00, and 18:00 every day. For another example, the food management application controls the camera to collect pictures of the interior of the storage room every 24 hours.

[0102] In other embodiments, when an input interior picture acquisition instruction is received, the camera is controlled to acquire an interior picture of the storage room. The user may input the interior picture acquisition instruction by clicking an operation control on the food management interface or by inputting a voice command.

[0103] In an optional implementation of S310, the food management application includes a service for controlling the camera in the background, and the food management application controls the camera to collect pictures of the interior of the storage room through the background service. Exemplarily, when a preset trigger condition is met, the background service turns on the camera; after the camera is turned on for a first time, the camera is controlled to take pictures of the interior of the storage room. In this way, the frame loss problem and low brightness problem that are very likely to occur after the camera is just turned on can be avoided, and the picture shooting effect can be ensured. Exemplarily, the first time length is 1 second.

[0104] It is worth noting that the control logic of the food management application for the camera is related to factors such as the number of cameras and their location. Figure 1A The food storage devices with different structures shown in the examples may have different control logics for the camera in the food management application. For example, if the camera can slide along the mechanical sliding track on the camera mounting plate 130 (see the above embodiment for details), then when the preset trigger condition is met, it is necessary to control the camera to move to the position opposite to each storage compartment in turn, and then control the camera to take pictures of the interior of the corresponding storage compartment, which will not be described in detail here.

[0105] In some embodiments, the operating system installed on the food storage device is an Android system. Based on the working mechanism of the Android system, after the camera is turned on, it is necessary to output a preview picture before taking pictures of the interior. Based on this, after the food management application turns on the camera, it controls the window size to be smaller than the preset size, displays the preview window in the area of ​​the user interface outside the display viewport range, and outputs the camera's preview picture to the preview window. Since the size of the preview window is smaller than the preset size and is displayed outside the display viewport range, the camera preview picture can be output without the user's perception, preventing the preview window from affecting the user's viewing of the current user interface and the operation of the controls in the user interface.

[0106] Exemplarily, the size of the preview window can be set to 1×1 (smaller than the preset size), and the margin value can be set to a negative value. It should be understood that the aforementioned margin value represents the distance between the display position of the preview window and the border of the user interface. When the margin value is a negative value, the preview window will be displayed outside the display viewport of the user interface.

[0107] S320, determining whether the detector detects an opening signal of the door during the process of the camera collecting the interior scene image.

[0108] In some embodiments, due to the location of the camera, the camera can only face the interior of the storage compartment when the door is closed, and when the door is open, the camera will also shift as the door status changes. Figure 1A In the food storage device shown as an example, cameras 131-133 are respectively opposite to the containing compartments 1121-1123 when and only when the refrigerator door is in a closed state. If the detector detects the opening signal of the refrigerator door during the process of the camera collecting interior pictures, it means that the user has opened the refrigerator door, causing the camera to not be opposite to the containing compartment. In this way, the interior pictures collected by the camera are very likely to be invalid pictures. The invalid picture refers to a picture that cannot meet the image recognition requirements and the interior display requirements, such as a picture that does not clearly and completely record the image of the food inside the storage room. Correspondingly, a valid picture refers to a picture that does not meet the image recognition requirements and the interior display requirements. It is worth noting that if the user opens the door during the process of the camera collecting interior pictures, the camera can easily capture the user and the family scene, resulting in the leakage of user privacy and troubling the user.

[0109] Based on this, by judging whether the detector detects the opening signal of the box door during the process of the camera collecting interior pictures, it is determined whether the user has opened the box door during the process of the camera collecting interior pictures; if the user has opened the box door during the process of the camera collecting interior pictures, the interior picture collected this time can be determined as an invalid picture.

[0110] In some optional implementations, the switch status of the door is monitored in real time during the operation of the camera. When the door is detected to be open, the camera is turned off. For example, the food management application polls the door switch status information sent by the detector to the controller to obtain the real-time switch status of the door. When it is obtained that a door changes from a closed state to an open state, and the door is a door with a camera, the camera is turned off. For example, for Figure 1A The food storage device shown turns off the camera when it is detected that the refrigerator door changes from a closed state to an open state.

[0111] In some other optional implementations, when the interior picture acquisition is completed, the completion time is recorded. Based on the latest door switch status information sent by the detector, determine the time when the detector last detected the door opening signal, that is, the door opening time, which is earlier than the aforementioned completion time. Then determine the time difference between the completion time and the aforementioned door opening time. If the time difference is less than the preset time difference, it is determined that the camera detected the door opening signal during the process of acquiring the interior picture; if the time difference is not less than the preset time difference, it is determined that the camera did not detect the door opening signal during the process of acquiring the interior picture. Among them, the preset time difference can be determined according to the time required for the camera to acquire the interior picture. For example, if the camera needs 10 seconds of acquisition time, then the preset time difference can be 10 seconds. In addition, the interior picture acquisition can be considered to be completed when the last interior picture taken by the camera is obtained. For example, with Figure 1A Taking the food storage device shown as an example, assuming that the control order of the food management application for cameras 131-133 is camera 131, camera 132, and camera 133, then the interior picture taken by camera 133 is the last interior picture, and when the interior picture taken by camera 133 is obtained, it is considered that the interior picture collection is completed. Alternatively, it is also possible to consider that the interior picture collection is completed when the camera is turned off. Furthermore, the recorded completion time can be the shooting time of the last interior picture, or the camera turning off time.

[0112] In some other optional implementations, it is determined whether the door opening signal is detected based on the door opening and closing status information received during the working time period of the camera. Figure 1A Taking the food storage device shown as an example, it is assumed that the control order of the food management application for cameras 131-133 is camera 131, camera 132, and camera 133. When the food management application turns on camera 131, it records the turn-on time, and when the camera 133 is turned off, it records the turn-off time, and then determines the working time period of the camera according to the turn-on time and the turn-off time. After turning off camera 133, the cabinet door switch status information received during the aforementioned working time period is obtained. If the acquisition result is empty, it is determined that the detector did not detect the opening signal of the cabinet door during the process of the camera collecting the interior picture. If the acquisition result is not empty, and the cabinet door switch status information shows that the refrigerator door changes from a closed state to an open state, it is determined that the detector detected the opening signal of the cabinet door during the process of the camera collecting the interior picture.

[0113] It can be seen that the above embodiment determines the validity of the collected interior picture by judging whether the detector detects the opening signal of the box door during the process of the camera collecting the interior picture. It should be noted that in other embodiments of the present application, other methods can also be used to analyze the validity of the interior picture. For example, based on image recognition technology, it is identified whether the interior picture contains specific image features. If it contains specific image features, the collected interior picture is determined to be a valid picture, otherwise, the collected interior picture is determined to be an invalid picture. Among them, the specific image features can be storage room interior features shared by a certain number of valid pictures, such as the features of the partitions that constitute the storage compartment, including but not limited to the shape, color, size and other features of the partitions.

[0114] In S320, if it is determined that the detector detects the opening signal of the door during the process of the camera collecting interior pictures, S330 is executed. If it is determined that the detector does not detect the opening signal of the door during the process of the camera collecting interior pictures, S340 is executed.

[0115] It is worth noting that in some embodiments, if the camera setting position is not affected by the door switch state, the step of determining the validity of the interior image can be skipped, that is, S320-S340 can be skipped, and the interior image processing process can be directly executed. For example, in some embodiments, the flow is transferred from S310 to S340. Figure 6 S610 in, or flow to Fig.17 In S171. In some embodiments, if the location of the camera is not affected by the door switch state, the validity of the interior view image is determined by other means.

[0116] S330: If the detector detects an opening signal of the door during the process of the camera collecting the interior scene picture, it is determined that the collected interior scene picture is an invalid picture.

[0117] Based on Figure 1A In the example of the food storage device shown, if an opening signal of the refrigerator door is detected during the process of collecting interior pictures, the interior pictures taken by the cameras 131-133 during this time period are all determined to be invalid pictures.

[0118] In some optional implementations, when it is determined that the interior picture is an invalid picture, the invalid picture is deleted to avoid privacy disclosure and to avoid occupying storage space of the food storage device.

[0119] In some embodiments, when it is determined that the collected interior picture is an invalid picture, the camera is controlled to re-collect the interior picture of the storage room. In these embodiments, determining that the collected interior picture is an invalid picture is a preset trigger condition for controlling the camera to collect the interior picture.

[0120] S340: If the detector does not detect an opening signal of the door during the process of the camera collecting the interior scene picture, it is determined that the collected interior scene picture is a valid picture.

[0121] Based on Figure 1A In the example of the food storage device shown, if the opening signal of the refrigerator door is not detected during the process of collecting interior pictures, the interior pictures taken by the cameras 131-133 during this time period are all determined as valid pictures.

[0122] In other embodiments, when it is determined that no door opening signal is detected during the process of collecting interior pictures, it is further determined whether the format size of each interior picture is a preset format size, or whether the size of each interior picture is greater than a preset minimum value. If the format size of each interior picture is the preset format size, or the size of each interior picture is greater than the preset minimum value, then the collected interior pictures are determined to be valid pictures. If the format size of at least one interior picture is not the preset format size, or the size is not greater than the preset minimum value, then the group of interior pictures is determined to be invalid pictures. Figure 1A In the example of the food storage device shown, if the format size of the interior pictures taken by the cameras 131-133 are all preset format sizes, or the sizes are all greater than the preset minimum value, then it is determined that the group of interior pictures are all valid pictures. If the format size of at least one interior picture among the interior pictures collected by the cameras 131-133 is not the preset format size, or the size is not greater than the preset minimum value, then it is determined that the group of interior pictures are all invalid pictures. In these embodiments, by judging the format size or size of each interior picture, the accuracy of judging the validity of the picture can be further improved.

[0123] In the embodiment of the present application, when it is determined that the interior picture is a valid picture, the valid picture is stored.

[0124] It can be seen from the above embodiments that the food storage device provided in the embodiments of the present application, when the preset trigger conditions are met, collects the interior pictures of the storage room through the camera, so as to display the interior pictures to the user. The validity of the collected interior pictures is determined by judging whether the detector detects the opening signal of the box door during the process of the camera collecting the interior pictures. If the detector detects the opening signal of the box door during the process of the camera collecting the interior pictures, it is determined that the collected interior pictures are invalid pictures. Not only can it be ensured that the collected interior pictures can meet the image recognition requirements and the interior display requirements, but it can also avoid leaking user privacy.

[0125] In some embodiments, in order to enable the camera to capture the full view of the interior of the storage compartment, the camera's shooting mode is set to a wide-angle mode. However, this results in that when photographing a certain storage compartment, the upper storage compartment and / or the lower storage compartment of the storage compartment are also photographed, resulting in redundant areas in the photographed interior pictures. Alternatively, since the partitions between the storage compartments of the refrigerator are transparent, when photographing the images inside the partitions, the ingredients in the storage compartments above the partitions will also be captured in the image. In order to remove redundant areas in the interior pictures, the interior pictures can be cropped according to preset cropping rules so that only the key areas of the interior pictures are retained. The key area is the image area corresponding to the storage compartment.

[0126] It should be noted that, since the size and position of the key area in the interior picture are different when the camera is installed at different positions, shooting focal lengths and / or shooting angles, the above-mentioned preset cropping rules can be determined based on the camera's installation position, shooting focal length and shooting angle.

[0127] by Figure 1A The camera 132 shown in FIG. Figure 4 ,and Figure 5a to 5c . Figure 4 Shows Figure 1A The interior view of the storage compartments 1121-1123 in the food storage device shown in FIG. If the lower side boundary of the shooting range of the camera 132 is flush with the bottom partition of the storage compartment 1122, the interior view picture taken will be as follows: Figure 5a As shown, it can be seen that the interior Figure 5a In the example, area a1 is the image area corresponding to the accommodation grid 1122, that is, the key area, and area a2 is the redundant area. Figure 5a , the redundant area a2 can be removed by using the bottom cropping rule. If the upper boundary of the shooting range of the camera 132 is flush with the top of the containing grid 1122, the interior scene image captured will be as follows Figure 5b As shown, it can be seen that the interior Figure 5b In the example, area b1 is the image area corresponding to the accommodation grid 1122, that is, the key area, and area b2 is the redundant area. Figure 5b , the upper cropping rule can be used to remove the redundant area b2. If the lower boundary of the shooting range of the camera 132 is lower than the bottom partition of the containing grid 1122, and the upper boundary is higher than the top partition of the containing grid 1122, the interior scene picture taken will be as follows: Figure 5c As shown, it can be seen that the interior Figure 5c In the example, area c1 is the image area corresponding to the accommodation grid 1122, that is, the key area, and areas c2 and c3 are both redundant areas. Figure 5c , the center cropping rule can be used to remove the redundant areas c2 and c3.

[0128] It is understandable that the cropping rules used for the interior pictures of different storage cells may be the same or different. For example, the top cropping rule is used to crop the interior pictures of storage cells 1121 and 1122, and the center cropping rule is used to crop the interior picture of storage cell 1123.

[0129] It can also be understood that, since the sizes of different storage grids can be the same or different, the focal lengths used by the camera when shooting the interior scenes of different storage grids can be the same or different, and therefore the sizes of key areas in different interior scene pictures are the same or different, that is, the sizes of the cropped interior scene pictures are the same or different.

[0130] In some embodiments, after removing the redundant areas of each interior scene image, a position identifier is assigned to each interior scene image. The position identifier is used to characterize the relevant information of the containing cell, such as the position, name, etc. The position identifier can specifically be a character containing the relevant information of the containing cell, or a character associated with the relevant information of the containing cell. Since a group of interior scene images respectively correspond to different containing cells, different interior scene images in a group of interior scene images have different position identifiers. Furthermore, the containing cell corresponding to the interior scene image can be determined based on the position identifier. Exemplarily, for Figure 1A In the food storage device shown in FIG, a first position identifier is assigned to the interior picture of the storage compartment 1121, a second position identifier is assigned to the interior picture of the storage compartment 1122, and a third position identifier is assigned to the interior picture of the storage compartment 1123.

[0131] In some embodiments, the same time stamp is assigned to the interior pictures taken in the same time period, and the time stamp is used to represent the shooting time of the group of interior pictures.

[0132] In some embodiments, the image name of the interior image is generated according to the location identifier of the interior image, so that its location identifier is included in its image name. Exemplarily, the image name of the interior image of storage compartment 1121 may be "01. interior of the refrigerator"; the image name of the interior image of storage compartment 1122 may be "02. interior of the refrigerator"; the image name of the interior image of storage compartment 1123 may be "03. interior of the refrigerator". "01", "02" and "03" in the aforementioned image names are location identifiers.

[0133] In some embodiments, the picture name of the interior picture is generated according to the position identifier and time identifier of the interior picture, so that its position identifier and time identifier are included in its picture name. Exemplarily, the picture name of the interior picture of the storage compartment 1121 can be "01.202106051200"; the picture name of the interior picture of the storage compartment 1122 can be "02.01.202106051200"; the picture name of the interior picture of the storage compartment 1123 can be "03.01.202106051200". The "202106051200" in the aforementioned picture names is the time identifier, which represents the time of 12:00 on June 5, 2021.

[0134] In some embodiments, the interior pictures with the above-mentioned picture names are saved in a collection of interior pictures. Among them, interior pictures with the same time stamp constitute a group of interior pictures. The position marks of the interior pictures in each group of interior pictures are different. The interior pictures in the collection can be arranged in chronological order. When it is necessary to show the interior of the food storage device to the user, one or more groups of interior pictures with the most recent shooting time can be extracted from the collection according to the time stamp or the picture storage time, and displayed in the user interface.

[0135] Optionally, when the number of pictures in the above interior picture collection exceeds an upper threshold, one or more groups of interior pictures with the earliest shooting time in the collection are deleted according to the time stamp to ensure that the number of pictures in the collection is always less than the upper threshold. Optionally, the food management application periodically detects whether expired pictures are generated in the interior picture collection according to the time stamp, and deletes the expired pictures when they are generated in the collection. The expired picture refers to a picture whose shooting time and the time interval between the current time exceed a preset interval. Exemplarily, the preset interval can be 7 days, so that the food management application will regularly delete the interior pictures in the collection that were taken 7 days ago.

[0136] In some embodiments, the interior pictures with the above-mentioned picture names are saved in a collection of interior pictures, and at the same time, the interior pictures with the above-mentioned picture names are sent to the server together with the identity information of the food storage device. Based on the identity information, the server stores the group of interior pictures under the path associated with the identity information. Among them, the identity information of the food storage device can be the account information for logging into the food management application on the food storage device, or it can be the device information of the food storage device. Optionally, before sending the interior pictures to the server, a compression operation is performed on the interior pictures to be sent to reduce the size of the interior pictures. Afterwards, the compressed interior pictures are sent to the server together with the identity information of the food storage device to reduce the amount of data transmission, save resources, and save storage space.

[0137] In some embodiments, the user can control the food storage device to display a picture of the interior view of the storage room by inputting an instruction to view the interior view of the storage room.

[0138] The controller 220 is configured to execute Figure 6 The following steps are shown:

[0139] S610: In response to an input storage interior scene viewing instruction, at least one group of interior scene pictures is acquired, each group of interior scene pictures including interior scene pictures of each storage compartment collected in the same time period.

[0140] In some embodiments, the user enters a storage room interior view viewing instruction in the food management application. The user can also use the voice control function of the food storage device to enter the storage room interior view viewing instruction.

[0141] In some optional implementations, the food management application obtains one or more sets of interior pictures that have been recently collected from the interior picture collection according to the time stamps of the interior pictures or the picture storage time.

[0142] In some other optional implementations, the food management application obtains one or more sets of recently collected interior pictures from the server according to the identity information of the food storage device. For example, the food management application sends a picture acquisition request to the server, and the picture acquisition request includes the identity information of the food storage device. The server searches for one or more sets of recently collected interior pictures from the associated path according to the identity information and returns them to the food management application.

[0143] S620: Display at least one group of interior scene images in the user interface, wherein the display position relationship between the interior scene images in the same group is consistent with the position relationship between the containing cells corresponding to the interior scene images.

[0144] In some optional implementations, at least one template view is displayed in a user interface, each template view includes multiple image display positions, and the positional relationship between the multiple display positions is consistent with the positional relationship between the multiple containing cells. For each group of interior scene images, for the position identifier of each interior scene image, the containing cell corresponding to each interior scene image is determined; and according to the containing cell corresponding to each interior scene image, the display position corresponding to each interior scene image is determined. Thus, when each interior scene image is displayed in a corresponding display position, the positional relationship between each interior scene image is consistent with the positional relationship between the corresponding containing cells.

[0145] It should be noted that the size of the picture display position can be set according to the size of the interior picture. For example, the size of the picture display position can be completely equal to the size of the interior picture, and the width or height of the picture display position can also be greater than the width or height of the interior picture.

[0146] In some optional implementations, the template view also includes a spacing area set between the image display positions, and the spacing area is used to simulate the layer partitions of the containing grid. The template view may include one or more layers, and a template view including the image display positions and the spacing area may be generated by setting the color, transparency and other parameters of the layer. By setting the color, transparency and other parameters of the spacing area, the display effect of the spacing area can simulate the effect of the layer partitions of the containing grid, so that the presentation effect of the interior picture is more realistic and three-dimensional. For example, the color of the spacing area is set to the same color as the layer partitions of the containing grid, and the height of the spacing area can be set to be equal to the thickness of the layer partitions of the containing grid.

[0147] Exemplarily, the template view may include a first layer and a second layer, and the first layer is located on the upper layer of the second layer. The first layer is a transparent layer, including a preset picture display area, i.e., a picture display position, for setting the interior picture. There is a spacing area of ​​a preset size between adjacent picture display areas. The transparency of the second layer is 0%, and the filling color is black. When the interior picture is displayed in the picture display area of ​​the first layer, the area corresponding to the picture display area in the second layer is covered by the interior picture, and the area in the second layer not covered by the interior picture forms a black spacing area, which can simulate the layer partition effect of the containing grid.

[0148] In another exemplary embodiment, the template view includes a layer, which includes a preset picture display area, and there is a preset size interval between adjacent picture display areas. The transparency of the layer is 0%, and the fill color is black. When the interior picture is displayed in the picture display area, the interior picture covers the picture display area, and the area not covered by the interior picture forms a black interval area, which can simulate the effect of a compartment partition.

[0149] Figure 7 This is a template view according to an exemplary embodiment of the present application. The template view includes display positions 701, 702 and 703. The positional relationship between the display positions 701, 702 and 703 is as follows: Figure 1AThe positional relationship between the storage compartments 1121, 1122 and 1123 of the food storage device shown is consistent, and they are all upper-middle-lower relationships. Therefore, the display position 701 corresponds to the storage compartment 1121, the display position 702 corresponds to the storage compartment 1122, and the display position 703 corresponds to the storage compartment 1123. There is a spacing area 704 between the display positions 701 and 702, and the spacing area 704 can simulate the effect of a partition between the storage compartments 1121 and 1122. There is a spacing area 705 between the display positions 703 and 704, and the spacing area 705 can simulate the effect of a partition between the storage compartments 1122 and 1123. When the interior pictures of the storage compartments 1121, 1122 and 1123 are displayed in the display positions 701, 702 and 703 respectively, the positional relationship of the interior pictures is consistent with the positional relationship between the storage compartments, and they are all upper-middle-lower relationships.

[0150] In some embodiments, the controller 220 can stitch the interior images of different storage areas acquired by different cameras. The display 210 is used to receive the stitched interior images of different storage areas in the storage room output by the controller 220, and to display the stitched interior images to show the actual interior scene of the storage room.

[0151] Alternatively, the controller 220 may also directly transmit the interior images of the storage areas on different levels acquired by different cameras to the display 210, and the display 210 displays the interior images of the storage areas on different levels. For example, the display 210 may be a display with an integrated touch function, and the user may select the interior image of one or more storage areas to be viewed by touching the display 210.

[0152] Figure 8 FIG. 2 is a flow chart of an image processing method according to an exemplary embodiment of the present application. In some embodiments, the controller 220 executes Figure 8 The following steps are shown in:

[0153] S810: Acquire interior images of storage areas on different levels.

[0154] In the example shown in FIG1 , in order to obtain the interior images of the storage areas at different levels in the storage room, a camera panel can be embedded on the inside of the left or right food storage device door. For example, the camera panel is embedded with three cameras 131, 132 and 133 from top to bottom. The heights of the three cameras 131, 132 and 133 correspond to the three storage areas 1121, 1122 and 1123 in the food storage device, respectively. Each camera is used to collect the interior images of the corresponding storage area. When the user completes the operation of placing or taking out the food and closes the door of the food storage device, the lighting 140 on the top of the food storage device will be delayed for a certain period of time, for example, 20 seconds before turning off, to provide a good lighting environment for the camera to take pictures. In order to prevent the camera from taking pictures against the light, resulting in unclear halos in the photos, the backlight of the food storage device will be turned off immediately after the door of the food storage device is closed. At the same time, after learning that the refrigerator door of the food storage device is closed, the controller will initiate a photo-taking process, that is, triggering each camera in a certain order to take pictures of the interior of the three-layer storage area in the food storage device. The triggering order of the camera can be set to correspond to the order in which the backlight is turned off.

[0155] In some embodiments, the camera takes pictures of storage areas on different levels in the storage room and transmits interior images of the storage areas on different levels to the controller. After the controller obtains the interior images of the storage areas on different levels, it first performs grayscale processing on the interior images of the storage areas on different levels. Fig. 9 It is a schematic diagram of interior images of object storage areas at different levels after grayscale processing according to an exemplary embodiment of the present application. Fig. 9 The three pictures in the middle are interior images of the first-floor storage area 1121 to the third-floor storage area 1123 of the storage room from top to bottom.

[0156] Fig. 9 In the example shown, when multiple cameras are used to shoot storage areas on different levels, the positions of the light sensors of different cameras on the circuit board may have tolerances, the lens installation positions may have tolerances, the circuit board and housing assembly may have tolerances, and the positions of multiple cameras on the elastic plastic vertical partitions may have large tolerances due to heat or extrusion. These factors ultimately lead to different deviations in the shooting angles of different cameras. It is difficult to ensure the consistency of the relative positions of objects across the shelves inside the food storage device after imaging. There are fault problems in the interior images of storage areas on different levels. For example, Figures 1 and Fig. 9 The back plate 150 shown spans across the storage areas of each layer. Fig. 9 As shown, the left and right boundaries of the back panel 150 are obviously not aligned, and there is a discontinuity problem in the interior images of the back panel 150 in different storage areas, which affects the display effect of the final interior image.

[0157] S820: Obtain reference alignment positions of fixed reference objects in interior images of object storage areas at different levels.

[0158] The fixed reference object is arranged inside the storage room, and the fixed reference object vertically spans each layer of storage area in the storage room, that is, each layer of storage area is provided with a part of the fixed reference object, and the fixed reference object is arranged corresponding to each layer of storage area in the storage room along the vertical direction. In some embodiments, the reference alignment position of the fixed reference object in the interior image of the storage area at different layers is obtained, and the position on the fixed reference object that is located in the same straight line along the vertical direction and located in the storage area at different layers can be determined as the reference alignment position.

[0159] Exemplarily, the fixed reference object can be a marking line extending in the vertical direction. The center position of the marking line portion corresponding to the different storage areas can be set as the reference alignment position. Specifically, the center position of the marking line portion corresponding to the first storage area, the center position of the marking line portion corresponding to the second storage area, and the center position of the marking line portion corresponding to the third storage area can be set as the reference alignment positions of the fixed reference objects in the interior images of the different storage areas, respectively.

[0160] It should be noted that a non-central position on the marking line that is located on the same straight line in the vertical direction and located in different storage areas may also be set as a reference alignment position, and this embodiment of the present application does not specifically limit this.

[0161] Exemplarily, the fixed reference object may also be a physical structure in the food storage device. In the example shown in FIG. 1 , the fixed reference object may be set as a back panel 150, which is a whole plate-like structure arranged inside the storage chamber. The back panel 150 vertically spans each storage area in the storage chamber. The positions of the back panel 150 located in the same straight line along the vertical direction and located in different storage areas may be set as reference alignment positions. For example, the center position of the back panel 150 in the interior image of different storage areas may be used as the reference alignment position.

[0162] It should be noted that a non-central position on the back plate 150 that is located in the same straight line in the vertical direction and in different storage areas may also be set as a reference alignment position, and the embodiment of the present application does not specifically limit this. Fig. 9 In the example shown, the reference alignment positions on the back plate 150 of the interior images of the different storage areas corresponding to the three images can be intuitively seen. For example, the center position of the back plate 150 in the interior images of the different storage areas is not aligned, and there will be a fault problem on the back plate 150 in the interior images of the different storage areas, which affects the display effect of the final interior image.

[0163] It should be noted that the above embodiment only illustrates the fixed reference object by taking the marking lines and the back panel as examples, and does not limit the specific implementation form of the fixed reference object. Any structure or line that spans the storage areas on each layer in the storage room can be used as the fixed reference object of the embodiment of the present application.

[0164] In some embodiments, the fixed reference object is a back panel 150 disposed inside the storage room. To obtain the reference alignment position of the fixed reference object in the interior images of different storage areas, the left and right boundaries of the back panel 150 in the interior images of different storage areas can be obtained first, and then the center position of the back panel 150 can be obtained based on the left and right boundaries. The interior images of the different storage areas can be aligned and corrected based on the reference alignment position. The interior images of the different storage areas can be aligned and corrected based on the reference alignment position so that the center position of the back panel 150 is longitudinally aligned. That is, the center position of the back panel 150 in the interior images of different storage areas can be aligned and corrected so that the center position of the back panel 150 is longitudinally aligned.

[0165] In some embodiments, the left and right boundaries of the back plate 150 in the interior images of the different storage areas can be obtained by obtaining vertical lines in which the number of non-zero pixels in the interior images of the different storage areas is greater than or equal to a set threshold. Exemplarily, the Hough transform algorithm can be used to obtain vertical lines in which the number of non-zero pixels in the interior images of the different storage areas is greater than or equal to a set threshold.

[0166] Specifically, the Hough transform algorithm can separate geometric shapes with certain common features from the image, such as geometric shapes such as straight lines or circles in the separated image, and can effectively reduce noise interference. The left and right boundaries of the back plate 150 should be displayed as vertical lines in the interior image. In order to obtain the left and right boundaries of the back plate 150 in the interior images of different storage areas, the embodiment of the present application needs to use the Hough transform algorithm to separate the vertical lines in the interior images of each storage area. The specific process is as follows:

[0167] Convert the rectangular coordinate system to the polar coordinate system. For any point A(x0,y0) in the rectangular coordinate system, the straight line passing through point A satisfies the equation Y0=k*X0+b, where k is the slope and b is the intercept. The straight line cluster passing through point A(x0,y0) in the XY plane can be represented by Y0=k*X0+b, but for vertical lines, their slope is infinite, so vertical lines cannot be represented using the rectangular coordinate system, and the rectangular coordinate system needs to be converted to the polar coordinate system.

[0168] The equation for a straight line in the polar coordinate system is ρ=xcos(θ)+ysin(θ), where ρ is the distance from the origin to the straight line, and θ is the polar angle of the corresponding point. In polar coordinates, ρ=xcos(θ)+ysin(θ) represents a straight line, where (ρ,θ) determines a straight line. Therefore, each non-zero pixel point on the interior image of each layer of the storage area can be transformed into a straight line under polar coordinates, and then a hist(ρ,θ) two-dimensional histogram is constructed. The points in the two-dimensional histogram where θ is equal to π / 2 are located on the vertical lines. The number of non-zero pixels on the vertical line can be determined based on the value of ρ. In the two-dimensional histogram, find the vertical lines whose number of non-zero pixels is greater than or equal to the set threshold, which are the vertical lines in the interior images of different layers of storage areas that need to be detected in the embodiment of the present application. Exemplarily, the set threshold may be equal to but not limited to 30, that is, vertical lines containing more than 30 non-zero pixels in the interior images of different storage areas are the vertical lines that need to be confirmed in the embodiment of the present application. By setting a threshold value for the number of non-zero pixels, the detection accuracy of vertical lines in the interior images of different storage areas is effectively improved.

[0169] The left and right boundaries of the backboard in the interior images of the different layers of object storage areas are obtained. After obtaining the vertical lines in which the number of non-zero pixels contained in the interior images of the different layers of object storage areas is greater than or equal to the set threshold, the left and right boundaries of the backboard are obtained according to the vertical distance between the vertical lines and the horizontal center pixel of the interior image. The vertical line located to the left of the horizontal center pixel and having the smallest vertical distance to the horizontal center pixel is determined as the left boundary of the backboard, and the vertical line located to the right of the horizontal center pixel and having the smallest vertical distance to the horizontal center pixel is determined as the right boundary of the backboard.

[0170] Fig. 9 In the example shown, the vertical white lines detected on the left and right sides of the back panel 150 are the vertical lines in which the number of non-zero pixels contained in the interior images of different storage areas is greater than or equal to the set threshold. For example, two vertical lines are detected on the left and right sides of the back panel 150 in the interior images of the first storage area, four vertical lines are detected on the left side of the back panel 150 and two vertical lines are detected on the right side in the interior images of the second storage area, and three vertical lines are detected on the left side of the back panel 150 and two vertical lines are detected on the right side in the interior images of the third storage area. Fig. 9 In the example shown, it is difficult to confirm the left and right boundaries of the back plate 150 in the interior image of the storage area at different levels, whether from the left side to the right side or from the right side to the left side.

[0171] Fig. 9 In the example shown, the horizontal center pixel point (100, width / 2) of the interior image can be determined first, where the height of the horizontal center pixel point can be but is not limited to the 100th row in the interior image, and the width is Fig. 9 The width of the interior image in the example shown is then searched from the horizontal center pixel point A of the interior image to the left and right sides of the interior image. Exemplarily, the first vertical lines found in the two directions are used as the left and right boundaries of the back panel 150, that is, the second vertical line from the left in the interior image of the first storage area is confirmed as the left boundary of the back panel 150, and the second vertical line from the right is used as the right boundary of the back panel 150. The fourth vertical line from the left in the interior image of the second storage area is confirmed as the left boundary of the back panel 150, and the second vertical line from the right is used as the right boundary of the back panel 150. The third vertical line from the left in the interior image of the third storage area is confirmed as the left boundary of the back panel 150, and the second vertical line from the right is used as the right boundary of the back panel 150.

[0172] After obtaining the left and right boundaries of the back plate 150 in the interior images of the different layered object areas, the center position of the back plate 150 is obtained according to the left and right boundaries of the back plate 150, and the center position of the back plate 150 is used as the reference alignment position corresponding to the interior images of the different layered object areas. Specifically, after obtaining the left and right boundaries of the back plate 150 in the interior images of the different layered object areas, the center position of the back plate 150 in each interior image can be calculated according to the left and right boundaries of the back plate 150.

[0173] For example, upLeftCol and upRightCol can be set to represent the left boundary horizontal coordinate value and the right boundary horizontal coordinate value of the back plate 150 in the first layer storage area interior image, medLeftCol and medRightCol can represent the left boundary horizontal coordinate value and the right boundary horizontal coordinate value of the back plate 150 in the second layer storage area interior image, downLeftCol and downRightCol can represent the left boundary horizontal coordinate value and the right boundary horizontal coordinate value of the back plate 150 in the third layer storage area interior image, and then the horizontal coordinate value upCenter of the center position of the back plate 150 in the first layer storage area interior image satisfies the following calculation formula:

[0174] upCenter=(upLeftCol+upRightCol) / 2

[0175] By referring to the above calculation formula, the horizontal coordinate values ​​upCenter, medCenter and downCenter of the center position of the back plate 150 in the interior image of the upper, middle and lower storage areas can be obtained.

[0176] In some embodiments, the controller is further configured to select the area where the backboard is located in the interior image to obtain vertical lines before obtaining vertical lines in which the number of non-zero pixels contained in the interior image is greater than or equal to a set threshold.

[0177] Specifically, the standard Hough transform algorithm uses an exhaustive method to calculate all pixels. Fig. 9 In the example shown, the interior image of the storage area captured by the camera includes not only the interior image of the area where the back panel 150 is located, but also the interior image of the area where other structures in the storage room of the food storage device are located, such as the interior image of the area where part of the storage partition 160 is located. The interior image of this part of the area also contains many non-zero pixel areas, but the interior image of this part of the area has no effect on the calibration of the interior image positions of different layers of storage areas in the embodiment of the present application. If the Hough transform algorithm is still used to process the interior image of the storage area of ​​the entire area, the calculation process will be time-consuming and the calculation efficiency will be low.

[0178] In order to solve this problem, the embodiment of the present application does not perform the process of acquiring vertical lines on the interior image of the entire storage area, but first selects the area where the back plate 150 is located in the interior image before acquiring vertical lines in which the number of non-zero pixels contained in the interior image is greater than or equal to a set threshold. Fig. 9 In the example shown, for example, the first 160 lines from top to bottom in the interior image of the storage area can be selected for calculation, that is, only the area where the back plate 150 is located in the interior image is selected to obtain vertical lines. In this way, about 2 / 3 of the calculation amount in the vertical line acquisition process can be saved, effectively improving the calculation efficiency of the vertical line acquisition process.

[0179] It should be noted that the 160 rows described in the above embodiment are only used to exemplarily illustrate and distinguish the area where the back panel 150 is located and the areas where other structures in the storage room are located. The embodiment of the present application does not specifically limit the number of image rows corresponding to the area where the back panel 150 is located, and the row value can be adjusted according to the installation position and shooting angle of the corresponding camera.

[0180] S830: align and correct interior images of storage areas at different levels according to the reference alignment position.

[0181] Fig. 9 In the example shown, the fixed reference object is a back panel 150 disposed inside the storage room, and the interior images of different storage areas are aligned and corrected according to the reference alignment position, which may be to align and correct the center position of the back panel 150 in the interior images of different storage areas.

[0182] In some embodiments, the center position of the back plate 150 in the interior images of different layers of storage areas is aligned and corrected so that the center position of the back plate 150 is longitudinally aligned. The maximum center position value can be obtained based on the center position of the back plate 150 in the interior images of different layers of storage areas. The first image capture width is determined based on the center position of the back plate 150 in the interior images and the maximum center position value, and the side image in the interior image corresponding to the first image capture width is cut off.

[0183] In some embodiments, the alignment correction performed on the center position of the back plate 150 in the interior images of different layers of object storage areas includes a left alignment correction or a right alignment correction. The left alignment correction performed on the center position of the back plate 150 in the interior images of different layers of object storage areas can be performed by obtaining the minimum value of the center position based on the center position of the back plate 150 in the interior images of different layers of object storage areas, determining the difference between the center position of the back plate 150 in the interior image and the minimum value of the center position as the first image capture width, and cutting off the left side image within the range of the first image capture width in the interior image.

[0184] The center position of the back plate 150 in the interior images of different storage areas is left-aligned, that is, the center position of the back plate 150 is used to align the left edge of the back plate 150 in the interior images of different storage areas to achieve the alignment correction of the back plate in the interior images of different storage areas. Fig. 9 In the example shown, referring to the description of the aforementioned embodiment, the horizontal coordinate values ​​upCenter, medCenter and downCenter of the center position of the back plate 150 in the interior image of the upper, middle and lower storage areas have been obtained, and then the minimum center position value minCenter is determined among the aforementioned three values, and the difference between the center position of the back plate 150 in the interior image and the minimum center position value minCenter is determined as the first image capture width, and the left side image within the first image capture width in the interior image is cut off, that is, the calibration parameters of the interior image of each storage area are calculated, and the calculation method is as follows:

[0185] upLeftStart=upCenter–minCenter

[0186] medLeftStart=medCenter–minCenter

[0187] downLeftStart=downCenter–minCenter

[0188] Among them, upLeftStart, medLeftStart and downLeftStart are respectively the left starting positions of the interior image of the three-layer storage area, that is, the width of the left side of the interior image that needs to be cut off. Fig. 9In the example shown, the center position medCenter of the backboard 150 in the interior image of the second storage area is the minimum center position minCenter, the difference between the center position corresponding to the first storage area and the minimum center position minCenter is also equal to the distance between the left border of the backboard 150 in the interior image of the first and second storage areas, and the difference between the center position corresponding to the third storage area and the minimum center position minCenter is also equal to the distance between the left border of the backboard 150 in the interior image of the third and second storage areas. Therefore, Fig. 9 It can be seen from the example shown that by cutting off the image within the difference range between the center position and the minimum center position minCenter on the left side of the first and third layer interior images, the left boundary of the back plate in the three-layer storage area interior image can be aligned, thereby achieving alignment correction of the back plate 150 in the three-layer storage area interior image.

[0189] Fig. 9 In the example shown, the minimum center position value minCenter corresponds to the interior image of the middle storage area. Therefore, the interior image of the first storage area can be cut off by cutting off the left side image within the width range of upCenter-minCenter, and the interior image of the third storage area can be cut off by cutting off the left side image within the width range of downCenter-minCenter. The interior image of the second storage area does not need to be cut off, so that the left boundary of the back panel 150 in the interior images of the three storage areas can be aligned.

[0190] Thereby, the alignment of the left border of the back panel 150 in the interior image of the three-layer storage area is achieved, that is, the alignment correction of the back panel 150 in the interior image of the three-layer storage area is achieved, and the display 210 is used to receive the interior images of the storage areas on different layers in the storage room output by the controller 220, and to display the interior images of the storage areas on different layers to show the actual interior of the storage room, so that the relative positions of the fixed reference objects spanning the storage areas on different layers in the interior images of the storage areas on different layers displayed by the display 210 are consistent after imaging, and the corresponding parts of the fixed reference objects spanning the storage areas on different layers in the interior images of the storage areas on different layers are aligned, which solves the problem of tomographic display of the interior images of the storage areas on different layers and optimizes the imaging display effect of the interior images of the storage room.

[0191] In some embodiments, the center position of the back plate 150 in the interior images of different layers of storage areas is right-aligned, and the maximum center position is obtained based on the center position of the back plate 150 in the interior images of different layers of storage areas, and the difference between the maximum center position and the center position of the back plate 150 in the interior image is determined as the first image capture width, and the right side image within the first image capture width in the interior image is captured.

[0192] The center position of the back plate 150 in the interior images of the different layered object storage areas is right-aligned and corrected. That is, the center position of the back plate 150 is utilized to align the right edge of the back plate 150 in the interior images of the different layered object storage areas to achieve the alignment correction of the back plate 150 in the interior images of the different layered object storage areas. Fig. 9 In the example shown, referring to the description of the aforementioned embodiment, the horizontal coordinate values ​​upCenter, medCenter and downCenter of the center position of the back plate 150 in the interior images of the upper, middle and lower storage areas have been obtained, and then the maximum value of the center position among the aforementioned three values ​​is determined as maxCenter, and the difference between the maximum value of the center position maxCenter and the center position of the back plate 150 is determined as the first image capture width, and the right side image within the first image capture width in the interior image is cut off, that is, the calibration parameters of the interior images of each storage area are calculated, and the calculation method is as follows:

[0193] upRightStart=maxCenter–upCenter

[0194] medRightStart=maxCenter–medCenter

[0195] downRightStart=maxCenter–downCenter

[0196] Among them, upRightStart, medRightStart and downRightStart are respectively the starting positions of the right side of the interior image of the three-layer storage area, that is, the width of the right side of the interior image that needs to be cut off. Fig. 9 In the example shown, the center position medCenter of the backboard 150 in the interior image of the third storage area is the maximum center position maxCenter. The difference between the maximum center position maxCenter and the center position corresponding to the first storage area is also equal to the distance between the right edge of the backboard 150 in the interior image of the first and third storage areas. The difference between the maximum center position maxCenter and the center position corresponding to the second storage area is also equal to the distance between the right edge of the backboard 150 in the interior image of the second and third storage areas. Therefore, Fig. 9 It can be seen from the example shown that by cutting off the rightmost side of the interior image within the difference range between the maximum center position of maxCenter and the center position, the right boundary of the back panel in the interior image of the three-layer storage area can be aligned, thereby achieving the alignment correction of the back panel 150 in the interior image of the three-layer storage area.

[0197] Fig. 9In the example shown, the maximum value of the center position maxCenter corresponds to the interior image of the third storage area. Therefore, the right side image within the width range of maxCenter-upCenter can be cut off from the interior image of the first storage area, and the right side image within the width range of maxCenter-medCenter can be cut off from the interior image of the second storage area. The interior image of the third storage area does not need to be cut off, so that the right boundary of the back panel 150 in the interior images of the three storage areas can be aligned.

[0198] Thereby, the alignment of the right border of the back panel 150 in the interior image of the three-layer storage area is achieved, that is, the alignment correction of the back panel 150 in the interior image of the three-layer storage area is achieved. The display 210 is used to receive the interior images of the storage areas on different layers in the storage room output by the controller 220, and is used to display the interior images of the storage areas on different layers to show the actual interior of the storage room, so that the relative positions of the fixed reference objects spanning the storage areas on different layers in the interior images of the storage areas on different layers displayed by the display 210 are consistent after imaging, and the corresponding parts of the fixed reference objects spanning the storage areas on different layers in the interior images of the storage areas on different layers are aligned, which solves the problem of tomographic display of the interior images of the storage areas on different layers and optimizes the imaging display effect of the interior images of the storage room.

[0199] In some embodiments, the controller is further configured to, after aligning and correcting the center position of the back plate 150 in the interior images of the different layers of storage areas, perform a cropping process on the interior images according to the minimum width of the interior images of the different layers of storage areas.

[0200] In some implementations, the interior image is cropped based on the minimum width among the widths of the interior images of different storage areas, and the side images of the interior image may be cropped based on the original width of the interior image, the maximum value at the center, and the minimum value at the center; wherein, after the cropping process, the widths of the interior images of different storage areas are all equal to the minimum width, and the minimum width is equal to the sum of the result of subtracting the maximum value at the center from the original width of the interior image and the minimum value at the center.

[0201] Specifically, after the center position of the back plate 150 in the interior images of the storage areas at different levels is aligned and corrected, the interior images of the storage areas at different levels are intercepted within different width ranges, resulting in different widths of the interior images of the storage areas at different levels after interception. To ensure that the widths of the interior images of the storage areas at different levels finally displayed by the display 210 are equal, it is necessary to recalculate the widths of the interior images of the storage areas at different levels finally displayed by the display 210. The width of the interior images of each storage area at each level finally displayed by the display 210 satisfies the following calculation formula:

[0202] newPicWidth=width–maxCenter+minCenter

[0203] Among them, newPicWidth is the width of the interior image of each storage area finally displayed by the display 210, width is the width of the original interior image before any processing, maxCenter is the maximum value of the center position of the back panel 150 in the interior image, and minCenter is the minimum value of the center position of the back panel 150 in the interior image.

[0204] Fig.10 The figure is a schematic diagram of a width adjustment process of an interior scene image corresponding to left alignment correction according to an exemplary embodiment of the present application. Fig.10 (a) in the figure represents the interior image of the three-layer storage area before any processing, wherein 150 represents the back plate and 170 represents the interior image of the entire storage area. Fig.10 (a) in the figure exemplarily shows the width of the back panel 150 and the width of the entire storage area interior image 170. Fig.10 (b) in FIG. 1 shows the interior image of the three-layer storage area after left alignment correction by intercepting the left image of the interior image. Fig.10 (c) in the figure shows the interior images of the three-layer storage area after adjusting the interior images of the three-layer storage area to be unified into the minimum width.

[0205] Fig.10 In the example shown in (a), the left boundary is taken as the zero point of the horizontal axis, upCenter is equal to 9a, medCenter is equal to 6a, downCenter is equal to 11a, minCenter is equal to 6a, and maxCenter is equal to 11a.

[0206] Fig.10 In the example shown in (b), referring to the above embodiment, in order to realize the alignment of the left border of the back panel 150 in the interior images of different storage areas, the interior images of the second storage area are kept unchanged, and the upCenter minCenter in the interior image of the first storage area, that is, the leftmost interior image within the width of 3a, is cut off, and the downCenter minCenter minCenter in the interior image of the third storage area, that is, the leftmost interior image within the width of 5a, is cut off. At this time, the minimum width of the interior images of different storage areas is the width of the interior image of the third storage area, which is 13a. This value is equal to the sum of the original width of the interior image 18a minus the maximum value 11a of the center position 7a and the minimum value 6a of the center position.

[0207] Fig.10In the example shown in (c), the width of the interior image of all storage areas needs to be adjusted to the minimum width of 13a. The interior image of the third storage area does not need to be clipped. The rightmost interior image within the width of 2a in the interior image of the first storage area is clipped to make the width of the interior image of the first storage area equal to the minimum width 13a. The rightmost interior image within the width of 5a in the interior image of the second storage area is clipped to make the width of the interior image of the second storage area equal to the minimum width 13a. In this way, the widths of the interior images of different storage areas are unified under the premise of ensuring that the back panels 150 in the interior images of different storage areas are aligned, so that the interior images of different storage areas are regular images with consistent widths.

[0208] Fig.11 It is a schematic diagram of a width adjustment process of an interior scene image corresponding to right alignment correction according to an exemplary embodiment of the present application. Fig.11 (a) shows the interior image of the three-layer storage area before any processing. Fig.11 (a) in the figure exemplarily shows the width of the back panel 150 and the width of the entire storage area interior image 170. Fig.11 (b) in FIG. 1 shows the interior image of the three-layer storage area after right alignment correction by intercepting the right image of the interior image. Fig.11 (c) in the figure shows the interior images of the three-layer storage area after adjusting the interior images of the three-layer storage area to be unified into the minimum width.

[0209] Fig.11 In the example shown in (a), the left boundary is taken as the zero point of the horizontal axis, upCenter is equal to 9a, medCenter is equal to 6a, downCenter is equal to 11a, minCenter is equal to 6a, and maxCenter is equal to 11a.

[0210] Fig.11 In the example shown in (b), referring to the above embodiment, in order to realize the alignment of the right border of the back panel 150 in the interior images of different storage areas, the interior images of the third storage area are kept unchanged, and the interior images of the first storage area are cut off by subtracting maxCenter from upCenter, that is, the rightmost interior image within the width of 2a, and the interior images of the second storage area are cut off by subtracting maxCenter from medCenter, that is, the rightmost interior image within the width of 5a. At this time, the minimum width of the interior images of different storage areas is the width of the interior images of the second storage area, which is 13a. This value is equal to the sum of the original width of the interior image 18a minus the maximum value 11a of the center position 7a and the minimum value 6a of the center position.

[0211] Fig.11In the example shown in (c), the widths of the interior images of all storage areas need to be adjusted to the minimum width of 13a, and the interior images of the second storage area do not need to be cropped. The leftmost interior image within a width of 3a in the interior images of the first storage area is cropped so that the width of the interior image of the first storage area is equal to the minimum width 13a. The leftmost interior image within a width of 5a in the interior images of the third storage area is cropped so that the width of the interior image of the third storage area is equal to the minimum width 13a. In this way, while ensuring that the back panels 150 in the interior images of different storage areas are aligned, the widths of the interior images of different storage areas are unified, so that the interior images of different storage areas finally displayed on the display 210 are regular images with consistent widths.

[0212] It should be noted that the width values ​​in the above embodiments are merely numerical examples for illustrating the alignment correction process and the width unification process, and are not limitations on the widths of corresponding structures in the interior image and their width ratios.

[0213] In some embodiments, after the interior images of different storage areas are aligned and corrected according to the reference alignment position, the aligned and corrected interior images of different storage areas can be spliced ​​into a spliced ​​interior image, and the display 210 is used to display the spliced ​​interior image. Fig.12 It is a schematic diagram of interior images of object storage areas on different levels after alignment correction according to an exemplary embodiment of the present application. Fig.12 In the example shown, after the alignment correction and width unification processing of the above-mentioned embodiment, the left and right boundaries of the back panel 150 in the interior scene images of different storage areas are aligned, and the widths of the interior scene images of different storage areas are consistent. The controller 220 stitches the interior scene images of different storage areas after the alignment correction into a stitched interior scene image. The display 210 is used to receive the stitched interior scene images of different storage areas in the storage room output by the controller 220, and is used to display the stitched interior scene images to show the actual interior scene of the storage room, so that the relative positions of the fixed reference objects spanning the storage areas on each layer in the stitched interior scene image displayed by the display 210 are consistent after imaging, and the corresponding parts of the fixed reference objects spanning the storage areas on each layer in the stitched image are aligned, which solves the problem of tomographic display after stitching of different interior scene images and optimizes the imaging display effect of the storage room interior scene images after stitching.

[0214] It can be seen from the above embodiments that the embodiments of the present application provide a method for aligning the composite images shot by multiple cameras on the screen of an intelligent food storage device. By calibrating the center of the back plate, the interior images of the storage areas on different levels are cut, and finally the imaging alignment of the storage areas on different levels is ensured. Before the food storage device leaves the factory, the interior images of the storage areas on different levels can be aligned and calibrated, and the offset parameters can be saved. After leaving the factory, each time the storage areas on different levels are photographed, the photos are aligned and calibrated according to the calibrated parameters to ensure that the objects across the storage areas on each level in the food storage device can maintain the same relative position in the final imaging, and optimize the imaging display effect after the splicing of the interior images of the storage room.

[0215] like Fig.13 As shown, the interior display interface includes a picture display area 1310 and a content recognition area 1320. The picture display area 1310 displays interior pictures 1311, 1312 and 1313, and the positional relationship of the interior pictures 1311, 1312 and 1313 in the interior display interface is consistent with the positional relationship between the storage compartments 1121, 1122 and 1123. In addition, below the interior pictures, the shooting time "June 5" of the group of interior pictures is displayed. In this way, the user can view the ingredients stored in the storage compartments 1121-1123 on "June 5" based on the interior pictures taken on "June 5". For example, by observing the interior picture 1311, it can be seen that the storage compartment 1121 has "grapes", "bananas" and "potatoes", and by observing the interior picture 1312, it can be seen that the storage compartment 1122 has "peach" and "milk", and by observing the interior picture 1312, it can be seen that the storage compartment 1123 has "pineapple" and "durian". In addition, the user can also trigger the display of the previous set of interior pictures in the interior viewing interface. Usually, the shooting time of the previous set of interior pictures is earlier than the shooting time of the previous set of interior pictures. For example, the user can trigger the display of the previous set of interior pictures by sliding the current interior picture to the left. Fig.13 The interface changes to Fig.14 The interface shown. Fig.13 The interface shown is different in that Fig.14In the interior view interface shown, the picture display area 1310 displays interior pictures 1314, 1315 and 1316, which are interior pictures of the storage cells 1121, 1122 and 1123 collected on "June 4". The "Picture Recognition" control 1321 and the "One-key Entry" control 1322 are displayed below the content recognition area 1320. The user can click the "Picture Recognition" control 1321 to indicate that the food object in the interior picture is to be identified, and the identified food icon will be displayed in the content recognition area. The user can select the food icon displayed in the content recognition area, and enter the selected food into the food management application by clicking the "One-key Entry" control 1322.

[0216] It should be noted that Fig.13 and Fig.14 The interior view display interface shown is an example of the present application and does not constitute a limitation on the protection scope of the present application. For example, two groups of interior view pictures can be displayed simultaneously in the interior view display interface. By simultaneously displaying two groups of interior view pictures taken at different times on the interior view display interface, it can help users understand which ingredients have been added and which ingredients have been reduced. For another example, the interior view picture display area can be limited to the left area of ​​the interface, and the content recognition area can be limited to the right area of ​​the interface.

[0217] In some embodiments, a user can view the interior view of the food storage device storage room through a terminal device associated with the food storage device. The terminal device associated with the food storage device can be a terminal device associated with the food storage device based on identity information. The identity information can be device information, account information, etc., for example, an account logged in by the food management application.

[0218] Exemplarily, the account logged into the food management application on the food storage device is called the main business account. Then, the terminal device associated with the food storage device can be a terminal device bound to the main business account, such as a mobile phone, TV, tablet computer and other terminal devices bound to the main business account. It can also be a terminal device corresponding to a business account associated with the main business account, such as a mobile phone, computer, etc. corresponding to the WeChat account associated with the main business account. It can also be other terminal devices logged in to the main business account. For example, when a user installs a food management application on a TV and uses the main business account to log in to the food management application on the TV, the TV is called a terminal device associated with the food storage device.

[0219] In some embodiments, the user can log in to the food management application on other terminal devices such as mobile phones and televisions, and enter the instruction to view the interior of the food storage device in the food management application. Fig.15A food management interface displayed on a terminal device, including an interior view control. A user can click on the interior view control to input an interior view instruction. In response to the instruction, the terminal device sends a picture acquisition request containing the identity information of the terminal device to a server, wherein the identity information of the terminal device can be account information for logging into a food management application, or device information of the terminal device. The server searches for one or more groups of interior view pictures of a food storage device associated with the terminal device based on the association between the identity information of the terminal device and the identity information of the food storage device, and returns them to the terminal device. The terminal device displays the one or more groups of interior view pictures returned by the server in a user interface, and makes the display position relationship between the interior view pictures in the same group consistent with the position relationship between the containing grids corresponding to each interior view picture. Fig.16 It is an interior display interface displayed on a terminal device. Fig.16 In the figure, interior scene pictures 111, 112 and 113 are displayed. The positional relationship between the interior scene pictures 111, 112 and 113 in the interior scene display interface is consistent with the positional relationship between the containing grids 1121, 1122 and 1123, which are all top-middle-bottom relationships.

[0220] It can be seen from the above embodiments that the food storage device provided by the embodiment of the present application, the storage chamber includes a plurality of storage compartments, and the interior pictures of the storage chamber include a group of interior pictures of each storage compartment taken in the same time period. The food storage device provided by the embodiment of the present application can obtain at least one group of interior pictures in response to an input interior viewing instruction; and display the at least one group of interior pictures in a user interface, wherein the display position relationship between the interior pictures in the same group is consistent with the position relationship between the storage compartments corresponding to each interior picture. It can be seen that the present application displays the interior scene of the storage chamber to the user by displaying interior pictures to the user. Since the arrangement of the interior pictures in the user interface is consistent with the setting position of the storage compartments in the storage chamber, the presentation effect of the interior scene of the storage chamber is more realistic and the degree of restoration is very high.

[0221] In some embodiments, when the display of the storage room interior picture is triggered, the user can input a food identification instruction to instruct the food storage device to identify the food objects in the displayed interior picture. It should be noted that the food storage device can also automatically identify the food objects in the displayed interior picture while displaying the storage room interior picture, without requiring additional user operation.

[0222] Based on the above embodiment, the controller 220 is configured to execute Fig.17 Steps shown:

[0223] S171, in response to the input interior view instruction, displaying an interior picture of the storage room, the interior picture including food objects.

[0224] For Figure 1A The food storage device shown in the figure can display the interior picture of the storage room in the manner shown in S610-620 in the above embodiment, and the display effect can be as follows: Figure 7 , Fig.13 or Fig.14 As shown, no further description is given here.

[0225] by Fig.13 For example, the food objects in the interior picture 1311 include "grapes", "bananas" and "potatoes".

[0226] It should be noted that, depending on the structure of the food storage device, the number and position of the camera, the number of interior pictures of the storage room may be the same or different. For example, the interior picture may be a panoramic picture of the interior of the storage room, or may be composed of interior pictures of different parts of the storage room. Depending on the number, size, etc. of the interior pictures, they may be displayed in different ways.

[0227] S172, in response to the input ingredient recognition instruction, displaying an ingredient icon that matches the ingredient object in the interior picture.

[0228] In some embodiments, the user can click on the "image recognition" control in the interior display interface to input an ingredient recognition instruction, which is used to instruct the ingredient storage device to recognize the interior image displayed in the current interface.

[0229] In some interior display interfaces, multiple "image recognition" controls are displayed, and the multiple interior images displayed by the multiple "image recognition" controls correspond to each other. The user can click the "image recognition" control corresponding to a certain interior image to enter an instruction to recognize ingredients for the interior image. In response to the instruction, the ingredient management application displays an ingredient icon that matches the ingredient object in the interior image. For example, in Fig.18 In the interior display interface shown, interior pictures 1801, 1802 and 1803 are displayed in the interior picture display area, wherein interior picture 1801 includes food object ①, food object ② and food object ③, interior picture 1802 includes food object ④, food object ⑤ and food object ⑥, and interior picture 1803 includes food object ⑦ and food object ⑧. An "image recognition" control is displayed in the upper right corner of each interior picture. When the user clicks the "image recognition" control in the upper right corner of interior picture 1801, the food icon that matches the food object in interior picture 1801 will be displayed in the content recognition area. For example, in Fig.18 In the content recognition area, food icon 1 matching food object ①, food icon 2 matching food object ②, and food icon 3 matching food object ③ are displayed.

[0230] In other embodiments, the user can click on a displayed interior picture to input an ingredient identification instruction for the interior picture. In response to the instruction, the ingredient management application displays an ingredient icon that matches the ingredient object in the interior picture. In these embodiments, the "Picture Identification" control may be displayed in the interior display interface, or the "Picture Identification" control may not be displayed. In the implementation scenario where the "Picture Identification" control is displayed, the user can still input an ingredient identification instruction by operating the "Picture Identification" control.

[0231] In some interior display interfaces, a "picture recognition" control is displayed, and the user can click the "picture recognition" control to enter an ingredient recognition instruction for all displayed interior pictures. In response to the instruction, the ingredient management application displays ingredient icons matching the ingredient objects in all interior pictures in the content recognition area. For example, in Fig.19 In the interior display interface shown, interior pictures 1801, 1802 and 1803 are displayed in the interior picture display area, and an "image recognition" control is displayed in the content recognition area. When the user clicks the "image recognition" control, the content recognition area displays the food icons that match the food objects in the interior pictures 1801, 1802 and 1803, including food icons 1-5 and food icons 7-8 that match food objects ①-⑤ and food objects ⑦-⑧. Since food objects ⑤ and ⑥ are the same, only one matching food icon is displayed in the content recognition area.

[0232] In some implementations, the food management application identifies food objects in the interior image through the server. Fig. 20 , the food management application on the food storage device responds to the input food recognition instruction and sends a food recognition request to the server, and the food recognition request includes relevant information of the interior picture. After receiving the food recognition request, the server recognizes the food object in the interior picture based on the relevant information of the interior picture, obtains a recognition result, and returns the recognition result to the food storage device, and the recognition result includes identification information of the food object, such as food ID or food name. After receiving the recognition result returned by the server, the food management application on the food storage device obtains food information matching the recognition result from the preset food library. For example, food information including the food name of the food object is obtained from the preset food library. Finally, according to the food information matching the recognition result, the food icon matching the food object is displayed.

[0233] In the above embodiment, the relevant information of the interior picture includes the unique identifier of the interior picture and / or the picture data of the interior picture. In the implementation method using the unique identifier of the interior picture as the relevant information, the server needs to search for the interior picture represented by the unique identifier based on the unique identifier, so as to perform picture recognition based on the found interior picture. The unique identifier can be a picture name. In the implementation method using the picture data of an interior picture as the relevant information, the server can directly perform image recognition based on the picture data.

[0234] See also Fig.21 In some embodiments, the food management application on the food storage device responds to the input food recognition instruction and sends a food recognition request to the server, and the food recognition request includes relevant information of the interior picture. After receiving the food recognition request sent by the food storage device, the server generates a corresponding picture recognition task according to the food recognition request, and adds the picture recognition task to the specified task queue to sequentially execute the picture recognition tasks in the queue, so as to avoid excessive processing pressure on the server due to the concurrent execution of a large number of picture recognition tasks. After the server generates the picture recognition task, it generates a result identifier and returns the result identifier to the food storage device, and the result identifier corresponds to the food recognition request or the picture recognition task generated based on the food recognition request. The food management application on the food storage device receives the result identifier, and after a second time (such as 2 seconds), sends a result acquisition request containing the result identifier to the server. The server searches for the recognition result of the corresponding picture recognition task according to the result identifier, and returns the recognition result to the food storage device. In this way, the server does not need to process the picture recognition request of the food storage device in real time, so that excessive processing pressure will not be generated, and server crashes are avoided. In addition, by sending an image recognition request to the server and requesting result data from the server, the result request process is separated from the image recognition request process, avoiding the food management application from being blocked by waiting for the recognition result.

[0235] It should be noted that after the server or food storage device obtains the recognition result, it performs deduplication processing on the recognition result. For example, when the recognition result contains multiple repeated food names, the redundant food names are deleted and only one food name is retained. Exemplarily, the original recognition result and the deduplication recognition result are shown in the following table:

[0236]

[0237] In some embodiments, in S171, in response to the input interior view instruction, the most recently collected interior picture of the storage room is displayed; in S172, according to the recognition result of the food object in the interior picture returned by the server, the new food in the storage room is determined, and the new food refers to the food that has not been determined as an existing food. Then, the food icon of the new food is displayed in the user interface, and the food icon of the non-new food is not displayed, so as to achieve the effect of filtering the non-new food, avoid the user from selecting the food icon of the non-new food to add, and improve the accuracy and pertinence of the user operation. In an optional implementation, after receiving the food name of the food object returned by the server, the food name set of the existing food is obtained (from the local food information set); if the food name of the food object is included in the food name set of the existing food, then it is determined that the food object is not a new food, and if the food name of the food object is not included in the food name set of the existing food, then it is determined that the food object is a new food.

[0238] In some embodiments, through S172, the food icon that matches the food object in the interior picture is displayed as a state to be deleted. In this way, by clicking a certain food icon and deleting the food icon, the last retained food icon is the selected target food icon.

[0239] S173: According to the target ingredient icon selected by the user, the ingredient information corresponding to the target ingredient icon is determined as the ingredient information of the existing ingredients in the storage room.

[0240] By determining the ingredient information corresponding to the target ingredient icon as the ingredient information of the existing ingredient, the ingredient adding operation is completed, and the added ingredient is the ingredient represented by the target ingredient icon.

[0241] In some embodiments, in response to an ingredient input instruction input by a user, the ingredient information corresponding to the target ingredient icon selected by the user is determined as the ingredient information of the existing ingredients in the storage room.

[0242] It should be understood that the user can select the target ingredient icon from the ingredient icons displayed in the content identification area through a variety of operation methods. For example, by long pressing a certain ingredient icon, the ingredient icon is selected. For another example, by long pressing a certain ingredient icon, a certain ingredient icon is deleted, and the last retained ingredient icon is the selected target ingredient icon. For another example, by long pressing a certain ingredient icon, all ingredient icons are triggered to enter a waiting state, and then the ingredient icon is selected by clicking a certain ingredient icon. For another example, by long pressing a certain ingredient icon, all ingredient icons are triggered to enter a waiting state, and then the ingredient icon is deleted by clicking a certain ingredient icon, and the last retained ingredient icon is the selected target ingredient icon.

[0243] In some embodiments, based on the user's operation on the food icon in the content identification area, the content identification area is updated in real time, so that the user can confirm the selection of the food icon according to the display of the content identification area. After completing the selection operation of the target food icon, the user can operate the "one-key entry" control in the content identification area to input the food entry instruction. In response to the instruction, the food management application determines the food information corresponding to the target food icon as the food information of the existing food in the storage room, thereby completing the operation of adding (entering) the new food into the food management application.

[0244] In an optional implementation, the ingredient information corresponding to the target ingredient icon is determined as the ingredient information of the existing ingredients in the storage room. Specifically, the ingredient information corresponding to the target ingredient icon is added to the local ingredient information set and marked as being in the put-in state.

[0245] In some embodiments, after the food information corresponding to the target food icon selected by the user is determined as the food information of the existing food, the food information of all the existing food is read, and the interface is updated to the food management interface. The updated food management interface displays the food icons of all the existing food.

[0246] It can be seen from the above embodiments that the food storage device provided by the embodiments of the present application can display the interior picture of the storage room in response to the input interior view instruction, and the interior picture includes food objects; in response to the input food recognition instruction, it can display the food icon that matches the food object in the interior picture; according to the target food icon selected by the user, the food information corresponding to the target food icon is determined as the food information of the existing food in the storage room. The present application can obtain the food information of the food in the storage room based on image recognition, and display it in the form of food icons. By determining the food information corresponding to the target food icon selected by the user as the food information of the existing food, the food adding operation is completed. The added food is the food represented by the target food icon, and the user operation of manually inputting the food information is omitted.

[0247] According to the food storage device provided in the above embodiment, the present application also provides an image acquisition method, and the execution subject of the image acquisition method includes but is not limited to the controller of the above food storage device. The specific implementation of the image acquisition method can refer to the above embodiment, especially Figure 3 The illustrated embodiments are not described in detail here.

[0248] According to the food storage device provided in the above embodiment, the present application also provides a food input method, the execution subject of the food input method includes but is not limited to the controller of the above food storage device. The specific implementation of the food input method can refer to the above embodiment, especially Fig.17 The illustrated embodiments are not described in detail here.

[0249] In a specific implementation, the present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment of the method provided by the present invention. The storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0250] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention or some parts of the embodiments.

[0251] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the food storage device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0252] The above-described embodiments of the present invention do not limit the protection scope of the present invention.

Claims

1. A food storage device, comprising: Box body, box door, display, camera, detector and controller, wherein: The box body has a storage room inside for storing food; A box door, arranged at the opening of the storage room; A display, arranged on the box door or the outside of the box body, for displaying a user interface; A camera, disposed on one side of the door located inside the storage room, for collecting pictures of the interior of the storage room; A detector, used to detect a switch signal of the box door; The food storage device is configured as follows: When a preset trigger condition is met, the camera is controlled to start to collect an interior picture of the storage room; Determine whether the detector detects an opening signal of the door within a preset time after the camera captures the interior scene image; If the detector detects an opening signal of the door within a preset time after the camera captures the interior picture, the interior picture is not displayed on the display; If the detector does not detect an opening signal of the door within a preset time after the camera captures the interior picture, the interior picture is displayed on the display.

2. The food storage device according to claim 1, wherein the preset trigger condition comprises at least one of the following conditions: When the detector detects a closing signal of the box door; When receiving an input interior image acquisition instruction; When the current time reaches the preset time point.

3. The food storage device according to claim 1, wherein the food storage device is further configured as: If the detector detects an opening signal of the door within a preset time after the camera captures the interior picture, the interior picture is deleted; If the detector does not detect an opening signal of the door within a preset time after the camera captures the interior picture, the interior picture is stored and displayed on the display.

4. The food storage device according to claim 1, wherein the food storage device is further configured as: When the preset trigger condition is not met, determining whether the camera is in an on state; If the camera is on, turn off the camera; in, The trigger condition is not met including when the detector detects an opening signal of the door.

5. An image acquisition method, applied to a food storage device, the method comprising: When a preset trigger condition is met, the camera of the food storage device is controlled to start up to collect an interior picture of the storage room of the food storage device; Determine whether the detector detects an opening signal of the door within a preset time after the camera captures the interior scene image; If the detector detects an opening signal of the door within a preset time after the camera captures the interior picture, the interior picture is not displayed on the display; If the detector does not detect an opening signal of the door within a preset time after the camera captures the interior picture, the interior picture is displayed on the display.

6. A food storage device, characterized in that: include: A box body, wherein a storage chamber is provided inside, and the storage chamber includes a plurality of storage compartments for storing food; A box door, arranged at the opening of the storage room; A camera, disposed on one side of the door located inside the storage room, for collecting pictures of the interior of the storage room; A display, arranged on the box door or the outside of the box body, for displaying a user interface; The food storage device is configured as follows: In response to an input instruction for viewing interior scenes of a storage room, acquiring at least one group of interior scene pictures, each group of interior scene pictures including interior scene pictures of each of the storage compartments collected in the same time period; The at least one group of interior scene pictures is displayed in the user interface, wherein the display position relationship of the interior scene pictures in the same group is consistent with the position relationship between the containing grids corresponding to the interior scene pictures.

7. The food storage device according to claim 6, characterized in that: The number of the cameras is the same as the number of the storage compartments, and different cameras are used to capture interior pictures of corresponding storage compartments.

8. The food storage device according to claim 6, characterized in that: The food storage device is further configured as follows: A plurality of picture display positions and spacing areas are displayed in the user interface; the spacing areas are used to simulate the layer partitions of the containing grid; the picture display positions are used to set the interior scene pictures; According to the accommodating grid corresponding to each interior picture in each group of interior pictures, each interior picture is displayed in a corresponding picture display position.

9. The food storage device according to claim 6, characterized in that: The food storage device is further configured as follows: The interior pictures taken in the same time period are assigned the same time mark, and the time mark is used to indicate the acquisition time of the interior pictures.

10. The food storage device according to claim 6, characterized in that: The interior image of a single container does not include the upper container area and the lower container area of ​​the single container.

11. The food storage device according to claim 10, characterized in that: For a single containing grid, the containing grid area of ​​the single containing grid is a key area of ​​the interior scene picture, and the upper containing grid area or the lower containing grid area of ​​the single containing grid is a redundant area of ​​the interior scene picture; the picture taken by the camera for the single containing grid includes the key area and the redundant area; The food storage device is further configured as follows: The picture taken by the camera for a single storage grid is cropped to retain the key area of ​​the picture and remove the redundant area of ​​the picture to obtain the interior picture of the single storage grid.

12. A method for displaying pictures of interior scenes of a storage room, characterized in that: Applied to a food storage device comprising a plurality of storage compartments, the method comprises: In response to an input storage interior scene viewing instruction, acquiring at least one group of interior scene pictures, each group of interior scene pictures including interior scene pictures of each storage compartment collected in the same time period; The at least one group of interior scene pictures is displayed in the user interface, wherein the display position relationship between the interior scene pictures in the same group is consistent with the position relationship between the containing grids corresponding to the interior scene pictures.

Images