Intelligent refrigerator and food material recognition method

Abstract

The application provides a smart refrigerator and a food material recognition method, and relates to the technical field of home appliances. A plurality of images are acquired by an image acquisition device, and then image features of each image in the plurality of images are extracted, and the image features of each image are sequentially fused according to the acquisition time of each image to obtain food material fusion features. Based on the food material fusion features, the type information of the food material accessed by the user in the door opening and closing process is determined. Through the method, the image features of the plurality of images acquired in the opening and closing process of the smart refrigerator can be fused to obtain the food material fusion features, and the type information of the food material accessed by the user is determined, and thus the accuracy of recognizing the type information of the food material is improved.

Description

Technical Field

[0001] This invention relates to the field of home appliance technology, and in particular to an intelligent refrigerator and a method for identifying food ingredients. Background Technology

[0002] Refrigerators, as a household appliance, are widely used in people's daily lives. With the continuous development of technology, the precise management of food information has become the core function of refrigerator intelligence. In the precise management of food information, the accurate identification of food type information is an important prerequisite for achieving precise management of food information.

[0003] Currently, the process of identifying food types typically involves extracting features from a single frame of an image captured during the user's food retrieval process, and then analyzing these features to determine the type of food accessed. However, this method only analyzes the features of a single frame, which reduces the accuracy of identifying food types. Summary of the Invention

[0004] This application provides an intelligent refrigerator and a food identification method, which can improve the accuracy of identifying food type information.

[0005] The specific technical solutions provided in this application are as follows:

[0006] In a first aspect, embodiments of this application provide a smart refrigerator, comprising:

[0007] The casing has multiple storage compartments inside, and each storage compartment has a door;

[0008] An image acquisition device, used to acquire images of the active area of ​​the door;

[0009] The processor is configured to acquire multiple frames of images via the image acquisition device during the process of the door of the smart refrigerator opening and closing.

[0010] Extract the image features of each frame from multiple images, and fuse the image features of each frame sequentially according to the acquisition time of each frame to obtain the food fusion features;

[0011] Based on the fusion characteristics of the ingredients, the types of ingredients accessed by the user during the opening and closing of the door are determined.

[0012] In this embodiment, during the opening and closing process of the smart refrigerator door, multiple frames of images can be acquired using an image acquisition device. Image features are then extracted from each frame, and the image features of each frame are sequentially fused according to the acquisition time to obtain food characteristics. Based on these characteristics, the type of food accessed by the user during the door opening and closing process is determined. This method allows for the fusion of image features from multiple frames acquired during the smart refrigerator's opening and closing process to obtain food characteristics and determine the type of food accessed by the user, thereby improving the accuracy of food type identification.

[0013] In one alternative implementation, the processor is specifically configured to:

[0014] Based on the acquisition time of each frame, each frame is sequentially used as the current image, and the following operations are performed on the current image:

[0015] The first feature is updated using the image features of the current image to obtain the updated first feature; the first feature is obtained based on the image features of the previous images.

[0016] The first feature, updated using the image features of the last frame, is used as the fusion feature of the ingredients.

[0017] In one alternative implementation, the processor is specifically configured to:

[0018] By identifying the input gate in the model, the key food features contained in the image features of the current image are retained in the first feature;

[0019] The first feature, which retains the key features of the ingredients, and the image features of the current image are input into the output gate of the recognition model to obtain the updated first feature.

[0020] In an alternative implementation, the processor is further configured to:

[0021] By identifying the forget gate in the model, non-food interference features contained in the image features of the current image are determined;

[0022] The non-food interference features are forgotten from the first feature.

[0023] In one alternative implementation, the key features of the food in an image located in the middle contain more information than the key features of the food in an image located at the edge.

[0024] In one alternative implementation, the processor is specifically configured to:

[0025] By identifying the fully connected layer in the model, the fusion features of the ingredients are classified to obtain the types of ingredients accessed by the user during the opening and closing of the door.

[0026] In one alternative implementation, the processor is specifically configured to:

[0027] Image features of each frame in multiple images are extracted using an image feature extraction network to obtain the initial features of each frame.

[0028] By identifying the convolutional layers in the model, the initial features of each frame of the image are enhanced to obtain the enhanced features of each frame of the image.

[0029] By identifying the pooling layers in the model, the enhancement features of each frame are reduced in dimensionality to obtain the image features of each frame.

[0030] In one alternative implementation, the processor is specifically configured to:

[0031] By identifying the convolutional layers in the model, image features of each frame in multiple frames are extracted to obtain the initial features of each frame.

[0032] By identifying the pooling layer in the model, the initial features of each frame are dimensionality-reduced to obtain the image features of each frame.

[0033] Secondly, embodiments of this application provide a method for identifying food ingredients, including:

[0034] Acquire multiple frames of images;

[0035] Extract the image features of each frame from multiple images, and fuse the image features of each frame sequentially according to the acquisition time of each frame to obtain the food fusion features;

[0036] Based on the fusion characteristics of the ingredients, the type information of the ingredients is determined.

[0037] In one optional implementation, the step of sequentially fusing the image features of each frame of the image according to the acquisition time of each frame to obtain the food ingredient fusion features includes:

[0038] Based on the acquisition time of each frame, each frame is sequentially used as the current image, and the following operations are performed on the current image:

[0039] The first feature is updated using the image features of the current image to obtain the updated first feature; the first feature is obtained based on the image features of the previous images.

[0040] The first feature, updated using the image features of the last frame, is used as the fusion feature of the ingredients.

[0041] Thirdly, embodiments of this application provide a food ingredient identification device, the device comprising:

[0042] The acquisition unit acquires multiple frames of images;

[0043] The fusion unit is used to extract the image features of each frame in multiple frames of images, and to fuse the image features of each frame in sequence according to the acquisition time of each frame of images to obtain the food fusion features;

[0044] The determining unit is used to determine the type information of the ingredients based on the fusion characteristics of the ingredients.

[0045] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the food storage and retrieval action recognition methods described in the first aspect.

[0046] The technical effects of any of the implementation methods in the second to fourth aspects can be found in the technical effects of the corresponding implementation methods in the first aspect, and will not be repeated here. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This application provides an illustration of an application scenario for a smart refrigerator.

[0049] Figure 2 A schematic diagram of the hardware structure of a smart refrigerator provided in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram of the structure of a smart refrigerator provided in an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of the structure of a smart refrigerator provided in an embodiment of this application;

[0052] Figure 5 This is a side view of a smart refrigerator provided in an embodiment of this application;

[0053] Figure 6 A configuration block diagram of a smart refrigerator provided in an embodiment of this application;

[0054] Figure 7 This application provides an architectural configuration block diagram of a memory operating system.

[0055] Figure 8 A flowchart illustrating a method for identifying food ingredients provided in this application embodiment;

[0056] Figure 9 A schematic diagram of a food ingredient image provided in an embodiment of this application;

[0057] Figure 10 A schematic diagram of a food ingredient image provided in an embodiment of this application;

[0058] Figure 11 A schematic diagram of a food ingredient image provided in an embodiment of this application;

[0059] Figure 12 A schematic diagram of a food ingredient image provided in an embodiment of this application;

[0060] Figure 13 A schematic diagram of a food ingredient image provided in an embodiment of this application;

[0061] Figure 14 A structural diagram of a recognition model provided in an embodiment of this application;

[0062] Figure 15 A flowchart for determining the fusion characteristics of ingredients provided in this application embodiment;

[0063] Figure 16 A structural diagram of a long short-term memory network model provided in an embodiment of this application;

[0064] Figure 17 A schematic diagram of the structure of a long short-term memory network model provided in an embodiment of this application;

[0065] Figure 18 A schematic diagram of the structure of a long short-term memory network model provided in an embodiment of this application;

[0066] Figure 19 A schematic diagram of the structure of a long short-term memory network model provided in an embodiment of this application;

[0067] Figure 20 This is a structural block diagram of a food ingredient identification device provided in an embodiment of this application. Detailed Implementation

[0068] To enable those skilled in the art to better understand the technical solutions in this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of this disclosure.

[0069] In the description of this disclosure, it should be understood that the terms “center,” “upper,” “lower,” “front,” “rear,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” and “outer,” etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this disclosure and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure.

[0070] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0071] In the description of this disclosure, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linkage" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this disclosure based on the specific circumstances.

[0072] Refrigerators, as a household appliance, are widely used in people's daily lives. With the continuous development of technology, precise management of food information has become a core function of refrigerator intelligence. Accurate identification of food type information is a crucial prerequisite for this precise management. Currently, the process of identifying food types typically involves extracting features from a single frame of an image captured during the user's food retrieval process and analyzing these features to determine the type of food being retrieved. However, this method only analyzes the features of a single frame, thus reducing the accuracy of food type identification.

[0073] To address this problem, this application proposes a food identification method. During the opening and closing process of a smart refrigerator door, multiple frames of images can be acquired using an image acquisition device. Image features are then extracted from each frame, and these features are sequentially fused according to the acquisition time of each frame to obtain food fusion features. Based on these fusion features, the type of food accessed by the user during the door opening and closing process is determined. This method allows for the fusion of image features from multiple frames acquired during the smart refrigerator's opening and closing process, resulting in food fusion features and the determination of the type of food accessed by the user, thereby improving the accuracy of food identification.

[0074] Figure 1 This is an application scenario diagram of a smart refrigerator provided in an embodiment of this application. See also... Figure 1 As shown, the smart refrigerator 100 communicates with the server 200 via various communication methods, including the Internet. Specifically, the smart refrigerator 100 communicates with other networks via a local area network (LAN), a wireless local area network (WLAN), and others.

[0075] Server 200 can provide various content and interactions to smart refrigerator 100. For example, smart refrigerator 100 can send and / or receive information from server 200, such as receiving moving target recognition data, accessing a remotely stored digital media library, or sending image data to be detected. Server 200 can be a group or multiple groups, and can be one or more types of servers. Server 200 can be deployed locally or in the cloud, enabling functions such as image recognition.

[0076] It should be noted that, Figure 1 The application scenario described above is merely an example of one application scenario for implementing the embodiments of this application, and the embodiments of this application are not limited to the above. Figure 1 The application scenario is illustrated. For example, in this scenario, there may be no server 200; the processor can directly identify the type of food stored by the user based on the image features of multiple frames of images obtained during the user's food storage process.

[0077] In the embodiments of this application, the type, size, and power of the smart refrigerator may vary.

[0078] In some embodiments, such as Figure 2 and Figure 3As shown, the smart refrigerator 100 includes at least a casing 110, a door 120, an image acquisition device 130, and a processor 140. The casing 110 has multiple storage compartments, each with a door 120. The door 120 is movably connected to the casing 110 and is used to cover the opening of the storage compartment. The image acquisition device 130 is connected to the casing 110 and is used to acquire images of the moving area of ​​the door. The processor 140 can be located in the casing 110 or in the door 120; this application is not limited to either.

[0079] In this embodiment, the smart refrigerator 100 can be a smart refrigerator with an electronic display screen on the refrigerator door, such as... Figure 4 As shown, when a user opens the door of the refrigerator's cold storage compartment to store food, the electronic display screen can show a structural model of the refrigerator, highlighting the open cold storage compartment and indicating that the user is storing food in the cold storage space. Optionally, the electronic display screen can also show the internal structure of the cold storage compartment. Other types of smart refrigerators can also be used, such as those with voice broadcast functions or those without an electronic display screen; this application does not limit the application to these types.

[0080] In this embodiment, the image acquisition device 130 can be disposed on the top of the housing 110, or in other parts of the housing 110, such as the bottom of the housing 110. The image acquisition device can be a color camera or other types of cameras, such as a depth camera. This embodiment uses a color camera as an example for explanation. Figure 3 and Figure 4 As shown, the color camera is installed at the top center of the outer wall of the smart refrigerator via a camera pop-up device. The color camera is located inside the camera pop-up device. When the user opens the door of the smart refrigerator, the camera pop-up device will pop the color camera out together.

[0081] Assume the smart refrigerator includes three storage compartments: a refrigerator compartment 150, a first freezer compartment 160, and a second freezer compartment 170. When the doors of all three compartments are closed, the camera pop-out device 131, installed at the top center of the outer wall of the refrigerator casing, is in a retractable state. Figure 5 As shown in the solid line, when the user opens any one of the three storage compartments of the smart refrigerator, the camera pop-up device 131 installed at the top center of the outer wall of the smart refrigerator casing is activated and then pops out, as shown in the solid line. Figure 5 As shown by the dashed line. Figure 5 This is a side view of the smart refrigerator and the camera pop-up device.

[0082] In this embodiment, the camera pop-up device 131 may include a control switch and a telescopic rod, with the image acquisition device fixed to the end of the telescopic rod. When the system senses that a user has opened the door of any storage compartment of the smart refrigerator, the processor controls the control switch of the camera pop-up device, causing the telescopic rod to pop out, which in turn causes the image acquisition device to pop out.

[0083] In the embodiments of this application, using Figure 3 The image acquisition device 130 is shown to be installed at the top center of the outer side of the smart refrigerator casing 110.

[0084] Figure 6 This is a configuration block diagram of a smart refrigerator provided in an embodiment of this application. Figure 6 As shown, it includes a controller 610, a memory 620, a communicator 630, a user input interface 640, a user output interface 650, a power supply 660, an image acquisition interface 670, and a display.

[0085] The controller 610 includes a random access memory (RAM) 611, a read-only memory (ROM) 612, a processor 140, a communication interface, and a communication bus. The controller 610 is used to control the operation of the above-mentioned devices, as well as the communication and cooperation between internal components and external and internal data processing functions.

[0086] For example, when an interaction of pressing the button 644 or touching the touchpad 642 is detected, the controller 610 can control the processor 140 to generate a signal corresponding to the detected interaction and send the signal to the display screen 50 so that the display screen 50 can display the corresponding content or image.

[0087] In some examples, processor 140 may be a central processing unit (CPU), a graphics processing unit (GPU), or a combination of a CPU and a GPU. The processor may further include hardware chips. These hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0088] Memory 620 is used to store various running programs, data, and applications for driving and control under the control of controller 610. Memory 620 can store various control signal instructions input by the user. In some examples, the memory is connected to the processor via a bus or other means, and the memory stores at least one instruction, at least one program, code set, or instruction set, which is loaded and executed by the processor. In some examples, the memory can be volatile memory, non-volatile memory, or a combination thereof. Volatile memory can be random-access memory (RAM), such as static random-access memory (SRAM) or dynamic random-access memory (DRAM). Non-volatile memory can be read-only memory (ROM), such as programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM). Non-volatile memory can also be flash memory, magnetic storage such as magnetic tape, floppy disk, and hard disk. Non-volatile memory can also be optical disc.

[0089] The communicator 630 can be a component used to communicate with external devices or external servers according to various communication protocol types. For example, a refrigerator can send content data to an external device connected via the communicator 630, or browse and download content data from an external device connected via the communicator 630. The communicator 630 may include network communication protocol modules or near-field communication protocol modules such as an infrared signal interface 631, a radio frequency signal interface 632, a WIFI module 633, a Bluetooth communication protocol module 634, and a wired Ethernet communication protocol module 635. Thus, the communicator 630 can realize communication of control signals and data signals with external devices or external servers according to the control of the controller 610. For example, with an infrared signal interface, the infrared control signal sent by the user needs to be converted according to the infrared control protocol and then output to the controller 610. For example, with a radio frequency signal interface, the command of the radio frequency control signal mode input by the user needs to be converted and then output to the controller 610. For example, the refrigerator receives control signals from external devices via the WIFI module 633, Bluetooth communication protocol module 634, and wired Ethernet communication protocol module 635, and outputs the processed control signals to the controller 610.

[0090] User input interface 640 may include at least one of microphone 641, touchpad 642, sensor 643, button 644, etc., so that the user can input user commands to control the refrigerator through voice, touch, gesture, pressing, etc.

[0091] User output interface 650 outputs user commands received from user input interface 640 to controller 610, which in turn controls processor 140 to execute corresponding program steps. Alternatively, after processor 140 executes the corresponding program steps, it controls display 680 to display a corresponding image or outputs corresponding content through the user output interface. Here, user output interface 650 may include LED interface 651, vibration interface 652, and sound output interface 653, etc. For example, display devices in external devices can receive audio, video, or data output signals from user output interface 650 and display the output signals as images on the display device, output audio through sound output interface 653, or output vibration through vibration interface 652.

[0092] Image acquisition interface 670 is used to connect the hyperspectral camera to the refrigerator. For example, the hyperspectral image and spectral response curve acquired by the hyperspectral camera can be transmitted to the processor 140 in the controller 610 through image acquisition interface 670.

[0093] The display screen 50 is used to receive image signals input from the processor 140 and to display video content, images, and a menu control interface. The displayed video content can be processed by the processor 140 or input from the communicator 630 or the user input interface 640. The display screen 50 can also simultaneously display the user interface (UI) for controlling the refrigerator.

[0094] The power supply 660 provides power to the various components in the refrigerator under the control of the controller 610. It can be in the form of a battery and related control circuitry.

[0095] Figure 7 The diagram illustrates an example of the architecture configuration of the operating system in memory 620. The operating system architecture, from top to bottom, consists of the application layer, middleware layer, and kernel layer.

[0096] The application layer includes both built-in system applications and non-system-level applications. It is responsible for direct interaction with the user. The application layer can include multiple applications, such as settings applications, e-post applications, and media center applications. These applications can be implemented as web applications, running on the WebKit engine, and specifically developed and executed using HTML5, Cascading Style Sheets (CSS), and JavaScript.

[0097] Here, HTML, short for HyperText Markup Language, is a standard markup language used to create web pages. It uses tags to describe web pages. HTML tags are used to describe text, graphics, animations, sounds, tables, links, etc. Browsers read HTML documents, interpret the content of the tags in the document, and display them in the form of web pages.

[0098] CSS, short for Cascading Style Sheets, is a computer language used to style HTML documents. It defines style structures such as fonts, colors, and positions. CSS styles can be stored directly in HTML web pages or in separate stylesheet files, allowing for control over the styles on a webpage.

[0099] JavaScript is a language used for web page programming. It can be inserted into HTML pages and interpreted and executed by the browser. The interactive logic of web applications is implemented using JavaScript. JavaScript can also communicate with the kernel layer through browser-defined JavaScript extension interfaces.

[0100] The middleware layer provides standardized interfaces to support operation in various environments and systems. For example, the middleware layer can be implemented as the Multimedia and Hypermedia Information Coding Experts Group (MHEG) middleware related to data broadcasting, as DLNA middleware related to communication with external devices, or as middleware providing the browser environment for applications running within display devices.

[0101] The kernel layer provides core system services, such as file management, memory management, process management, network management, and system security and permission management. The kernel layer can be implemented as a kernel based on various operating systems, such as a kernel based on the Linux operating system.

[0102] The kernel layer also provides communication between system software and hardware, and provides device driver services for various hardware, such as display drivers for monitors, camera drivers for cameras, button drivers for remote controls, WiFi drivers for WiFi modules, audio drivers for audio output interfaces, and power management drivers for power management (PM) modules.

[0103] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. In actual processing or when the device executes the method, it may be executed in the order shown in the embodiments or drawings, or in combination.

[0104] Figure 8 A flowchart illustrating a food ingredient identification method provided in an embodiment of this application is shown. Figure 8 As shown, the method may include the following steps:

[0105] Step S801: Acquire multiple frames of images.

[0106] Specifically, in one optional implementation, a preset number of images can be acquired according to a pre-set image acquisition event and image acquisition time interval.

[0107] The pre-set image acquisition event can be detecting that the refrigerator door is open and a user's hand is detected in the target area, or detecting that the refrigerator door changes from closed to open and a user's hand is detected in the target area. The preset image acquisition time interval can be 1 second; the preset number of images can be 5 frames. After detecting the image acquisition event, the smart refrigerator can start acquiring the first frame, and then acquire one frame every 1 second until 5 frames are acquired. In one embodiment, the acquired 5 frames are as follows: Figures 9-13 As shown.

[0108] In the embodiments of this application, the target area can be any of the storage compartments of the smart refrigerator or the door movement area; this application does not limit this.

[0109] Step S802: Extract the image features of each frame in the multi-frame images, and fuse the image features of each frame in sequence according to the acquisition time of each frame image to obtain the food fusion features.

[0110] In one optional implementation, during the process of extracting image features from each frame of a multi-frame image, the image features of each frame of the multi-frame image can be extracted first by using the convolutional layer in the recognition model to obtain the initial features of each frame of the image. Then, the initial features of each frame of the image can be reduced in dimensionality by using the pooling layer in the recognition model to obtain the image features of each frame of the image.

[0111] In this embodiment of the application, the recognition model may include multiple convolutional layers, each of which corresponds to a pooling layer. The initial feature vector of each frame image can be obtained through the convolutional layer, and the initial feature vector of each frame image can be reduced in dimensionality through the pooling layer.

[0112] For example, in one embodiment, assuming that 5 frames of images are acquired through step S801, and the recognition model contains 5 convolutional layers, each corresponding to a pooling layer, that is, the recognition model also contains 5 pooling layers, the acquired 5 frames of images can be input into the 5 convolutional layers of the recognition model respectively to obtain the initial feature vector of each frame of image, and then the initial feature vector of each image is dimensionality reduced through the corresponding pooling layer to obtain the image feature vector of the preset dimension, such as... Figure 14As shown, X1 is the image feature vector corresponding to the first frame, X2 is the image feature vector corresponding to the second frame, X3 is the image feature vector corresponding to the third frame, X4 is the image feature vector corresponding to the fourth frame, and X5 is the image feature vector corresponding to the fifth frame. Assuming the vectors in the pooling layer are one-dimensional, after dimensionality reduction of the initial feature vectors of each image through the corresponding pooling layer, we can obtain one-dimensional feature vectors for each image, i.e., X1, X2, X3, X4, and X5 are all one-dimensional vectors. These one-dimensional feature vectors are then used as the image feature vectors for each image.

[0113] In an optional implementation, after obtaining the image feature vectors of each image in the above manner, each frame can be sequentially taken as the current image according to the acquisition time of each frame, and the image feature vector of the current image can be taken as the current image feature vector, and then referred to... Figure 15 The method shown determines the fusion characteristics of ingredients, such as Figure 15 As shown, the method includes the following steps:

[0114] Step S1501: Update the first feature using the image features of the current image to obtain the updated first feature.

[0115] Step S1502: Determine whether the current image is the last frame; if yes, proceed to step S1503; if no, proceed to step S1501.

[0116] Step S1503: The first feature updated from the image features of the last frame image is used as the food fusion feature.

[0117] Specifically, after obtaining the image feature vectors of each image through convolutional and pooling layers, the image feature vector of the current image can be used to update the first feature vector corresponding to the first feature, resulting in the updated first feature vector. The first feature is obtained based on the image features of all images preceding the current image.

[0118] In one optional implementation, during the process of updating the first feature using the image features of the current image, the non-food interference features contained in the image features of the current image can be determined first by passing the forget gate in the recognition model, and the non-food interference features can be forgotten from the first feature. Then, the key food features contained in the image features of the current image can be retained in the first feature by passing the input gate in the recognition model. Finally, the first feature with the key food features retained and the image features of the current image are input into the output gate in the recognition model to obtain the updated first feature.

[0119] In this embodiment of the application, the recognition model may further include a Long Short-Term Memory (LSTM) network model, which may contain multiple layers, such as... Figure 16 As shown, each layer contains a forget gate, an input gate, and an output gate. The forget gate removes interference information from the network model, the input gate adds new information to the network model, and the output gate outputs the information from the network model.

[0120] Specifically, in some embodiments, the current image feature vector can be first input into the forget gate of the long short-term memory network model, and the interference feature vector corresponding to the non-food interference feature can be determined by formula 1, and the key feature vector corresponding to the key food feature can be determined by formulas 2 and 3.

[0121] f t =σ(W f ·[h t-1 ,x t ]+b f ) (Formula 1)

[0122] i t =σ(W i ·[h t-1 ,x t ]+b i ) (Formula 2)

[0123]

[0124] Where σ represents the sigmoid activation function, W f The weight of the forget gate, x t h represents the current image feature vector input at the current time. t-1 b represents the image feature vector output at the previous time step. f f represents the bias coefficient of the forget gate. t f represents the forgetting probability of the first feature vector output at the previous time step. t The value of f is greater than or equal to 0 and less than or equal to 1, where, when f t =1 indicates that the first feature vector output at the previous time step is completely retained, i.e., the forgetting probability is 0; when f t =0 indicates that the first feature vector output at the previous time step is completely discarded, i.e., the forgetting probability is 1; W i and W g The weight of the input gate, b i and b g i represents the bias coefficient of the input gate. t The value is greater than or equal to 0 and less than or equal to 1; The value is greater than or equal to -1 and less than or equal to 1.

[0125] After determining the interference feature vector and the key feature vector using the above method, the interference feature vector, the key feature vector and the first feature vector can be fused according to Formula 4 to obtain the updated first feature vector. That is, the interference feature vector is first deleted from the first feature vector, and then the key feature vector is added to the first feature vector.

[0126]

[0127] Among them, C t Let C represent the updated first feature vector. t-1 This represents the first eigenvector.

[0128] For example, in one embodiment, assuming the Long Short-Term Memory network model contains 5 layers, and the current image feature vector is the image feature vector corresponding to the first frame image, such as... Figure 17 As shown, at this time t=1, the current image feature vector is X1, and the first feature vector is 0. First, the current image feature vector X1 is input into the forget gate of the first layer, and the forgetting probability is determined according to formula 1. Then, the current image feature vector X1 is input into the input gate of the first layer, and the i is determined according to formulas 2 and 3. t and C1, the updated first feature vector, is determined according to Formula 4. Then, the current image feature vector X1 is input into the output gate of the first layer, and h1 is determined according to Formulas 5 and 6, where h1 is the hidden state output at the current time, i.e., the image feature vector.

[0129] o t =σ(W o ·[h t-1 ,x t ]+b o ) (Formula 5)

[0130] h t =o t *tanh(C t ) (Formula 6)

[0131] Among them, W o b represents the weight of the output gate. o This represents the bias coefficient of the output gate, o t The value is greater than or equal to 0 and less than or equal to 1.

[0132] After determining h1 using the above method, h1 can be input into the second layer of the Long Short-Term Memory network model. For example, suppose the current image feature vector is the image feature vector corresponding to the second frame image, such as... Figure 18As shown, at this time t=2, the current image feature vector is X2, and the first feature vector is C1. First, the current image feature vector X2 and the hidden state of the previous time step, i.e., h1, are input into the forget gate of the second layer. The forgetting probability is determined according to formula 1. Then, the current image feature vector X2 and h1 are input into the input gate of the second layer. The i2 and h1 are determined according to formulas 2 and 3. C2 is determined according to Formula 4. Then, the current image feature vectors X2 and h1 are input into the output gate of the second layer, h2 is determined according to Formulas 5 and 6, and h2 is input into the third layer of the Long Short-Term Memory network model.

[0133] Then, the image feature vectors corresponding to the third frame, the fourth frame, and the fifth frame are sequentially input into the third, fourth, and fifth layers of the Long Short-Term Memory network model to determine the updated first feature vector corresponding to the fifth frame. The specific implementation method is the same as that for the second frame, and will not be repeated here.

[0134] In one alternative implementation, during the process of determining the key feature vectors corresponding to the key features of the ingredients contained in each image through the input gate in the Long Short-Term Memory Network model, based on the position information of each frame image in the acquired multi-frame images, more feature information is retained for images in the middle position, i.e., the input gate is strengthened at this moment; and less feature information is retained for images in the edge position, i.e., the input gate is weakened at this moment.

[0135] Specifically, assuming a total of N frames of images are acquired, then the image located at... Images within the time range are determined using Formula 7. i within the time range t The graph at t=1 or t=N is used to determine i at t=1 or t=N using Formula 8. t .

[0136] i t =min{σ(W i ·[h t-1 ,x t ]+b i )+w t ,1} (Formula 7)

[0137] i t =max{σ(W i ·[h t-1 ,x t ]+b i )-w t ,0} (Formula 8)

[0138] Among them, w tThis represents the coefficient in the input gate that alters the value of the activation function.

[0139] For example, in one embodiment, assuming a total of 5 frames of images have been acquired, the current image feature vector is the image feature vector corresponding to the third frame image, at t=3, the current image feature vector is X3, and the first feature vector is C2. First, the current image feature vector X3 and the hidden state of the previous time step, i.e., h2, are input into the forget gate of the third layer. The forgetting probability is determined according to Formula 1. Then, the current image feature vector X3 and h2 are input into the input gate of the third layer because... Therefore, i3 and i3 can be determined based on formulas 7 and 3. C3 is then determined according to Formula 4. Next, the current image feature vectors X3 and h2 are input into the output gate of the third layer, h3 is determined according to Formulas 5 and 6, and h3 is input into the fourth layer of the Long Short-Term Memory network model.

[0140] In another embodiment, assuming a total of 5 frames were acquired, the current image feature vector is the image feature vector corresponding to the fifth frame, such as... Figure 19 As shown, at this time t=5, the current image feature vector is X5, and the first feature vector is C4. First, the current image feature vector X5 and the hidden state of the previous time step, i.e., h4, are input into the forget gate of the fifth layer. The forgetting probability is determined according to formula 1. Then, the current image feature vector X5 and h4 are input into the input gate of the fifth layer. Since t=N, i5 and h4 can be determined according to formulas 8 and 3. C5 is then determined according to Formula 4. Next, the current image feature vectors X5 and h4 are input into the output gate of the fifth layer, and h5 is determined according to Formulas 5 and 6. Since the fifth frame image is the last frame image, the food fusion features can be determined based on h5.

[0141] It should be noted that before using the Long Short-Term Memory (LSTM) network model to determine the food fusion features, a large amount of sample data needs to be obtained to train the LTM network model and obtain the optimal LTM network model.

[0142] Specifically, in the process of training a long short-term memory network model, a training sample set can be obtained first, and an initial long short-term memory network model can be constructed, that is, the weights and bias coefficients in the forget gate, and the weights, bias coefficients, and coefficient w in the input gate can be set. tThe weights and bias coefficients in the output gate are set to initial values. The training sample set is input into the initialized long short-term memory network model to obtain food features. Based on the difference between the food features and the actual food features, a loss function is determined. The long short-term memory network model is continuously optimized based on the loss function, thereby improving the weights and bias coefficients in the forget gate, and the weights, bias coefficients, and coefficient w in the input gate. t The weights and bias coefficients in the output gate are optimized, and the optimal long short-term memory network model is constructed based on the optimal value to predict each frame of the acquired image.

[0143] In an optional implementation, during the process of extracting image features from each frame of multiple images, the image features of each frame of multiple images can be extracted first through an image feature extraction network to obtain the initial features of each frame; then, the image features of each frame can be obtained through the convolutional and pooling layers in the Long Short-Term Memory network model.

[0144] In the embodiments of this application, the image feature extraction network can be ResNet34, ResNet50, etc., and this application does not limit it.

[0145] Step S803: Determine the type information of ingredients based on the fusion characteristics of the ingredients.

[0146] Specifically, the fusion features of the ingredients can be classified using the fully connected layer in a long short-term memory network model to obtain information on the types of ingredients accessed by the user during the opening and closing of the door.

[0147] In this embodiment of the application, the recognition model may also include a fully connected layer, which can classify image features based on image feature vectors to determine the types of food contained in the image.

[0148] After obtaining the first feature updated from the image features of the last frame image in step S802, the fully connected layer can classify the data based on the first feature updated from the image features of the last frame image to determine the type of food that the user accessed during the opening and closing of the door.

[0149] By using the above method, the image features of multiple frames of images acquired during the switching process of a smart refrigerator can be fused to obtain food fusion features and determine the type of food stored by the user, thereby improving the accuracy of identifying the type of food.

[0150] Based on the same inventive concept, embodiments of this application also provide a food ingredient identification device, such as... Figure 20 As shown, the food ingredient recognition device includes: an acquisition unit 2001, a fusion unit 2002, and a determination unit 2003, wherein:

[0151] Acquisition unit 2001 acquires multiple frames of images;

[0152] The fusion unit 2002 is used to extract the image features of each frame in multiple frames of images, and to fuse the image features of each frame in sequence according to the acquisition time of each frame of images to obtain the food fusion features;

[0153] The determining unit 2003 is used to determine the type information of the ingredients based on the fusion characteristics of the ingredients.

[0154] In an optional embodiment, the fusion unit 2002 is specifically used for:

[0155] Based on the acquisition time of each frame, each frame is sequentially used as the current image, and the following operations are performed on the current image:

[0156] The first feature is updated using the image features of the current image to obtain the updated first feature; the first feature is obtained based on the image features of the previous images.

[0157] The first feature, updated using the image features of the last frame, is used as the fusion feature of the ingredients.

[0158] In an optional embodiment, the fusion unit 2002 is specifically used for:

[0159] By identifying the input gate in the model, the key food features contained in the image features of the current image are retained in the first feature;

[0160] The first feature, which retains the key features of the ingredients, and the image features of the current image are input into the output gate of the recognition model to obtain the updated first feature.

[0161] In an optional embodiment, the fusion unit 2002 is specifically used for:

[0162] By identifying the forget gate in the model, non-food interference features contained in the image features of the current image are determined;

[0163] The non-food interference features are forgotten from the first feature.

[0164] In one alternative embodiment, the key features of the food in an image located in the middle contain more information than the key features of the food in an image located at the edge.

[0165] In an optional embodiment, the determining unit 2003 is specifically used for:

[0166] By identifying the fully connected layer in the model, the fusion features of the ingredients are classified to obtain the types of ingredients accessed by the user during the opening and closing of the door.

[0167] In an optional embodiment, the acquisition unit 2001 is specifically used for:

[0168] Image features of each frame in multiple images are extracted using an image feature extraction network to obtain the initial features of each frame.

[0169] By identifying the convolutional layers in the model, the initial features of each frame of the image are enhanced to obtain the enhanced features of each frame of the image.

[0170] By identifying the pooling layers in the model, the enhancement features of each frame are reduced in dimensionality to obtain the image features of each frame.

[0171] In an optional embodiment, the acquisition unit 2001 is specifically used for:

[0172] By identifying the convolutional layers in the model, image features of each frame in multiple frames are extracted to obtain the initial features of each frame.

[0173] By identifying the pooling layer in the model, the initial features of each frame are dimensionality-reduced to obtain the image features of each frame.

[0174] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0176] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0177] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0178] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A smart refrigerator, characterized in that, include: The casing has multiple storage compartments inside, and each storage compartment has a door; An image acquisition device, used to acquire images of the active area of ​​the door; The processor is configured to acquire multiple frames of images via the image acquisition device during the process of the door of the smart refrigerator opening and closing. Extract the image features of each frame from multiple images, and fuse the image features of each frame sequentially according to the acquisition time of each frame to obtain the food fusion features; Based on the fusion characteristics of the ingredients, the types of ingredients accessed by the user during the opening and closing of the door are determined. The processor is specifically configured as follows: Based on the acquisition time of each frame, each frame is sequentially used as the current image, and the following operations are performed on the current image: The first feature is updated using the image features of the current image to obtain the updated first feature; the first feature is obtained based on the image features of the previous images. The first feature, updated using the image features of the last frame, is used as the fusion feature of the ingredients; The processor is specifically configured as follows: By identifying the input gate of the long short-term memory network model contained in the model, the key features of the ingredients contained in the image features of the current image are retained in the first feature; the input gate is used to retain more feature information for images in the middle position and less feature information for images in the edge position, based on the position information of each frame image in the acquired multi-frame images; The first feature, which retains the key features of the ingredients, and the image features of the current image are input into the output gate of the Long Short-Term Memory Network model to obtain the updated first feature.

2. The intelligent refrigerator according to claim 1, characterized in that, The processor is also configured to: By identifying the forget gate in the model, non-food interference features contained in the image features of the current image are determined; The non-food interference features are forgotten from the first feature.

3. The intelligent refrigerator according to claim 1, characterized in that, The key features of ingredients in an image located in the middle contain more information than those in an image located at the edge.

4. The intelligent refrigerator according to claim 1, characterized in that, The processor is specifically configured as follows: By identifying the fully connected layer in the model, the fusion features of the ingredients are classified to obtain the types of ingredients accessed by the user during the opening and closing of the door.

5. The intelligent refrigerator according to claim 1, characterized in that, The processor is specifically configured as follows: Image features of each frame in multiple images are extracted using an image feature extraction network to obtain the initial features of each frame. By identifying the convolutional layers in the model, the initial features of each frame of the image are enhanced to obtain the enhanced features of each frame of the image. By identifying the pooling layers in the model, the enhancement features of each frame are reduced in dimensionality to obtain the image features of each frame.

6. The intelligent refrigerator according to claim 1, characterized in that, The processor is specifically configured as follows: By identifying the convolutional layers in the model, image features of each frame in multiple frames are extracted to obtain the initial features of each frame. By identifying the pooling layer in the model, the initial features of each frame are dimensionality-reduced to obtain the image features of each frame.

7. A method for identifying food ingredients, characterized in that, include: Acquire multiple frames of images; Extract the image features of each frame from multiple images, and fuse the image features of each frame sequentially according to the acquisition time of each frame to obtain the food fusion features; Based on the fusion characteristics of the ingredients, the type information of the ingredients is determined; The process involves extracting image features from each frame of a multi-frame image dataset and then fusing these features sequentially according to the acquisition time of each frame to obtain food ingredient fusion features, including: Based on the acquisition time of each frame, each frame is sequentially used as the current image, and the following operations are performed on the current image: The first feature is updated using the image features of the current image to obtain the updated first feature; the first feature is obtained based on the image features of the previous images. The first feature, updated using the image features of the last frame, is used as the fusion feature of the ingredients; The updated first feature is determined in the following manner: By identifying the input gate of the long short-term memory network model contained in the model, the key features of the ingredients contained in the image features of the current image are retained in the first feature; the input gate is used to retain more feature information for images in the middle position and less feature information for images in the edge position, based on the position information of each frame image in the acquired multi-frame images; The first feature, which retains the key features of the ingredients, and the image features of the current image are input into the output gate of the Long Short-Term Memory Network model to obtain the updated first feature.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the method of claim 7.

Images