Food material recognition method, smart refrigerator, and storage medium

Abstract

This application provides a food identification method, a smart refrigerator, and a storage medium. The method includes: acquiring multi-band images of packaged food to be placed from various perspectives, the multi-band images including images in the visible light band, a first near-infrared band, a second near-infrared band, and a third near-infrared band; generating fused images from various perspectives based on the multi-band images from various perspectives and a pre-acquired spectral library of packaging materials; identifying the category of the food to be placed based on the fused images from various perspectives and a pre-trained visual transformation model; reading real-time occupancy data of the smart refrigerator; determining the target placement area of ​​the food to be placed in the smart refrigerator based on the category of the food to be placed, the real-time occupancy data, and a pre-acquired food knowledge base; and outputting prompt information to prompt the user to place the food to be placed in the target placement area, thereby improving the recognition accuracy and placement recommendation accuracy of the smart refrigerator for packaged food to be placed.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a food identification method, a smart refrigerator, and a storage medium. Background Technology

[0002] With the development of artificial intelligence, smart refrigerators are gradually becoming an important part of home kitchens. Smart refrigerators automatically identify the types of food stored in them through built-in visible light cameras and image recognition algorithms.

[0003] However, most everyday food items come in plastic bags, cling film, or opaque packaging, which are difficult for visible light to penetrate. This makes it impossible to obtain effective information about the food itself, which can easily lead to misidentification or omission of food items, thus affecting the accuracy of food placement in smart refrigerators.

[0004] Currently, some solutions attempt to use Radio Frequency Identification (RFID) to identify food categories, but RFID requires RFID tags to be pre-attached to the food packaging and the food information is obtained by reading the RFID tags, which is costly.

[0005] Therefore, how to overcome the limitations of packaging obstruction and improve the accuracy of food recognition and food placement recommendations in smart refrigerators is an urgent problem to be solved. Summary of the Invention

[0006] The purpose of this application is to address the shortcomings of the prior art by providing a food identification method, a smart refrigerator, and a storage medium, so as to solve the practical problems of low identification accuracy and low placement recommendation accuracy for packaged food in the prior art.

[0007] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a method for identifying food ingredients, the method comprising: Acquire multi-band images of packaged food to be placed from various perspectives, wherein the multi-band images include: images in the visible light band, images in the first near-infrared band, images in the second near-infrared band, and images in the third near-infrared band; Based on the multi-band images of the food to be placed from various perspectives and the pre-acquired spectral library of packaging materials, a fused image of the food to be placed from various perspectives is generated. Based on the fused images of the ingredients to be placed from various perspectives and the pre-trained visual transformation model, the category of the ingredients to be placed is identified. The system reads the real-time occupancy data of the smart refrigerator, determines the target placement area of ​​the food to be placed in the smart refrigerator based on the category of the food to be placed, the real-time occupancy data, and the pre-acquired food knowledge base, and outputs a prompt message to remind the user to place the food to be placed in the target placement area.

[0008] As an optional implementation, generating a fused image of the food to be placed from various perspectives based on multi-band images of the food to be placed from different viewpoints and a pre-acquired spectral library of packaging materials includes: Based on the multi-band images of the food to be placed from various perspectives and the spectral library of the packaging material, the packaging material of the food to be placed is determined. Based on the packaging material, multi-band images of the food to be placed from various perspectives are fused to generate a fused image of the food to be placed from various perspectives.

[0009] As an optional implementation, determining the packaging material of the food to be placed based on multi-band images of the food from various viewpoints and the spectral library of packaging materials includes: Based on the images of the food to be placed in the first near-infrared band, the second near-infrared band, and the third near-infrared band from various viewing angles, the reflectance spectral characteristics of the packaging of the food to be placed are determined. The reflectance spectral characteristics of the packaging of the food to be placed are matched with the characteristics in the packaging material spectral library to obtain the matching result, and the packaging material of the food to be placed is determined based on the matching result.

[0010] As an optional implementation, the step of fusing multi-band images of the food to be placed from various viewpoints based on the packaging material to generate a fused image of the food to be placed from various viewpoints includes: Based on the packaging material, at least one band to be fused is determined, and an attention mechanism is used to determine the weight of each band to be fused. Based on the weights of each band to be fused, the images of the food to be placed in each band from each viewpoint are weighted and fused to generate a fused image of the food to be placed from each viewpoint.

[0011] As an optional implementation, determining at least one band to be fused includes: If the packaging material is the first material, then the first near-infrared band and the visible light band are taken as the bands to be fused. If the packaging material is the second material, then the second near-infrared band and the visible light band are used as the bands to be fused. If the packaging material is a third material, then the second near-infrared band and the third near-infrared band are used as the bands to be fused.

[0012] As an optional implementation, identifying the category of the food to be placed based on the fused images of the food from various viewpoints and a pre-trained visual transformation model includes: The fused images of the food to be placed from various viewpoints are input into the visual transformation model. The visual transformation model uses a sliding window self-attention mechanism to extract multi-scale features of the food to be placed from various viewpoints, and determines the category probability distribution of the food to be placed based on the multi-scale features. The category of the food to be placed is identified based on the probability distribution of the categories of the food to be placed.

[0013] As an optional implementation, determining the target placement area of ​​the food to be placed in the smart refrigerator based on the category of the food to be placed, the real-time occupancy data, and a pre-acquired food knowledge base includes: Based on the category of the food to be placed, query the storage attributes of the food to be placed in the food knowledge base; The target placement area is determined based on the storage attributes of the food to be placed and the current food list and remaining capacity of each area of ​​the smart refrigerator in the real-time occupancy data.

[0014] As an optional implementation, determining the target placement area based on the storage attributes of the food to be placed and the current food storage list and remaining capacity of each area of ​​the smart refrigerator in the real-time occupancy data includes: Based on the storage attributes of the food to be placed and the current food list and remaining capacity of each area of ​​the smart refrigerator in the real-time occupancy data, determine the food matching score, space score, food compatibility score and user history habit score for each area. Based on the ingredient matching score, space score, ingredient compatibility score, and user history habit score of each region, a comprehensive score for each region is determined. Based on the comprehensive score of each region and the remaining capacity, the target placement area is determined.

[0015] Secondly, embodiments of this application provide a food ingredient identification device, the device comprising: The acquisition module is used to acquire multi-band images of packaged food to be placed from various perspectives, wherein the multi-band images include: images in the visible light band, images in the first near-infrared band, images in the second near-infrared band, and images in the third near-infrared band. The generation module is used to generate a fused image of the food to be placed from various perspectives based on the multi-band images of the food to be placed from various perspectives and the pre-acquired spectral library of packaging materials. The recognition module is used to identify the category of the food to be placed based on the fused images of the food to be placed from various perspectives and a pre-trained visual transformation model. The determination module is used to read the real-time occupancy data of the smart refrigerator, determine the target placement area of ​​the food to be placed in the smart refrigerator based on the category of the food to be placed, the real-time occupancy data and the pre-acquired food knowledge base, and output a prompt message, which is used to prompt the user to place the food to be placed in the target placement area.

[0016] As an optional implementation, the generation module is specifically used for: Based on the multi-band images of the food to be placed from various perspectives and the spectral library of the packaging material, the packaging material of the food to be placed is determined. Based on the packaging material, multi-band images of the food to be placed from various perspectives are fused to generate a fused image of the food to be placed from various perspectives.

[0017] As an optional implementation, the generation module is specifically used for: Based on the images of the food to be placed in the first near-infrared band, the second near-infrared band, and the third near-infrared band from various viewing angles, the reflectance spectral characteristics of the packaging of the food to be placed are determined. The reflectance spectral characteristics of the packaging of the food to be placed are matched with the characteristics in the packaging material spectral library to obtain the matching result, and the packaging material of the food to be placed is determined based on the matching result.

[0018] As an optional implementation, the generation module is specifically used for: Based on the packaging material, at least one band to be fused is determined, and an attention mechanism is used to determine the weight of each band to be fused. Based on the weights of each band to be fused, the images of the food to be placed in each band from each viewpoint are weighted and fused to generate a fused image of the food to be placed from each viewpoint.

[0019] As an optional implementation, the generation module is specifically used for: If the packaging material is the first material, then the first near-infrared band and the visible light band are taken as the bands to be fused. If the packaging material is the second material, then the second near-infrared band and the visible light band are used as the bands to be fused. If the packaging material is a third material, then the second near-infrared band and the third near-infrared band are used as the bands to be fused.

[0020] As an optional implementation, the identification module is specifically used for: The fused images of the food to be placed from various viewpoints are input into the visual transformation model. The visual transformation model uses a sliding window self-attention mechanism to extract multi-scale features of the food to be placed from various viewpoints, and determines the category probability distribution of the food to be placed based on the multi-scale features. The category of the food to be placed is identified based on the probability distribution of the categories of the food to be placed.

[0021] As an optional implementation, the determining module is specifically used for: Based on the category of the food to be placed, query the storage attributes of the food to be placed in the food knowledge base; The target placement area is determined based on the storage attributes of the food to be placed and the current food list and remaining capacity of each area of ​​the smart refrigerator in the real-time occupancy data.

[0022] As an optional implementation, the determining module is specifically used for: Based on the storage attributes of the food to be placed and the current food list and remaining capacity of each area of ​​the smart refrigerator in the real-time occupancy data, determine the food matching score, space score, food compatibility score and user history habit score for each area. Based on the ingredient matching score, space score, ingredient compatibility score, and user history habit score of each region, a comprehensive score for each region is determined. Based on the comprehensive score of each region and the remaining capacity, the target placement area is determined.

[0023] Thirdly, embodiments of this application provide an intelligent refrigerator, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the intelligent refrigerator is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the food identification method as described in the first aspect above.

[0024] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the food identification method described in the first aspect above.

[0025] The beneficial effects of this application are: This application provides a food identification method, a smart refrigerator, and a storage medium. The method acquires multi-band images of packaged food to be placed from various viewpoints, including images in the visible light band, a first near-infrared band, a second near-infrared band, and a third near-infrared band. Based on these multi-band images and a pre-acquired spectral library of packaging materials, a fused image of the food to be placed from each viewpoint is generated. Using the fused image and a pre-trained visual transformation model, the method identifies the category of the food to be placed, reads real-time occupancy data of the smart refrigerator, and determines the target placement area for the food within the smart refrigerator based on the category, real-time occupancy data, and a pre-acquired food knowledge base. A prompt message is then output to guide the user to place the food in the target area. Based on multi-band images of the food to be placed from various perspectives and a spectral library of packaging materials, the system penetrates the outer packaging of the food to be placed, overcomes the limitations of packaging obstruction, and generates fused images of the food to be placed from various perspectives. This facilitates the improvement of the accuracy of the visual transformation model in identifying food based on the fused images of the food to be placed from various perspectives, thereby improving the accuracy of the food placement recommendation and extending the shelf life of the food to be placed. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 1 ; Figure 2 Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 2 ; Figure 3 Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 3 ; Figure 4 Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 4 ; Figure 5 Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 5 ; Figure 6 Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 6 ; Figure 7Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 7 ; Figure 8 Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 8 ; Figure 9 This is a schematic diagram of the internal space partitioning of a smart refrigerator provided in an embodiment of this application; Figure 10 This is a module structure diagram of the food identification device provided in the embodiments of this application; Figure 11 This is a schematic diagram of the structure of the smart refrigerator provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0029] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0030] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0031] Smart refrigerators automatically identify the types of food stored inside using built-in visible light cameras and image recognition algorithms. However, most everyday food items come in plastic bags, cling film, or opaque packaging, which visible light cannot penetrate. This makes it difficult to obtain effective information about the food itself, leading to frequent misidentification and missed identification, thus affecting the accuracy of food placement in the smart refrigerator. Currently, some solutions attempt to use RFID to identify food categories, requiring the pre-attachment of RFID tags to the food packaging and the acquisition of food information by reading the RFID tags, which is costly. Therefore, existing technologies have low accuracy in identifying and recommending the placement of packaged food.

[0032] Based on the above-mentioned problems, this application provides a food identification method that effectively overcomes the limitations of packaging obstruction and improves the accuracy of food identification and food placement recommendations in smart refrigerators.

[0033] Figure 1 Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 1 The subject executing this method can be a smart refrigerator. For example... Figure 1 As shown, the method includes: S101. Acquire multi-band images of packaged food ingredients to be placed from various perspectives.

[0034] The multi-band images include: images in the visible light band, images in the first near-infrared band, images in the second near-infrared band, and images in the third near-infrared band.

[0035] Optionally, a movable gimbal is installed on the internal guide rails of the smart refrigerator. The movable gimbal is equipped with a multispectral camera, which includes a visible light image sensor (covering the visible light band of 400nm-700nm) and multiple narrowband near-infrared image sensors (covering the first near-infrared band of 800nm-950nm, the second near-infrared band of 1000nm-1200nm, and the third near-infrared band of 1300nm-1500nm, respectively). When the user opens the smart refrigerator door, the camera captures images of the packaged food to be placed from multiple angles. Multispectral time-division imaging is used to obtain multi-band images of the packaged food from each angle, including images of the packaged food in the visible light band from each angle. Images in the first near-infrared band Images in the second near-infrared band and images in the third near-infrared band .

[0036] By acquiring multi-band images of packaged food items from various perspectives, blind spots caused by food stacking can be reduced. In the multi-band images, the visible light band can capture visual features such as surface color and texture of the food items; the first near-infrared band has good penetration through plastic packaging such as polyethylene and polypropylene; the second near-infrared band has good penetration through materials such as cling film and cardboard boxes; and the third near-infrared band can enhance the penetration effect through thicker packaging or composite material packaging.

[0037] It is worth noting that a single high-sensitivity camera can also be used in conjunction with a switchable filter wheel to acquire multi-band images in a time-division manner. Specifically, a motor-driven filter wheel is installed in front of a single high-sensitivity camera, sequentially switching between a visible light filter, a first near-infrared filter, a second near-infrared filter, and a third near-infrared filter to obtain multi-band images. This application does not impose specific limitations on the method of acquiring multi-band images.

[0038] S102. Based on the multi-band images of the food to be placed from various perspectives and the pre-acquired spectral library of packaging materials, generate a fused image of the food to be placed from various perspectives.

[0039] Optionally, the smart refrigerator communicates with a cloud server to obtain a packaging material spectral library, food knowledge base, and model parameter library from the cloud server, and stores these libraries locally. The cloud server can aggregate multi-source data, continuously optimize the packaging material spectral library, food knowledge base, and model parameter library through big data analysis, and distribute them to the smart refrigerator in real time. The smart refrigerator continuously updates its locally stored packaging material spectral library, food knowledge base, and model parameter library, thereby continuously improving its intelligence level.

[0040] The smart refrigerator uses multi-band images of the food to be placed from various perspectives and a locally stored spectral library of packaging materials to perform pixel-level fusion of these images. This process penetrates the outer packaging of the food to generate a fused image of the food from all perspectives. .

[0041] S103. Based on the fused images of the ingredients to be placed from various perspectives and the pre-trained visual transformation model, identify the category of the ingredients to be placed.

[0042] Optionally, the model parameter library stored locally in the smart refrigerator includes a visual transformation model pre-trained on a cloud server and the model parameters of the visual transformation model.

[0043] A composite image of ingredients to be placed from various perspectives. The input is fed into a visual transformation model, which then fuses the images of the food items to be placed from various viewpoints based on the model parameters. It accurately identifies the type of food to be placed, such as salmon and apples.

[0044] It is worth noting that if the food to be placed is unpackaged, images of the food in the visible light spectrum from various viewing angles will be used directly. By inputting the visual transformation model, the category of the food to be placed is obtained, without the need for image fusion processing.

[0045] S104. Read the real-time occupancy data of the smart refrigerator, determine the target placement area of ​​the food to be placed in the smart refrigerator based on the category of the food to be placed, the real-time occupancy data and the pre-acquired food knowledge base, and output a prompt message.

[0046] The prompt message is used to guide users to place the food items to be placed in the target placement area.

[0047] Optionally, the smart refrigerator reads real-time occupancy data, including the current list of stored food items and remaining capacity in each area of ​​the smart refrigerator. For example, the smart refrigerator may include eight areas: freezer drawer 1, freezer drawer 2, variable temperature compartment 1, variable temperature compartment 2, refrigerator drawer 1, refrigerator drawer 2, fruit and vegetable drawer, and door shelves.

[0048] Based on the category of the food to be placed, real-time occupancy data, and a locally stored food knowledge base, the smart refrigerator analyzes and determines the recommended placement area for the food within the refrigerator, which serves as the target placement area. The smart refrigerator then generates a prompt message based on the target placement area, guiding the user to place the food in the target area via screen display and / or voice prompts.

[0049] For example, the smart refrigerator displays different areas on the screen and highlights the target placement area with a highlighted border or fill color, while broadcasting voice commands such as "Please put the salmon into the second variable temperature compartment".

[0050] After detecting that the user has placed the food to be placed in the target area, the smart refrigerator updates the real-time occupancy data to ensure the accuracy of the real-time occupancy data.

[0051] In this embodiment, multi-band images of packaged food to be placed are acquired from various viewpoints, including images in the visible light band, the first near-infrared band, the second near-infrared band, and the third near-infrared band. Based on these multi-band images and a pre-acquired spectral library of packaging materials, a fused image of the food to be placed from each viewpoint is generated. Using this fused image and a pre-trained visual transformation model, the category of the food to be placed is identified. Real-time occupancy data of the smart refrigerator is read. Based on the category, real-time occupancy data, and a pre-acquired food knowledge base, the target placement area for the food in the smart refrigerator is determined, and a prompt message is output to guide the user to place the food in the target area. By using the multi-band images and spectral library of packaging materials, the fused image of the food is generated from its outer packaging, overcoming packaging obstruction. This improves the accuracy of the visual transformation model's food identification based on the fused image, thereby enhancing the accuracy of food placement recommendations and extending the shelf life of the food.

[0052] Figure 2 Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 2 ,like Figure 2 As shown, in step S102 above, based on the multi-band images of the food to be placed from various viewpoints and the pre-acquired spectral library of packaging materials, a fused image of the food to be placed from various viewpoints is generated, including: S201. Determine the packaging material of the food to be placed based on multi-band images of the food from various perspectives and the spectral library of packaging materials.

[0053] Optionally, the smart refrigerator performs spectral analysis on multi-band images of the food to be placed from various perspectives to obtain spectral analysis results. The spectral analysis results and the packaging material spectral library are then input into a packaging material classifier. The packaging material classifier uses a decision tree model based on the spectral analysis results and the packaging material spectral library to output the packaging material of the food to be placed.

[0054] For example, the packaging material of the food to be placed may include plastic materials such as polyethylene, polypropylene, and polyvinyl chloride, cling film, cardboard box, etc.

[0055] S202. Based on the packaging material, fuse the multi-band images of the food to be placed from various perspectives to generate a fused image of the food to be placed from various perspectives.

[0056] Optionally, the smart refrigerator determines a fusion strategy that matches the packaging material of the food to be placed, and performs pixel-level fusion of multi-band images of the food to be placed from various perspectives according to the fusion strategy to generate a fused image of the food to be placed from various perspectives. .

[0057] By employing a fusion strategy that adapts to the packaging material, the fused image can accurately represent the characteristics of the food to be placed after penetrating the packaging, thus overcoming the limitations of packaging obstruction.

[0058] In this embodiment, the packaging material of the food to be placed is determined based on multi-band images of the food from various viewpoints and a spectral library of packaging materials. Then, based on the packaging material, the multi-band images of the food from various viewpoints are fused to generate a fused image of the food from each viewpoint. This allows the fused image to accurately represent the characteristics of the food after it has penetrated the packaging, overcoming the limitations of packaging obstruction.

[0059] Figure 3 Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 3 ,like Figure 3 As shown, in step S201 above, the packaging material of the food to be placed is determined based on multi-band images of the food from various viewpoints and the spectral library of packaging materials, including: S301. Based on the images of the food to be placed in the first near-infrared band, the second near-infrared band, and the third near-infrared band from various viewing angles, determine the reflectance spectral characteristics of the packaging of the food to be placed.

[0060] Optionally, the smart refrigerator uses images of the food to be placed in the first near-infrared band from various viewing angles. Images in the second near-infrared band and images in the third near-infrared band The reflectance spectral characteristics of the packaging of the food to be placed were extracted.

[0061] For example, reflectance spectral characteristics may include the gray value ratio and peak position of each near-infrared band.

[0062] S302. Match the reflectance spectral characteristics of the packaging of the food to be placed with the characteristics in the packaging material spectral library to obtain the matching results, and determine the packaging material of the food to be placed based on the matching results.

[0063] Optionally, the smart refrigerator uses the reflectance spectral characteristics of the packaging of the food to be placed as the spectral analysis result, inputs it into the packaging material classifier, and the packaging material classifier uses a decision tree model to match the reflectance spectral characteristics with each reflectance spectral characteristic in the packaging material spectral library to obtain the matching result.

[0064] The packaging material spectral library stores the mapping relationship between various packaging materials and their reflectance spectral characteristics. The matching results are used to indicate the target reflectance spectral characteristics in the packaging material spectral library that match the reflectance spectral characteristics of the food to be placed. The smart refrigerator uses the packaging material corresponding to the target reflectance spectral characteristics as the packaging material for the food to be placed.

[0065] In this embodiment, the reflectance spectral characteristics of the packaging of the food to be placed are determined based on images of the food to be placed in the first near-infrared band, the second near-infrared band, and the third near-infrared band from various viewing angles. These reflectance spectral characteristics are then matched with features in a packaging material spectral library to obtain a matching result. The packaging material of the food to be placed is then determined based on the matching result, ensuring the accuracy of the identification of the packaging material.

[0066] Figure 4 Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 4 ,like Figure 4 As shown, in step S202 above, based on the packaging material, multi-band images of the food to be placed from various perspectives are fused to generate a fused image of the food to be placed from various perspectives, including: S401. Based on the packaging material, determine at least one band to be fused, and use an attention mechanism to determine the weight of each band to be fused.

[0067] Optionally, different packaging materials have different transmittance for different wavelengths. The smart refrigerator pre-stores the correspondence between each packaging material and the wavelength to be fused. Based on the identified packaging material of the food to be placed and the correspondence between each packaging material and the wavelength to be fused, at least one wavelength to be fused can be determined that can penetrate the packaging material and clearly present the characteristics of the food to be placed inside the packaging.

[0068] For each band to be merged, an attention mechanism is introduced to determine the weight of each band. Specifically, if a band to be merged clearly presents the characteristics of the food to be placed inside the packaging material, a higher weight is assigned to that band, while a lower weight is assigned to bands with poor transparency.

[0069] S402. Based on the weight of each band to be fused, perform weighted fusion of the images of the food to be placed in each band from each viewpoint to generate a fused image of the food to be placed from each viewpoint.

[0070] Optionally, based on the weights of each band to be fused, the smart refrigerator performs pixel-level weighted fusion of the images of the food to be placed in each band from each viewing angle to obtain a fused image of the food to be placed in each viewing angle. .

[0071] Among them, fused images It combines the penetrating advantages of different wavelengths, greatly enhancing the penetration of the food to be placed inside the packaging.

[0072] In this embodiment, at least one wavelength band to be fused is determined based on the packaging material. An attention mechanism is used to determine the weight of each wavelength band to be fused. Based on the weight of each wavelength band, the images of the food to be placed in each viewpoint are weighted and fused to generate a fused image of the food to be placed in each viewpoint. This allows the fused image to combine the penetration advantages of different wavelength bands, enhancing the penetration of the food to be placed in the packaging and facilitating accurate identification of the category of the food to be placed in the packaging.

[0073] Figure 5 Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 5 ,like Figure 5 As shown, in step S401 above, determining at least one band to be fused includes: S501. If the packaging material is the first material, then the first near-infrared band and the visible light band shall be used as the bands to be fused.

[0074] Optionally, if the packaging material is a first material, such as polyethylene, polypropylene, or polyvinyl chloride, then the first near-infrared band and the visible light band are used as the bands to be fused.

[0075] In this way, images in the first near-infrared band can be obtained. Images in the visible light band Perform pixel-level fusion to enhance the penetrating area and obtain a fused image. .

[0076] S502. If the packaging material is the second material, then the second near-infrared band and the visible light band shall be used as the bands to be fused.

[0077] Optionally, if the packaging material is a second material, and the second material is a plastic wrap, then the second near-infrared band and the visible light band are used as the bands to be fused.

[0078] In this way, images in the second near-infrared band can be obtained. Images in the visible light band Perform pixel-level fusion to obtain a fused image. Among them, the image in the second near-infrared band. Images in the visible light band are used to extract the penetration information of the food to be placed. Used to supplement the presentation details of the ingredients to be placed.

[0079] S503. If the packaging material is a third material, then the second near-infrared band and the third near-infrared band shall be used as the bands to be fused.

[0080] Optionally, if the packaging material is a third material, and the third material is a cardboard box, then the second near-infrared band and the third near-infrared band are used as the bands to be fused.

[0081] In this way, images in the second near-infrared band can be obtained. Images of the third near-infrared band Pixel-level fusion is performed to suppress paperboard fiber interference, resulting in a fused image. .

[0082] In this embodiment, if the packaging material is a first material, the first near-infrared band and the visible light band are used as the bands to be fused. If the packaging material is a second material, the second near-infrared band and the visible light band are used as the bands to be fused. If the packaging material is a third material, the second near-infrared band and the third near-infrared band are used as the bands to be fused. By flexibly selecting the appropriate bands to be fused according to the packaging material, the penetration of the packaging of the food to be placed is ensured, thus overcoming the limitations of packaging obstruction.

[0083] Figure 6 Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 6 ,like Figure 6 As shown, in step S103 above, the category of the food to be placed is identified based on the fused images of the food from various viewpoints and the pre-trained visual transformation model, including: S601. Input the fused images of the ingredients to be placed from various perspectives into the visual transformation model. The visual transformation model uses a sliding window self-attention mechanism to extract the multi-scale features of the ingredients to be placed from various perspectives, and determines the category probability distribution of the ingredients to be placed based on the multi-scale features.

[0084] Optionally, the smart refrigerator can merge images of the food to be placed from various perspectives. The data is input into a visual transformation model, which uses a sliding window self-attention mechanism based on the model parameters to extract multi-scale features of the food to be placed from various viewpoints. The sliding window self-attention mechanism allows for the inclusion of both local details and global relationships of the food across multiple scales.

[0085] The visual transformation model fuses or pools the multi-scale features of the extracted food ingredients from various viewpoints to obtain the feature vectors corresponding to the food ingredients. The classification layer of the visual transformation model outputs the probability values ​​of the food ingredients belonging to each category, thus obtaining the category probability distribution of the food ingredients. The category probability distribution is used to characterize the classification confidence of the food ingredients.

[0086] S602. Identify the category of the food to be placed based on the probability distribution of the categories of the food to be placed.

[0087] Optionally, based on the probability distribution of the categories of the ingredients to be placed, the category with the highest classification confidence that is greater than a preset confidence threshold is selected as the category of the ingredients to be placed.

[0088] For example, the preset confidence threshold can be 0.85.

[0089] In this embodiment, fused images of the ingredients to be placed from various viewpoints are input into a visual transformation model. The visual transformation model uses a sliding window self-attention mechanism to extract multi-scale features of the ingredients from each viewpoint, and determines the category probability distribution of the ingredients based on these multi-scale features. Based on this category probability distribution, the category of the ingredients is identified. This approach accurately identifies the category of ingredients inside the packaging, penetrating the limitations of the outer packaging. The sliding window self-attention mechanism balances local details with global correlations, significantly improving the accuracy of ingredient identification within the packaging.

[0090] Figure 7 Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 7 ,like Figure 7 As shown, in step S104 above, the target placement area of ​​the food to be placed in the smart refrigerator is determined based on the category of the food to be placed, real-time occupancy data, and a pre-acquired food knowledge base, including: S701. Based on the category of the ingredients to be placed, query the storage attributes of the ingredients to be placed in the ingredient knowledge base.

[0091] Optionally, the smart refrigerator uses the category of the food to be placed as an index to query the storage attributes of the food in a locally stored food knowledge base. These storage attributes may include the initial recommended storage area for the food, information on incompatible food pairings, suitable temperature range, and estimated shelf life.

[0092] Specifically, if the type of food to be placed is apples, the initial recommended storage area in the search results is the fruit and vegetable drawer. The prohibited pairing information is that apples release ethylene and should not be placed with potatoes. The suitable temperature range is -8℃ to 4℃.

[0093] S702. Determine the target placement area based on the storage attributes of the food to be placed and the current list of stored food and remaining capacity of each area of ​​the smart refrigerator in the real-time occupancy data.

[0094] Optionally, the real-time occupancy data records the current list of stored food items and the remaining capacity of each area of ​​the smart refrigerator. The current list of stored food items in each area is used to determine the compatibility between the food to be placed and the currently stored food items, and the remaining capacity is used to determine whether each area can currently accommodate the food to be placed.

[0095] The smart refrigerator performs multi-objective optimization recommendations based on the storage attributes of the food to be placed, the current food list and remaining capacity of each area of ​​the smart refrigerator in real-time occupancy data, and the matching between the category of the food to be placed and the area, the compatibility between the food to be placed and the food already stored in each area, the capacity of each area, and the user's historical habits as optimization objectives. This results in a recommended placement area for the food to be placed in the smart refrigerator, which serves as the target placement area. This ensures that the food to be placed can be safely and compatiblely stored with other food in the target placement area, thereby extending the shelf life of the food to be placed in the target placement area and the food currently stored.

[0096] In this embodiment, based on the category of the food to be placed, the storage attributes of the food to be placed are queried in the food knowledge base. Based on the storage attributes of the food to be placed, as well as the current food storage list and remaining capacity of each area of ​​the smart refrigerator in real-time occupancy data, the target placement area is determined. This improves the accuracy of the target placement area and extends the shelf life of the food to be placed and the currently stored food within the target placement area.

[0097] Figure 8 Flowchart of the food ingredient identification method provided in the embodiments of this application Figure 8 ,like Figure 8 As shown, in step S702 above, the target placement area is determined based on the storage attributes of the food to be placed and the current list of stored food and remaining capacity of each area of ​​the smart refrigerator in the real-time occupancy data, including: S801. Based on the storage attributes of the food to be placed and the current list of stored food and remaining capacity of each area of ​​the smart refrigerator in the real-time occupancy data, determine the food matching score, space score, food compatibility score and user history habit score for each area.

[0098] Optionally, the smart refrigerator, based on the storage attributes of the food to be placed and the current food list and remaining capacity of each zone in the real-time occupancy data, determines the matching degree between the category of the food to be placed and the zone, based on the initial recommended zone and suitable temperature range of the food in the storage attributes, and thus obtains a food matching degree score for each zone. Figure 9 This is a schematic diagram of the internal space partitioning of a smart refrigerator provided in an embodiment of this application. Figure 9 It shows the names, temperature ranges, and suitable food types for each zone inside the smart refrigerator.

[0099] Based on the remaining capacity of each zone in the smart refrigerator according to real-time occupancy data, the capacity of each zone is determined, thus obtaining a space score for each zone. Based on the incompatible food pairing information in the storage attributes and the current food list in each zone of the smart refrigerator according to real-time occupancy data, the capacity of each zone is determined, thus obtaining a food compatibility score for each zone. By learning the user's historical placement behavior, the user's historical habits are determined, thus obtaining a user historical habit score for each zone.

[0100] S802. Based on the ingredient matching score, space score, ingredient compatibility score, and user history habit score of each area, determine the comprehensive score of each area, and determine the target placement area based on the comprehensive score of each area and the remaining capacity.

[0101] Optionally, the smart refrigerator can sum up the food matching score, space score, food compatibility score, and user history habit score for each area to obtain a comprehensive score for each area.

[0102] If the area with the highest overall score has enough remaining capacity to accommodate the food to be placed, the smart refrigerator will use that optimal area as the target placement area. Otherwise, it will continue to check if the area with the second highest overall score has enough remaining capacity to accommodate the food to be placed. If so, that second-best area will be used as the target placement area.

[0103] In this embodiment, based on the storage attributes of the food to be placed and the current food list and remaining capacity of each area of ​​the smart refrigerator in real-time occupancy data, a food matching score, a space score, a food compatibility score, and a user historical habit score are determined for each area. Then, based on these scores, a comprehensive score is determined for each area. Finally, based on the comprehensive score and remaining capacity of each area, the target placement area is determined. This multi-objective optimization recommendation improves the accuracy of the target placement area.

[0104] Based on the same inventive concept, this application also provides a food identification device corresponding to the food identification method. Since the principle of the device in this application is similar to that of the food identification method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0105] Figure 10 This is a module structure diagram of the food identification device provided in the embodiments of this application, as shown below. Figure 10 As shown, the device includes: The acquisition module 901 is used to acquire multi-band images of packaged food to be placed from various perspectives. The multi-band images include: images in the visible light band, images in the first near-infrared band, images in the second near-infrared band, and images in the third near-infrared band. The generation module 902 is used to generate a fused image of the food to be placed from various perspectives based on the multi-band images of the food to be placed from various perspectives and the pre-acquired spectral library of packaging materials. The recognition module 903 is used to identify the category of the food to be placed based on the fused images of the food to be placed from various perspectives and the pre-trained visual transformation model. The determination module 904 is used to read the real-time occupancy data of the smart refrigerator, determine the target placement area of ​​the food to be placed in the smart refrigerator based on the category of the food to be placed, the real-time occupancy data, and the pre-acquired food knowledge base, and output a prompt message to remind the user to place the food to be placed in the target placement area.

[0106] As an optional implementation, the generation module 902 is specifically used for: Based on the multi-band images of the food to be placed from various angles and the spectral library of packaging materials, the packaging material of the food to be placed is determined. Based on the packaging material, multi-band images of the food to be placed from various perspectives are fused to generate a fused image of the food to be placed from various perspectives.

[0107] As an optional implementation, the generation module 902 is specifically used for: Based on the images of the food to be placed in the first near-infrared band, the second near-infrared band, and the third near-infrared band from various viewing angles, the reflectance spectral characteristics of the food packaging are determined. The reflectance spectral characteristics of the packaging of the food to be placed are matched with the characteristics in the packaging material spectral library to obtain the matching results. The packaging material of the food to be placed is then determined based on the matching results.

[0108] As an optional implementation, the generation module 902 is specifically used for: Based on the packaging material, at least one band to be fused is identified, and an attention mechanism is used to determine the weight of each band to be fused. Based on the weights of each band to be fused, the images of the food to be placed in each band from each viewpoint are weighted and fused to generate a fused image of the food to be placed from each viewpoint.

[0109] As an optional implementation, the generation module 902 is specifically used for: If the packaging material is the first material, then the first near-infrared band and the visible light band will be used as the bands to be fused. If the packaging material is the second material, then the second near-infrared band and the visible light band will be used as the bands to be fused. If the packaging material is a third material, then the second and third near-infrared bands will be used as the bands to be fused.

[0110] As an optional implementation, the identification module 903 is specifically used for: The fused images of the food to be placed from various perspectives are input into the visual transformation model. The visual transformation model uses a sliding window self-attention mechanism to extract the multi-scale features of the food to be placed from various perspectives, and determines the category probability distribution of the food to be placed based on the multi-scale features. Identify the category of the food to be placed based on the probability distribution of the food categories.

[0111] As an optional implementation, the determining module 904 is specifically used for: Based on the category of the food to be placed, query the storage attributes of the food in the food knowledge base; The target placement area is determined based on the storage attributes of the food to be placed, as well as the current food list and remaining capacity of each area of ​​the smart refrigerator in real-time occupancy data.

[0112] As an optional implementation, the determining module 904 is specifically used for: Based on the storage attributes of the food to be placed and the current food list and remaining capacity of each area of ​​the smart refrigerator in real-time occupancy data, determine the food matching score, space score, food compatibility score and user history habit score for each area. Based on the ingredient matching score, space score, ingredient compatibility score, and user history habit score of each region, a comprehensive score for each region is determined. Based on the comprehensive score of each region and the remaining capacity, the target placement area is determined.

[0113] This application also provides an intelligent refrigerator, such as... Figure 11 The diagram shown is a structural schematic of a smart refrigerator provided in an embodiment of this application, including a processor 101, a memory 102, and a bus 103. The memory 102 stores machine-readable instructions executable by the processor 101. When the smart refrigerator is running, the processor 101 communicates with the memory 102 via the bus 103, and the processor 101 executes the machine-readable instructions to perform the steps of the food identification method in the aforementioned embodiment.

[0114] This application also provides a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the food ingredient identification method described in the foregoing embodiments.

[0115] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0116] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0117] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for identifying food ingredients, characterized in that, include: Acquire multi-band images of packaged food to be placed from various perspectives, wherein the multi-band images include: images in the visible light band, images in the first near-infrared band, images in the second near-infrared band, and images in the third near-infrared band; Based on the multi-band images of the food to be placed from various perspectives and the pre-acquired spectral library of packaging materials, a fused image of the food to be placed from various perspectives is generated. Based on the fused images of the ingredients to be placed from various perspectives and the pre-trained visual transformation model, the category of the ingredients to be placed is identified. The system reads the real-time occupancy data of the smart refrigerator, determines the target placement area of ​​the food to be placed in the smart refrigerator based on the category of the food to be placed, the real-time occupancy data, and the pre-acquired food knowledge base, and outputs a prompt message to remind the user to place the food to be placed in the target placement area.

2. The method according to claim 1, characterized in that, The step of generating a fused image of the food to be placed from various perspectives based on multi-band images of the food to be placed from different viewpoints and a pre-acquired spectral library of packaging materials includes: Based on the multi-band images of the food to be placed from various perspectives and the spectral library of the packaging material, the packaging material of the food to be placed is determined. Based on the packaging material, multi-band images of the food to be placed from various perspectives are fused to generate a fused image of the food to be placed from various perspectives.

3. The method according to claim 2, characterized in that, The step of determining the packaging material of the food to be placed based on multi-band images of the food from various viewpoints and the spectral library of packaging materials includes: Based on the images of the food to be placed in the first near-infrared band, the second near-infrared band, and the third near-infrared band from various viewing angles, the reflectance spectral characteristics of the packaging of the food to be placed are determined. The reflectance spectral characteristics of the packaging of the food to be placed are matched with the characteristics in the packaging material spectral library to obtain the matching result, and the packaging material of the food to be placed is determined based on the matching result.

4. The method according to claim 2, characterized in that, The step of fusing multi-band images of the food to be placed from various perspectives based on the packaging material to generate a fused image of the food to be placed from various perspectives includes: Based on the packaging material, at least one band to be fused is determined, and an attention mechanism is used to determine the weight of each band to be fused. Based on the weights of each band to be fused, the images of the food to be placed in each band from each viewpoint are weighted and fused to generate a fused image of the food to be placed from each viewpoint.

5. The method according to claim 4, characterized in that, The determination of at least one band to be fused includes: If the packaging material is the first material, then the first near-infrared band and the visible light band are taken as the bands to be fused. If the packaging material is the second material, then the second near-infrared band and the visible light band are used as the bands to be fused. If the packaging material is a third material, then the second near-infrared band and the third near-infrared band are used as the bands to be fused.

6. The method according to claim 1, characterized in that, The step of identifying the category of the food to be placed based on the fused images of the food to be placed from various viewpoints and a pre-trained visual transformation model includes: The fused images of the food to be placed from various viewpoints are input into the visual transformation model. The visual transformation model uses a sliding window self-attention mechanism to extract multi-scale features of the food to be placed from various viewpoints, and determines the category probability distribution of the food to be placed based on the multi-scale features. The category of the food to be placed is identified based on the probability distribution of the categories of the food to be placed.

7. The method according to claim 1, characterized in that, The step of determining the target placement area of ​​the food to be placed in the smart refrigerator based on the category of the food to be placed, the real-time occupancy data, and the pre-acquired food knowledge base includes: Based on the category of the food to be placed, query the storage attributes of the food to be placed in the food knowledge base; The target placement area is determined based on the storage attributes of the food to be placed and the current food list and remaining capacity of each area of ​​the smart refrigerator in the real-time occupancy data.

8. The method according to claim 7, characterized in that, The step of determining the target placement area based on the storage attributes of the food to be placed and the current food storage list and remaining capacity of each area of ​​the smart refrigerator in the real-time occupancy data includes: Based on the storage attributes of the food to be placed and the current food list and remaining capacity of each area of ​​the smart refrigerator in the real-time occupancy data, determine the food matching score, space score, food compatibility score and user history habit score for each area. Based on the ingredient matching score, space score, ingredient compatibility score, and user history habit score of each region, a comprehensive score for each region is determined. Based on the comprehensive score of each region and the remaining capacity, the target placement area is determined.

9. A smart refrigerator, characterized in that, include: The refrigerator includes a processor, a memory, and a bus. The memory stores machine-readable instructions that the processor can execute. When the smart refrigerator is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the food identification method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the food ingredient identification method as described in any one of claims 1 to 8.

Images