Refrigeration equipment and food material identification method

Abstract

The invention belongs to the technical field of refrigeration equipment, and provides refrigeration equipment and a food material recognition method.The refrigeration equipment comprises a box body, the box body is provided with a plurality of storage areas used for containing food; the image acquisition equipment is arranged in at least one storage area; the control device is connected with the image acquisition equipment and is configured to respond to triggering of an image acquisition condition, an image in the storage area is acquired based on the image acquisition equipment, and an object in the image comprises one or more food materials; determining shielded food materials in the image according to food material contours of the food materials in the image; sheltering object removal processing and missing part complementation processing are carried out on sheltered food materials in the image, and a processed image is obtained; and identifying food material information of the shielded food materials in the processed image. The problem that a traditional refrigerator is low in recognition accuracy due to food material shielding can be solved.

Description

Technical Field

[0001] The present application belongs to the technical field of refrigeration equipment, and more specifically, to a refrigeration equipment and a food identification method. Background Art

[0002] In modern smart home applications, the intelligent management of refrigeration equipment (such as refrigerators) has attracted much attention, among which food image recognition through camera photography is one of the key technologies. However, traditional refrigerators face many problems when adopting this technology.

[0003] Due to the limited internal space of the refrigerator and the random placement of ingredients, ingredients frequently block each other, and key identification elements of ingredients are missing or incomplete, resulting in low recognition accuracy, which seriously affects the intelligence level of the refrigerator and is unable to provide users with accurate ingredient information. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a refrigeration device and a food identification method, aiming to solve the technical problem of low recognition accuracy in traditional refrigerators due to food obstruction.

[0005] To achieve the above object, according to a first aspect of the present application, a refrigeration device is provided, comprising:

[0006] A box body, wherein the box body is provided with a plurality of storage areas for accommodating food;

[0007] An image acquisition device disposed in at least one of the storage areas;

[0008] And, a control device, connected to the image acquisition device, is configured to:

[0009] In response to the triggering of the image acquisition condition, an image in the storage area is acquired based on the image acquisition device, and the objects in the image include: one or more food ingredients;

[0010] Determining the obscured food in the image according to the food outline of the food in the image;

[0011] Performing an obstruction removal process and a missing part completion process on the obstructed food in the image to obtain a processed image;

[0012] Identify the food information of the obscured food in the processed image.

[0013] According to a second aspect of the present application, a method for identifying food materials is provided, the method comprising:

[0014] In response to the triggering of the image acquisition condition, an image in the storage area is acquired based on an image acquisition device, and the objects in the image include: one or more food ingredients;

[0015] Determining the obscured food in the image according to the food outline of the food in the image;

[0016] Performing an obstruction removal process and a missing part completion process on the obstructed food in the image to obtain a processed image;

[0017] Identify the food information of the obscured food in the processed image.

[0018] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0019] According to a third aspect of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements any of the methods described in one embodiment.

[0020] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

[0021] According to a fifth aspect of the present application, a computer program product is provided. When the computer program product is run on an electronic device, the electronic device executes any one of the methods described in the first aspect.

[0022] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0023] The embodiment of the present application provides a refrigeration device and a food identification method, the refrigeration device comprising: a box body, the box body being provided with a plurality of storage areas for accommodating food; an image acquisition device arranged in at least one storage area; and a control device, connected to the image acquisition device and configured to: in response to the triggering of an image acquisition condition, acquire an image in the storage area based on the image acquisition device, wherein the objects in the image include: one or more food ingredients; determine the obscured food ingredients in the image according to the food ingredient contours in the image; perform an obstruction removal process and a missing part completion process on the obscured food ingredients in the image to obtain a processed image; since the obstructions in the processed image have been eliminated, the feature information of the previously obscured food ingredients becomes clearer and more complete, and then the food information of the obscured food ingredients in the processed image is identified, the accuracy of the obscured food ingredients can be significantly improved, so that the refrigeration device can more accurately identify the type, quantity, state and other information of the food ingredients, thereby effectively solving the problem of low recognition accuracy due to food obstruction in traditional refrigerators. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 It is a structural schematic diagram of a refrigeration device provided in an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of an optional image after preliminary segmentation of food provided in an embodiment of the present application;

[0027] Figure 3 is a schematic diagram of an optional calibrated image provided in an embodiment of the present application;

[0028] Figure 4a and Figure 4b They are schematic diagrams of an optional process of removing an obstruction from an image provided by an embodiment of the present application;

[0029] Figure 5 is a schematic diagram of an optional processed image provided in an embodiment of the present application;

[0030] Figure 6 is a schematic diagram of an optional real food image provided by an embodiment of the present application;

[0031] Figure 7 is a schematic flow chart of a food identification method provided in another embodiment of the present application;

[0032] Figure 8 is a schematic flow chart of an optional food identification method provided in another embodiment of the present application;

[0033] Fig. 9 is a schematic flow chart of an optional food identification method provided in another embodiment of the present application;

[0034] Fig.10 is a schematic flow chart of an optional food identification method provided in another embodiment of the present application; Fig.11 is a schematic flow chart of an optional food identification method provided in another embodiment of the present application; Fig.12 is a structural schematic diagram of a food identification device provided in an embodiment of the present application; Fig.13 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0036] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0037] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0038] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0039] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0040] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0041] This application example provides an example of a refrigeration device, please refer to Figure 1 As shown, Figure 1 The schematic diagram of the structure of a refrigeration device provided by the example of the present application is shown. As an example but not a limitation, the method can be applied to or run in the refrigeration device. The refrigeration device includes:

[0042] The box body 100 is provided with a plurality of storage areas 101 ( Figure 1 A storage area 101 is schematically drawn in FIG.

[0043] An image acquisition device 102 disposed in at least one storage area; and

[0044] The control device 103 is connected to the image acquisition device 102 and is configured to:

[0045] In response to the triggering of image acquisition conditions, an image in a storage area is acquired based on an image acquisition device, and the objects in the image include: one or more ingredients; based on the outlines of the ingredients in the image, the obscured ingredients in the image are determined; the obscured ingredients in the image are processed for removing obstructions and completing missing parts to obtain a processed image; and the ingredient information of the obscured ingredients in the processed image is identified.

[0046] Optionally, in the example of the present application, the refrigeration equipment can be any type of refrigerator, for example, a direct cooling refrigerator, an air cooling refrigerator, a variable frequency refrigerator, etc.

[0047] Optionally, in the example of the present application, the refrigeration equipment has multiple independent storage areas (for example, a refrigerator, a freezer, a temperature-changing room, etc.), each storage area is provided with a corresponding door body, and the temperature and humidity of different storage areas can be finely adjusted according to the storage requirements of different ingredients.

[0048] In the example of the present application, the box is the main structure of the refrigeration equipment, which is provided with a plurality of storage areas for storing food. The image acquisition device is installed in at least one storage area and is used to obtain image information in at least one storage area. When the image acquisition conditions are met (such as the door of the storage area is closed, a specific time interval or a manual trigger by the user, etc.), the control device sends an acquisition instruction to the image acquisition device to control the image acquisition device to acquire images of food in the storage area and transmit the image data to the control device.

[0049] Taking the above-mentioned image acquisition device as a camera as an example, the camera can be installed in the top corner of at least one refrigerated area to cover a larger field of view to ensure that most of the food placement area can be photographed; in addition, for the drawer-type storage area, a small camera can be installed on the side of the drawer-type storage area to photograph the food in the drawer-type storage area.

[0050] In this application example, after the control device receives the image, before determining the obscured food in the image based on the food outline in the image, the food in the image can be preliminarily segmented to obtain the following: Figure 2 In the image shown, ingredient A is the blocking ingredient and ingredient B is the blocked ingredient.

[0051] In order to avoid the inability to accurately identify the food information of food A after the blocked food A is removed from the image, in the example of this application, the control device analyzes the outline of the food in the image. If food A is determined to be a complete food based on the food outline of food A, food A in the image is directly identified first to obtain the food information of image A, for example, the type of food A is chili, the quantity is one, the color is green, etc. It should be understood that the complete food outline usually has continuity and specific geometric shape characteristics.

[0052] If the control device determines that the outline of ingredient B is discontinuous, partially missing, or abnormally intersecting with the outline of other ingredients by analyzing the outline of the ingredients in the image, it can be preliminarily inferred that ingredient B is likely to be blocked. As shown in Figure 2, the outline of ingredient B (eggplant) which should be a long strip is suddenly cut off, and the outline of another ingredient A (similar to a round pepper) that is similar to a circle appears at the cutoff point. In this case, it can be reasonably inferred that ingredient B is blocked by ingredient A.

[0053] For the obscured food in the image, in this application example, deep learning models, such as artificial intelligence (AI) models such as generative adversarial networks (GANs) and autoencoders, can be used to remove obstructions and complete missing parts to obtain processed images. These models are trained with a large amount of training data and can grasp key information such as the appearance characteristics and spatial relationships of food.

[0054] When removing occluders, the deep learning model uses the knowledge learned during training to identify the occluded parts of the image and remove them from the image. When completing the occluded ingredients, the deep learning model will refer to various data of the surrounding ingredients, including color, texture, shape and other information, to infer the missing parts of the occluded ingredients, and then complete the completion, so that the occluded ingredients are restored to their complete state in the image as much as possible. For example, if a piece of vegetable is occluded and there are similar types of vegetables around it, the appearance of the missing part of the occluded vegetable can be inferred based on the characteristics of the surrounding vegetables, so that targeted completion can be performed.

[0055] When removing occluders, the deep learning model is used to identify the occluded part of the image through the knowledge learned through training, and the occluded part is removed from the image. When completing the occluded food, the deep learning model is used to refer to various data of the surrounding food, such as color, texture, shape and other information, to infer the missing part of the occluded food, so as to complete the occluded food, so as to restore the occluded food to a complete state in the image as much as possible. For example, if the occluded object is a fruit and there are similar kinds of fruits around it, the missing part of the occluded fruit can be inferred based on the characteristics of these surrounding fruits, so as to complete it in a targeted manner.

[0056] The image after the occlusion removal and food completion processing contains the completed occluded food. At this time, since the occlusion in the processed image has been eliminated, the feature information of the previously occluded food becomes clearer and more complete. The control device uses the image recognition algorithm to identify the food information of the occluded food in the processed image, and obtains the food information, which can significantly improve the accuracy of identifying the occluded food, so that the refrigeration equipment can more accurately identify the type, quantity, status and other information of the food, thereby effectively solving the problem of low recognition accuracy of traditional refrigerators due to food occlusion.

[0057] In a possible implementation, the image acquisition condition includes at least one of the following:

[0058] Detecting a change in position of one or more food items in the storage area;

[0059] One or more new ingredients are detected in the storage area;

[0060] It is detected that one or more food items in the storage area are taken out;

[0061] It is detected that the door of the storage area has closed.

[0062] In the storage area of ​​refrigeration equipment (such as refrigerators), sensors can be installed to monitor the position status of food. For example, the sensor uses infrared sensing, ultrasonic ranging, etc. to sense the change of reflection signals of food at different positions in the storage area, so as to determine whether the position of the food has changed.

[0063] When the position change of food is detected, image acquisition is triggered, so that the refrigeration equipment can update the layout of food in the storage area in time. Since the change of food position means that the user has sorted and rearranged the food, by collecting new images, the current placement of food can be accurately grasped, so as to carry out accurate food management in the future, such as judging the freshness changes of food in different positions and planning more reasonable storage space utilization.

[0064] Similarly, sensors can also be used to detect the addition or removal of one or more ingredients in the storage area. For example, a weight sensor is set near the entrance of the storage area or at a specific location. When new ingredients are put in, the weight changes significantly, thereby determining whether new ingredients have been added to the storage area. Alternatively, image recognition technology can be used to compare the previous and next images (the image collected last and the image currently collected in real time). If a new object outline appears and meets the characteristics of the ingredients, it can also be determined that new ingredients have been added to the storage area.

[0065] Capturing images after detecting that an ingredient has been added or removed helps to update the ingredient inventory information in a timely manner. Capturing images after detecting that an ingredient has been added allows the refrigeration equipment to identify the added ingredient information as quickly as possible, determine the type and quantity of the added ingredient, and then update the ingredient list to provide users with an accurate ingredient storage status report. Capturing images after detecting that an ingredient has been removed is of great significance for rationally arranging subsequent procurement plans and avoiding food waste. It also allows users to more clearly understand the real-time dynamics of the ingredients in the refrigerator.

[0066] In addition, by installing a door magnetic sensor on the door of the storage area, when the door of the storage area is closed, the state of the door magnetic sensor will change, so that it can be detected that the door has performed a closing action. The reason why the door is closed is selected as one of the image acquisition conditions is that when the user opens the door of the storage area, he or she will generally put in, take out or sort the food, and when the door is closed, these operations are basically completed. At this time, the image acquisition can obtain a relatively stable and complete image of the food status, avoiding the acquisition of inaccurate image information due to the continuous changes of the food during the door opening process, so as to better perform subsequent food identification, management and other operations.

[0067] Through the above-mentioned optional implementation method, by collecting images in the storage area under the triggering of any image acquisition condition, it is possible to trigger the image acquisition device to shoot at an appropriate time, thereby obtaining accurate image information, so as to achieve effective management and precise monitoring of the food in the refrigeration equipment.

[0068] In a possible implementation, the control device determines the obscured food in the image according to the food outline of the food in the image, and is configured as follows:

[0069] An image recognition algorithm is used to perform integrity analysis on the contours of the ingredients in the image. The integrity analysis is used to analyze whether the contours of the ingredients are discontinuous or have gaps. If the integrity analysis result indicates that a first target ingredient exists in the image, it is determined that the first target ingredient is an occluded ingredient, and the contours of the first target ingredient are discontinuous or have gaps.

[0070] In the example of this application, an image recognition algorithm can be used to extract the outline of the food in the image, and then perform integrity analysis on the outline of the food in the image. For example, the image recognition algorithm can be based on edge detection technology, and the edge of the food can be determined by calculating the gradient change of the pixel points in the image, and then the outline of the food can be outlined.

[0071] Ideally, the outline of a complete ingredient should be a continuous and closed curve. For example, the outline of a complete round ingredient (such as an apple) should be a smooth, uninterrupted circular curve in the image; the outline of a square packaged ingredient should be a closed figure composed of four continuous edges. If the outline of an ingredient is discontinuous, there is a break in the normal outline line, or there is a gap, it means that the closedness of the ingredient outline is destroyed.

[0072] When an ingredient is blocked by other ingredients or objects, the blocked portion of the ingredient's outline cannot be displayed normally, resulting in interrupted or missing outlines. Therefore, when the result of the integrity analysis shows that the first target ingredient (an ingredient with a discontinuous outline or a gap) exists in the image, the first target ingredient is determined as the blocked ingredient. This implementation method analyzes the outline features of the ingredient based on an image recognition algorithm, and infers whether the ingredient is blocked by identifying abnormalities in the outline, thereby improving the accuracy of identifying the ingredient.

[0073] For example, if a part of the outline of a vegetable that should be long (such as a long eggplant, cucumber, etc.) is suddenly cut off, and the outline of another vegetable similar to a round shape (such as cabbage, tomato, etc.) appears at the cutoff point, it can be reasonably inferred that the long vegetable is blocked by the round vegetable. For another example, if a carrot-shaped food outline is found in the image, but other food (such as an apple, a milk bottle, etc.) blocks the middle part of the "carrot", resulting in a discontinuous outline of the "carrot", the "carrot" is determined to be the first target food, that is, the blocked food.

[0074] In one possible implementation, the object in the image includes multiple ingredients; the control device determines the obscured ingredients in the image based on the ingredient outlines of the multiple ingredients, and is configured to: determine the overlapping portion between the ingredient outlines of the multiple ingredients; determine at least two ingredients that obscure each other among the multiple ingredients based on the overlapping portion; determine a second target ingredient among the at least two ingredients that obscure each other as the obscured ingredient, and the ingredient outline of the second target ingredient is discontinuous or has a gap.

[0075] In this implementation, if the object covered by the image includes multiple ingredients, that is, the content in the image is relatively complex, multiple ingredients are arranged and stacked in the storage area of ​​the refrigerator, thus forming various possible occlusion relationships. For example, in the cold storage area of ​​the refrigerator, there may be apples, bananas, vegetables and other ingredients at the same time, and the placement and method of multiple ingredients are random, which creates conditions for mutual occlusion between ingredients.

[0076] In the example of the present application, the control device determines the overlapping parts between the food outlines of multiple ingredients by analyzing the food outlines of multiple ingredients. For example, the boundary detection and coordinate calculation of each food outline are performed to determine whether the different food outlines intersect in space. For another example, the control device can use an edge detection algorithm (such as the Canny edge detection algorithm) to determine the edge outline of each food, and then calculate the coordinate range and intersection of each food outline to determine whether there is an overlapping part between different food outlines.

[0077] Since an occlusion relationship may exist in reality only when two ingredients are in the overlapping part between the outlines of the ingredients in the image, it should be understood that such an overlap may be a partial overlap, or it may be a situation where one ingredient is mostly occluded by another ingredient.

[0078] After the overlapping parts between the contours of multiple ingredients are determined, these overlapping parts are further analyzed, and factors such as the overall shape, relative position, and degree of overlap of the contours of the ingredients are combined to determine which of the multiple ingredients are mutually obstructed. For example, if a circular contour of an ingredient (such as an orange) and a rectangular contour of an ingredient (such as boxed milk) partially overlap, and a portion of the circular contour of the ingredient cannot be fully displayed, that is, it is determined that a portion of the circular contour of the ingredient is obstructed behind the rectangular contour of the ingredient, then it is determined that there is mutual obstruction between the orange and the boxed milk.

[0079] In another example, for the complex situation where multiple ingredients overlap, you can first analyze each pair of overlapping ingredients to determine all mutually occluded ingredient combinations in the entire image. After determining at least two mutually occluded ingredients, select a second target ingredient (with a discontinuous outline or a gap) from these mutually occluded ingredients as the occluded ingredient. This is because when an ingredient is obscured by another ingredient, its outline will be interrupted or missing at the obscured location. For example, among two mutually occluded fruits, if the outline of one of the fruits has an obvious depression or truncated portion, while the outline of the other fruit is relatively complete, the fruit with the outline problem is determined to be the second target ingredient, i.e., the occluded ingredient.

[0080] Through the above optional method, based on the actual presentation of the ingredients in the image, the obscured ingredients can be accurately found through the analysis of contour features, providing accurate objects for subsequent operations such as occlusion removal and ingredient information recognition, which helps to improve the accuracy and reliability of ingredient recognition.

[0081] In one possible implementation, the control device performs occlusion removal processing and missing part completion processing on the occluded food in the image to obtain a processed image, and is configured to: calibrate the occlusions on the occluded food in the image to obtain a calibrated image; input the calibrated image into a pre-trained occlusion removal model, so as to use the occlusion removal model to perform occlusion removal processing and missing part completion processing on the occluded food in the calibrated image to obtain a processed image; the occlusion removal model is obtained by iteratively training a generative adversarial network model using multiple groups of training samples in advance, and each group of training samples includes: a first sample image with calibrated occlusions, and a second sample image with the missing parts completed corresponding to the first sample image.

[0082] In the above implementation, the control device first calibrates the obstructions on the obstructed food in the image, mainly to clarify which parts of the image are the obstructions that need to be processed, and the positions of the corresponding obstructions, so that the obstruction elimination model knows accurately which areas to process.

[0083] Specifically, in this application example, some image processing techniques and tools can be used to implement the calibration operation. For example, image annotation software can be used or a code can be used in a programming environment to mark a specific area. In this application example, the occluder is calibrated in a visual way, such as using a marker point, a rectangular box, a polygonal box, or a marker of a specific color, so as to obtain a calibrated image. For example Figure 3 As shown, by using the marker points to calibrate the occluder (i.e. corresponding to Figure 2 The occluded food A) in the image is removed to obtain the calibrated image.

[0084] In the example of this application, the occlusion removal model is built based on the generative adversarial network (GAN) model. The GAN model consists of two parts: the generator and the discriminator, which continuously optimizes the model performance through mutual confrontation. Specifically, the task of the generator is to generate new images or data based on the input, so that the image or data is as close to the real data as possible; the discriminator is responsible for judging whether the input data is real or generated by the generator, and through this adversarial mechanism, the generator is prompted to generate better quality images.

[0085] Moreover, in the example of the present application, before the occluder removal model is specifically applied to perform occluder removal processing and missing part completion processing, the initial GAN-based model is iteratively trained through multiple groups of training samples to obtain the occluder removal model. Each group of training samples contains two parts: a first sample image with occluders calibrated, and a second sample image corresponding to the first sample image with the missing part completed. During the training process, the first sample image is input into the initial GAN-based model, and the generator attempts to generate an image with occluders removed and missing parts completed based on the input image with occluders, and then the discriminator judges the image generated by the generator to determine whether it is similar to the corresponding second sample image (i.e., the real, completed image). By continuously iterating this process, the model adjusts the parameters of the generator according to the feedback of the discriminator, so that the image generated by the generator is closer and closer to the real completed image, thereby gradually optimizing the performance of the model so that it can accurately perform occluder removal processing and missing part completion processing on the newly input calibrated image.

[0086] After the calibrated image obtained through occlusion calibration is input into the pre-trained occlusion removal model, the occlusion removal model performs occlusion removal processing and missing part completion processing on the occluded food in the calibrated image according to the knowledge and capabilities learned during the training process. Figure 4a and Figure 4b As shown, the occluders in the calibration area are identified (i.e., corresponding to Figure 2 The occluded food A in the image is removed step by step, and the occluded food (i.e. the corresponding Figure 2The known part of the occluded food B) in the image and the surrounding environment are used to complete the missing part of the occluded food, and finally the following is obtained: Figure 5 The processed image shown ( Figure 6 ) Compared with the original image, the occluded food has its occluders removed and its missing parts completed, making the occluded food more complete in the image, so as to accurately identify and obtain the food information of the occluded food.

[0087] In a possible implementation, the control device performs an occlusion removal process and a missing portion completion process on the occluded food in the image to obtain a processed image, and is configured as follows:

[0088] The occluded food in the image is calibrated for occlusion to obtain a calibrated image; based on the image and / or the model of the refrigeration equipment, the liner color of at least one storage area is determined; the liner color and the calibrated image are input into a pre-trained occlusion elimination model, so that the occlusion elimination model is used to perform occlusion removal processing and missing part completion processing on the occluded food in the calibrated image based on the liner color to obtain a processed image; the occlusion elimination model is obtained by iteratively training a generative adversarial network model using multiple groups of training samples in advance, and each group of training samples includes: a first sample image with calibrated occlusions, a sample liner color as a constraint condition, and a second sample image corresponding to the first sample image with the missing part completed.

[0089] Since the inner tank of current refrigeration equipment is usually white or other dark colors, the inner tank color is consistent, and the model will be disturbed by the background color when completing the image, resulting in large differences in image completion. Therefore, in the example of this application, in the above implementation, the control device first calibrates the obstruction of the obstructed food to obtain a calibrated image, and then determines the inner tank color of the storage area in combination with the image and / or the model of the refrigeration equipment (such as a refrigerator), and finally inputs the inner tank color and the calibrated image into the obstruction elimination model for processing, thereby obtaining a processed image.

[0090] Specifically, first, the control device calibrates the occluded food in the image, mainly to clarify which parts of the image are occluded objects that need to be processed, and the positions of the corresponding occluded food, so that the occluder elimination model knows accurately which areas to process.

[0091] Afterwards, the control device determines the liner color of at least one storage area based on the image and / or the model of the refrigerator. First, it can be determined by analyzing the color distribution characteristics of the liner area in the image. For example, a color clustering analysis is performed on the background part of the image to find the dominant color as a candidate for the liner color. Second, since the liner colors of refrigerators of different models are usually known or can be obtained by querying relevant information, the liner color of at least one storage area can be determined based on the model information of the refrigeration equipment. The above two methods can complement each other and improve the accuracy of determining the liner color.

[0092] In the example of this application, the occlusion removal model is built based on the generative adversarial network (GAN) model. The GAN model consists of two parts: the generator and the discriminator, which continuously optimizes the model performance through mutual confrontation. Specifically, the task of the generator is to generate new images or data based on the input, making it as close to the real data as possible; the discriminator is responsible for judging whether the input data is real or generated by the generator, and through this adversarial mechanism, the generator is prompted to generate better quality images.

[0093] Before the occluder removal model is specifically applied to perform occluder removal processing and missing part completion processing, the initial GAN-based model is iteratively trained through multiple groups of training samples to obtain the occluder removal model. Each group of training samples includes: a first sample image with occluders calibrated, a sample liner color as a constraint condition, and a second sample image corresponding to the first sample image after the missing part is completed. During the training process, the model learns to generate a result similar to the second sample image based on the given first sample image and liner color. For example, when a first sample image with an occluder and a known liner color is input, the model attempts to generate an image with the occluder removed and the missing part of the occluded food completed, while ensuring that the fusion of the generated image and the liner color is natural and reasonable, and closer to the real second sample image. Through continuous iterative training with a large number of training samples, the model can gradually master how to accurately remove occluders and complete missing parts for different calibrated images according to the liner color, thereby obtaining a processed image and improving the accuracy of food recognition.

[0094] After the liner color and the calibrated image are input into the trained occlusion removal model, the occlusion removal model removes occlusions and completes missing parts of the occluded food based on the patterns and rules learned during the training process. For example, when removing occlusions, the occlusion removal model accurately identifies the occlusion part based on the liner color and other information in the image and removes it. At the same time, when completing the missing part of the occluded food, the liner color is considered to avoid unreasonable completion results, so that the completed image is naturally integrated with the surrounding environment (including the liner color), and finally a processed image is obtained, providing clearer and more accurate image data for subsequent food identification.

[0095] In a possible implementation, the refrigeration device further includes: a lamp assembly disposed in at least one storage area, connected to the control device, and configured to: in response to the triggering of the image acquisition condition, turn on the lighting mode under the drive of the control device. The image acquisition device is configured to: while the lamp assembly is running the lighting mode, under the control of the control device, take a picture of the storage area to obtain an image.

[0096] In the above optional implementation, a lamp assembly is also provided in at least one storage area of ​​the refrigeration equipment. The main function of these lamp assemblies is to provide lighting for the storage area so that the image acquisition device can clearly capture the food in the storage area. For example, in the refrigerator compartment and freezer compartment of the refrigerator, the lamp assembly can be installed at the top, side, etc. to ensure that the light can evenly cover the entire storage space, reduce the shadow area, and enable the image acquisition device to obtain high-quality images.

[0097] When the image acquisition condition is triggered, for example, the door of the refrigerator compartment is detected to be closed, which means that the user has completed the placement or removal of food, the control device drives the lamp assembly to turn on the lighting mode. In the lighting mode, the lamp assembly provides sufficient light for the storage area to ensure the brightness of the shooting environment. This is similar to the flash of a camera automatically turning on when shooting to provide sufficient light for the shooting scene.

[0098] At the same time, since the lamp tube assembly has turned on the lighting mode, the storage area has suitable lighting conditions. Under the control of the control device, the image acquisition device starts to shoot the storage area and can capture clear images. These images contain information about one or more ingredients in the storage area.

[0099] Through the above implementation, the lamp tube assembly and the image acquisition device work in coordination with the control device, ensuring the normal operation of the entire refrigeration equipment image acquisition system, ensuring the quality of the acquired images, and improving the accuracy and reliability of food identification.

[0100] In a possible implementation, the control device is further configured to adjust the lighting parameters of the lamp assembly during the process of controlling the image acquisition device to photograph the storage area.

[0101] Optionally, in the example of this application, the above lighting parameters include: brightness, color temperature, color, etc.

[0102] It should be understood that the brightness parameter directly affects the brightness of the image, and the appropriate brightness can make the details of the food clearly visible. For example, for foods with lighter colors or smooth surfaces, appropriately reducing the brightness can avoid reflection problems caused by excessive brightness, thereby better presenting the texture of the food. The color temperature parameter is related to the color temperature of the light, and different color temperatures will bring different tonal effects to the image. For example, a higher color temperature (cooler colors) may be more suitable for highlighting the freshness of the food, while a lower color temperature (warmer colors) can create a warmer visual effect, but the accurate presentation of the color of the food may need to be adjusted according to actual conditions. The color parameters involve the realization of special lighting effects. For example, in some refrigerators with special functions, the color of the light may be adjusted to assist in identifying specific types of food or detecting changes in the freshness of the food.

[0103] In addition, it is still necessary to explain that the basis for the control device to adjust the lighting parameters during the shooting process can be a variety of factors. On the one hand, it can be adjusted based on the preset program and algorithm according to the characteristics of different storage areas, types of ingredients or shooting time. For example, for the storage area where vegetables and fruits are placed, since the colors of such ingredients are brighter, higher brightness and appropriate color temperature can be used to highlight the colors; and for the meat storage area, in order to better observe the texture and color of the meat, the light angle and brightness can be adjusted. On the other hand, the control device can also make dynamic adjustments based on the image quality information fed back in real time by the image acquisition device. For example, if it is found that the captured image is too dark, too bright, or has color distortion, the control device can immediately adjust the lighting parameters of the lamp assembly to optimize the image quality and ensure that the captured image can accurately reflect the true state of the food.

[0104] In the present application example, by adjusting the lighting parameters during the shooting process, the refrigeration equipment can better adapt to different shooting scenes and food types, thereby improving the accuracy and effectiveness of image acquisition and further enhancing the performance of the refrigeration equipment in managing food.

[0105] In a possible implementation, the control device adjusts the lighting parameters of the lamp assembly during the process of controlling the image acquisition device to photograph the storage area, and is further configured as follows:

[0106] In the process of controlling the image acquisition device to shoot the storage area, characteristic information of the food in the storage area and internal environment information of the storage area are obtained; and the lighting parameters of the lamp assembly are adjusted according to the characteristic information and the internal environment information.

[0107] Optionally, the characteristic information is used to describe at least one of the following appearance characteristics of the food: color, transparency, glossiness, and the internal environment information is used to describe at least one of the following information of the storage area: liner color, material, and spatial layout.

[0108] It should be understood that different types of food often have specific color ranges. For example, apples may appear in different shades such as red and green, and tomatoes are mostly red. By accurately obtaining the color information of food, it can provide a basis for adjusting the lighting parameters according to the characteristics of the food, so as to better highlight the true color of the food and avoid color distortion caused by improper lighting that affects food identification.

[0109] It should be understood that some foods have certain transparency characteristics, such as some foods packaged in plastic products, drinks in glass bottles, or foods that are relatively transparent in nature (such as some jelly foods). Understanding the transparency of the food will help determine the appropriate lighting method.

[0110] It should be understood that the glossiness of food surfaces is also different. Some food surfaces are smooth and have a strong gloss (such as waxed fruits), while others are relatively dull (such as some untreated root vegetables). Based on the glossiness information, the lighting parameters can be adjusted to balance the glossiness effect, neither too prominent to cause reflections that affect image details, nor too dull to make the food features unclear.

[0111] It should be understood that the color of the inner container of the refrigerator storage area will have a certain degree of impact on the captured image. For example, the white inner container makes the overall image brighter under the reflection of light, while the dark inner container may absorb more light and cause local darkness. The lighting parameters are adjusted based on the color of the inner container to compensate or balance the light effect caused by the color of the inner container, so that the captured image can more accurately reflect the state of the food.

[0112] It should be understood that different materials have different optical properties such as reflection and absorption of light. For example, the surface of a metal material may produce a strong specular reflection, while a plastic material may be relatively soft. Based on the material information, parameters such as lighting angle and brightness can be reasonably adjusted to reduce image interference caused by material reflection.

[0113] It should still be understood that the spatial layout of the storage area affects the propagation path and distribution of light. If the storage area is narrow and the items are densely placed, the propagation of light may be more obstructed and shadows are easily created; while in a spacious space with sparsely placed items, the light distribution is relatively uniform. By understanding the spatial layout information, the lighting parameters can be adjusted to optimize the light distribution, ensuring that each area can receive appropriate lighting so that the food can be clearly photographed.

[0114] As an optional example, if the color of the food is bright, such as red tomatoes, in order to highlight the brightness of its color, the brightness of the lighting can be appropriately increased and the appropriate color temperature can be selected (for example, a slightly higher color temperature makes the light more cool, which can make the red more vivid). On the contrary, for lighter-colored food, such as white tofu, it may be necessary to reduce the brightness to avoid being too bright and making the tofu look too pale, and at the same time select the appropriate color temperature to maintain its original color.

[0115] Another optional example is that for highly transparent food materials, such as beverages in glass bottles, it is important to avoid direct strong light exposure. The brightness can be appropriately reduced and the light angle can be adjusted to reduce refraction and reflection, so that the liquid and label in the beverage bottle can be clearly presented. For opaque food materials, the lighting parameters can be adjusted comprehensively according to other characteristics such as their color and glossiness.

[0116] Another optional example is that for food with high surface gloss, such as waxed fruit, to prevent excessively bright light from causing reflections and affecting image details, the brightness can be appropriately reduced and the light angle can be adjusted to appropriately present the surface gloss without affecting the observation of other features of the food. For food with low gloss, the brightness can be appropriately increased to highlight its overall features.

[0117] Another optional example is that if the inner pot is white, the image may be brighter overall. In this case, the lighting brightness can be appropriately reduced, or the color temperature can be adjusted to make it more natural. If the inner pot is dark, the brightness may need to be increased to compensate for the dimness caused by light absorption, and the color temperature should be adjusted to ensure that the color of the food is not distorted.

[0118] Another optional example is that if the storage area is made of metal, in order to avoid the image quality being affected by specular reflection, the light angle can be adjusted to avoid vertical incidence as much as possible, and the brightness can be appropriately reduced. For storage areas made of plastic, the lighting parameters can be adjusted comprehensively according to the specific type and color of the plastic. Generally speaking, the brightness can be appropriately increased to compensate for the possible light absorption of the plastic.

[0119] Another optional example is that in a storage area with a narrow space layout and densely placed items, in order to eliminate shadows, the lighting brightness can be increased, and the light angle can be adjusted so that the light can better penetrate the gaps between items and evenly illuminate each area. In a relatively spacious storage area with sparsely placed items, the lighting parameters can be reasonably adjusted according to other factors such as the color of the ingredients and the liner to maintain uniform light distribution.

[0120] Through the above optional examples, by finely adjusting the lighting parameters of the lamp assembly according to the characteristic information of the food and the internal environment information of the storage area, higher quality images can be obtained during the shooting process, thereby providing a more accurate and reliable data basis for subsequent food identification, processing and food management functions of the entire refrigeration equipment.

[0121] In a possible implementation, the control device adjusts the lighting parameters of the lamp assembly during the process of controlling the image acquisition device to photograph the storage area, and is further configured as follows:

[0122] In the process of controlling the image acquisition device to take pictures of the storage area, characteristic information of the food in the storage area and the shooting information of the image acquisition device are obtained; and the lighting parameters of the lamp assembly are adjusted according to the characteristic information and the shooting information.

[0123] Optionally, the feature information is used to describe at least one of the following information of the food: color, transparency, and glossiness, and the shooting information is used to describe at least one of the following information of the image acquisition device: shooting angle and shooting distance.

[0124] It should be noted that, for the relevant description on the characteristic information of the ingredients in this implementation, please refer to the description on the characteristic information of the ingredients in the above implementation, which will not be repeated here.

[0125] In this implementation, the control device also obtains shooting information of the image acquisition device, and the shooting information mainly involves relevant parameter settings and states of the image acquisition device during the shooting process, specifically:

[0126] The shooting angle determines the viewing direction of the image acquisition device relative to the food in the storage area. Different shooting angles will change the way the food is presented in the image. For example, shooting vertically from above may highlight the top shape and layout of the food, while shooting from the side may better show the side profile and three-dimensional sense of the food. Understanding the shooting angle information will help adjust the lighting parameters according to the presentation characteristics of the food at different angles to obtain better image effects.

[0127] Shooting distance refers to the distance between the image acquisition device and the food in the storage area. The distance of shooting will affect the magnification and clarity of the image. Close-up shooting can show the detailed features of the food, but the coverage may be small; long-distance shooting can cover a wider area, but the details may not be clear enough. By understanding the shooting distance information, the lighting parameters can be adjusted according to the required level of detail and coverage of the image to ensure that a clear and appropriate image is obtained at the corresponding shooting distance.

[0128] As an optional example, for food with high surface gloss, such as waxed fruit, in order to prevent excessively bright light from causing reflections and affecting the observation of image details, the brightness can be appropriately reduced, and the light angle can be adjusted at the same time, so that the gloss of the fruit surface is appropriately presented without interfering with the observation of other features of the fruit. On the contrary, for food with low gloss, the brightness can be appropriately increased to highlight the overall features.

[0129] As an optional example, when the shooting angle is vertically shot from above, if you need to highlight the top shape and layout of the food, you can adjust the lighting brightness and color temperature appropriately according to the color, glossiness and other characteristics of the food. For example, for brightly colored and high-gloss food, you can lower the brightness and select a slightly higher color temperature to reduce reflections and highlight the top features. When the shooting angle is from the side, in order to better show the side profile and three-dimensional sense of the food, you can adjust the light angle and brightness according to the transparency, glossiness and other characteristics of the food. For example, for food with high transparency, you can adjust the light angle appropriately to make its side more three-dimensional under the light, and adjust the brightness according to its glossiness to avoid reflections affecting the outline.

[0130] As another optional example, if the shooting distance is close, the purpose is to present the detailed features of the food. At this time, the lighting brightness and color temperature can be adjusted appropriately according to the color, glossiness and other characteristics of the food. For example, for food with lighter color and lower glossiness, the brightness can be appropriately increased to ensure that its details can be clearly seen when shooting at a close distance. If the shooting distance is far, if you need to cover a wider area, you can appropriately increase the lighting brightness according to the overall color, transparency and other characteristics of the food to ensure that the entire area can get enough light when shooting at a long distance, so that the image can be clearly seen.

[0131] Through the above-mentioned optional implementation method, the lighting parameters of the lamp assembly are finely adjusted according to the characteristic information of the food and the shooting information of the image acquisition device, so that higher quality images can be obtained during the shooting process, thereby providing more accurate and reliable images for subsequent food identification, processing and food management functions of the entire refrigeration equipment.

[0132] In a possible implementation, the control device adjusts the lighting parameters of the lamp assembly during the process of controlling the image acquisition device to photograph the storage area, and is further configured as follows:

[0133] In the process of controlling the image acquisition device to shoot the storage area, characteristic information of the food in the storage area, shooting information of the image acquisition device and internal environment information of the storage area are obtained; and the lighting parameters of the lamp tube assembly are adjusted according to the characteristic information, shooting information and internal environment information.

[0134] Optionally, in the example of the present application, the feature information is used to describe at least one of the following information of the food: color, transparency, and glossiness; the shooting information is used to describe at least one of the following information of the image acquisition device: shooting angle, shooting distance; and the internal environment information is used to describe at least one of the following information of the storage area: inner liner color, material, and spatial layout.

[0135] It should be noted that for the relevant descriptions on the characteristic information, shooting information, and internal environment information of the ingredients in this implementation method, please refer to the descriptions on the characteristic information, shooting information, and internal environment information of the ingredients in the above implementation method, and no further details will be given here.

[0136] In this implementation, the control device, while controlling the image acquisition device to take pictures of the storage area, obtains the characteristic information of the food in the storage area, the shooting information of the image acquisition device and the internal environment information of the storage area. Based on the characteristic information of the food, the shooting information of the image acquisition device and the internal environment information of the storage area, the lighting parameters of the lamp tube assembly are finely adjusted, so that higher quality images can be obtained during the shooting process, thereby providing more accurate and reliable images for subsequent food identification, processing and food management functions of the entire refrigeration equipment.

[0137] This application example provides an example of a food identification method, please refer to Figure 7 As shown, Figure 7 The following is a schematic flow chart of a food identification method provided by the present application. As an example but not a limitation, the method can be applied to or run in a refrigeration device. The method includes:

[0138] S101, in response to the triggering of an image acquisition condition, an image in a storage area is acquired based on an image acquisition device, and objects in the image include: one or more food ingredients.

[0139] S102: Determine the obscured food in the image according to the food outlines in the image.

[0140] S103, performing an obstruction removal process and a missing part completion process on the obstructed food in the image to obtain a processed image.

[0141] S104, identifying the food information of the blocked food in the processed image.

[0142] In the example of the present application, a plurality of storage areas are provided in the box of the refrigeration equipment for storing food. An image acquisition device is installed in at least one storage area and is used to obtain image information in at least one storage area. When the image acquisition conditions are met (such as the door of the storage area is closed, a specific time interval or a manual trigger by the user, etc.), the control device (control unit, controller) sends an acquisition instruction to the image acquisition device to control the image acquisition device to acquire images of food in the storage area and transmit the image data to the control device.

[0143] Taking the above-mentioned image acquisition device as a camera as an example, the camera can be installed in the top corner of at least one refrigerated area to cover a larger field of view to ensure that most of the food placement area can be photographed; in addition, for the drawer-type storage area, a small camera can be installed on the side of the drawer-type storage area to photograph the food in the drawer-type storage area.

[0144] In the present application example, after the control device receives the image, before determining the blocked food in the image according to the food outline of the food in the image, the food in the image can be preliminarily segmented to obtain the food still as shown in FIG. Figure 2 In the image shown, ingredient A is the blocking ingredient and ingredient B is the blocked ingredient.

[0145] If the control device determines that the outline of ingredient B is discontinuous, partially missing, or abnormally intersecting with the outline of other ingredients by analyzing the outline of the ingredients in the image, it can be preliminarily inferred that the ingredient B may be blocked. As shown in Figure 2, the outline of a vegetable B (eggplant as shown in the figure) that should be long and narrow is suddenly cut off, and the outline of another round vegetable A (round pepper as shown in the figure) appears at the cutoff point. In this case, it can be reasonably inferred that the vegetable B is blocked by the vegetable A.

[0146] For the obscured food in the image, in this application example, deep learning models, such as artificial intelligence (AI) models such as generative adversarial networks (GANs) and autoencoders, can be used to remove obstructions and complete missing parts to obtain processed images. These models are trained with a large amount of training data and can grasp key information such as the appearance characteristics and spatial relationships of food.

[0147] When removing occluders, the deep learning model uses the knowledge learned during training to identify the occluded parts of the image and remove them from the image. When completing the occluded ingredients, the deep learning model will refer to various data of the surrounding ingredients, including color, texture, shape and other information, to infer the missing parts of the occluded ingredients, and then complete the completion, so that the occluded ingredients are restored to their complete state in the image as much as possible. For example, if a piece of vegetable is occluded and there are similar types of vegetables around it, the appearance of the missing part of the occluded vegetable can be inferred based on the characteristics of the surrounding vegetables, so that targeted completion can be performed.

[0148] When removing occluders, the deep learning model is used to identify the occluded part of the image through the knowledge learned through training, and the occluded part is removed from the image. When completing the occluded food, the deep learning model is used to refer to various data of the surrounding food, such as color, texture, shape and other information, to infer the missing part of the occluded food, so as to complete the occluded food, so as to restore the occluded food to a complete state in the image as much as possible. For example, if the occluded object is a fruit and there are similar kinds of fruits around it, the missing part of the occluded fruit can be inferred based on the characteristics of these surrounding fruits, so as to complete it in a targeted manner.

[0149] The image after the occlusion removal and food completion processing contains the completed occluded food. At this time, since the occlusion in the processed image has been eliminated, the feature information of the previously occluded food becomes clearer and more complete. The control device uses the image recognition algorithm to identify the food information of the occluded food in the processed image, and obtains the food information, which can significantly improve the accuracy of identifying the occluded food, so that the refrigeration equipment can more accurately identify the type, quantity, status and other information of the food, thereby effectively solving the problem of low recognition accuracy of traditional refrigerators due to food occlusion.

[0150] In a possible implementation, the image acquisition condition includes at least one of the following:

[0151] Detecting a change in position of one or more food items in the storage area;

[0152] One or more new ingredients are detected in the storage area;

[0153] It is detected that one or more food items in the storage area are taken out;

[0154] It is detected that the door of the storage area has closed.

[0155] In the storage area of ​​refrigeration equipment (such as refrigerators), sensors can be installed to monitor the position status of food. For example, the sensor uses infrared sensing, ultrasonic ranging, etc. to sense the change of reflection signals of food at different positions in the storage area, so as to determine whether the position of the food has changed.

[0156] When the position change of food is detected, image acquisition is triggered, so that the refrigeration equipment can update the layout of food in the storage area in time. Since the change of food position means that the user has sorted and rearranged the food, by collecting new images, the current placement of food can be accurately grasped, so as to carry out accurate food management in the future, such as judging the freshness changes of food in different positions and planning more reasonable storage space utilization.

[0157] Similarly, sensors can also be used to detect the addition or removal of one or more ingredients in the storage area. For example, a weight sensor is set near the entrance of the storage area or at a specific location. When new ingredients are put in, the weight changes significantly, thereby determining whether new ingredients have been added to the storage area. Alternatively, image recognition technology can be used to compare the previous and next images (the image collected last and the image currently collected in real time). If a new object outline appears and meets the characteristics of the ingredients, it can also be determined that new ingredients have been added to the storage area.

[0158] Capturing images after detecting that an ingredient has been added or removed helps to update the ingredient inventory information in a timely manner. Capturing images after detecting that an ingredient has been added allows the refrigeration equipment to identify the added ingredient information as quickly as possible, determine the type and quantity of the added ingredient, and then update the ingredient list to provide users with an accurate ingredient storage status report. Capturing images after detecting that an ingredient has been removed is of great significance for rationally arranging subsequent procurement plans and avoiding food waste. It also allows users to more clearly understand the real-time dynamics of the ingredients in the refrigerator.

[0159] In addition, by installing a door magnetic sensor on the door of the storage area, when the door of the storage area is closed, the state of the door magnetic sensor will change, so that it can be detected that the door has performed a closing action. The reason why the door is closed is selected as one of the image acquisition conditions is that when the user opens the door of the storage area, he or she will generally put in, take out or sort the food, and when the door is closed, these operations are basically completed. At this time, the image acquisition can obtain a relatively stable and complete image of the food status, avoiding the acquisition of inaccurate image information due to the continuous changes of the food during the door opening process, so as to better perform subsequent food identification, management and other operations.

[0160] Through the above-mentioned optional implementation method, by collecting images in the storage area under the triggering of any image acquisition condition, it is possible to trigger the image acquisition device to shoot at an appropriate time, thereby obtaining accurate image information, so as to achieve effective management and precise monitoring of the food in the refrigeration equipment.

[0161] For a possible implementation, please refer to Figure 8 As shown, Figure 8 A schematic flow chart of an optional food identification method provided by the present application is shown, in response to the triggering of the image acquisition condition, an image in the storage area is acquired based on the image acquisition device, including:

[0162] S201, in response to the triggering of the image acquisition condition, driving the lamp assembly in the storage area to turn on the lighting mode;

[0163] S202, while the lamp tube assembly is operating in a lighting mode, controlling an image acquisition device in the storage area to take pictures of the storage area to obtain an image.

[0164] In the above optional implementation, a lamp assembly is also provided in at least one storage area of ​​the refrigeration equipment. The main function of these lamp assemblies is to provide lighting for the storage area so that the image acquisition device can clearly capture the food in the storage area. For example, in the refrigerator compartment and freezer compartment of the refrigerator, the lamp assembly can be installed at the top, side, etc. to ensure that the light can evenly cover the entire storage space, reduce the shadow area, and enable the image acquisition device to obtain high-quality images.

[0165] When the image acquisition condition is triggered, for example, the door of the refrigerator compartment is detected to be closed, which means that the user has completed the placement or removal of food, the control device drives the lamp assembly to turn on the lighting mode. In the lighting mode, the lamp assembly provides sufficient light for the storage area to ensure the brightness of the shooting environment. This is similar to the flash of a camera automatically turning on when shooting to provide sufficient light for the shooting scene.

[0166] At the same time, since the lamp tube assembly has turned on the lighting mode, the storage area has suitable lighting conditions. Under the control of the control device, the image acquisition device starts to shoot the storage area and can capture clear images. These images contain information about one or more ingredients in the storage area.

[0167] Through the above implementation, the lamp tube assembly and the image acquisition device work in coordination with the control device, ensuring the normal operation of the entire refrigeration equipment image acquisition system, ensuring the quality of the acquired images, and improving the accuracy and reliability of food identification.

[0168] In addition, Figure 8 As shown, in another optional example, after acquiring the image in the storage area, the following method is also included:

[0169] S203, determining the complete food and the obscured food in the image according to the food outlines in the image

[0170] S204, identifying the food information of the finished food in the processed image

[0171] S205, performing a process of removing the obstructions and completing the missing parts on the obstructed food in the image to obtain a processed image

[0172] S206, identifying the food information of the blocked food in the processed image.

[0173] In this application example, before determining the obscured food in the image based on the food outline in the image, the food in the image can be preliminarily segmented to obtain the following: Figure 2 The image shown, in which ingredient A is the blocking ingredient and ingredient B is the blocked ingredient. In order to avoid the inability to accurately identify the ingredient information of ingredient A after the blocking ingredient A is subsequently removed from the image, in the example of the present application, the control device analyzes the outline of the ingredient in the image. If the ingredient A is determined to be a complete ingredient based on the ingredient outline of ingredient A, the ingredient A in the image is directly identified first to obtain the ingredient information of image A, for example, the type of ingredient A is chili, the quantity is one, the color is green, etc. It should be understood that the complete ingredient outline usually has continuity and specific geometric shape characteristics.

[0174] For a possible implementation, please refer to Fig. 9 As shown, Fig. 9 A schematic flow chart of an optional food identification method provided by the present application is shown, and the method determines the obscured food in the image according to the food outline of the food in the image, including:

[0175] S301, using an image recognition algorithm to perform integrity analysis on the contour of the food in the image, where the integrity analysis is used to analyze whether the contour of the food is discontinuous or has gaps;

[0176] S302: If the integrity analysis result indicates that the first target food exists in the image, it is determined that the first target food is an obscured food, and the food outline of the first target food is discontinuous or has a gap.

[0177] In the example of this application, an image recognition algorithm can be used to extract the outline of the food in the image, and then perform integrity analysis on the outline of the food in the image. For example, the image recognition algorithm can be based on edge detection technology, and the edge of the food can be determined by calculating the gradient change of the pixel points in the image, and then the outline of the food can be outlined.

[0178] Ideally, the outline of a complete ingredient should be a continuous and closed curve. For example, the outline of a complete round ingredient (such as an apple) should be a smooth, uninterrupted circular curve in the image; the outline of a square packaged ingredient should be a closed figure composed of four continuous edges. If the outline of an ingredient is discontinuous, there is a break in the normal outline line, or there is a gap, it means that the closedness of the ingredient outline is destroyed.

[0179] When an ingredient is blocked by other ingredients or objects, the blocked part of the ingredient's outline cannot be displayed normally, resulting in an interrupted or missing outline. Therefore, when the result of the integrity analysis shows that there is a first target ingredient in the image (an ingredient with a discontinuous outline or a gap), the first target ingredient is determined as an obscured ingredient. This implementation method analyzes the outline features of the ingredient based on an image recognition algorithm, and infers whether the ingredient is blocked by identifying abnormalities in the outline, thereby improving recognition accuracy.

[0180] For example, if a part of the outline of a vegetable that should be long (such as a long eggplant, cucumber, etc.) is suddenly cut off, and the outline of another vegetable similar to a round shape (such as cabbage, tomato, etc.) appears at the cutoff point, it can be reasonably inferred that the long vegetable is blocked by the round vegetable. For another example, if a carrot-shaped food outline is found in the image, but other food (such as an apple, a milk bottle, etc.) blocks the middle part of the "carrot", resulting in a discontinuous outline of the "carrot", the "carrot" is determined to be the first target food, that is, the blocked food.

[0181] For a possible implementation, please refer to Fig.10 As shown, Fig.10 A schematic flow chart of an optional food identification method provided by the present application is shown, wherein the object in the image includes multiple food ingredients; and according to the food outlines of the multiple food ingredients, the blocked food ingredients in the image are determined, including:

[0182] S401, determining overlapping portions between contours of a plurality of ingredients;

[0183] S402, determining at least two ingredients that obstruct each other among the plurality of ingredients according to the overlapping parts;

[0184] S403: Determine a second target food among the at least two mutually obstructing food as an obstructed food, wherein the food outline of the second target food is discontinuous or has a gap.

[0185] In this implementation, if the object covered by the image includes multiple ingredients, that is, the content in the image is relatively complex, multiple ingredients are arranged and stacked in the storage area of ​​the refrigerator, thus forming various possible occlusion relationships. For example, in the cold storage area of ​​the refrigerator, there may be apples, bananas, vegetables and other ingredients at the same time, and the placement and method of multiple ingredients are random, which creates conditions for mutual occlusion between ingredients.

[0186] In the example of the present application, the overlapping parts between the food outlines of the multiple food ingredients are determined by analyzing the food outlines of the multiple food ingredients. For example, the boundary detection and coordinate calculation of each food outline are performed to determine whether the different food outlines intersect in space. For another example, the edge outline of each food ingredient can be determined by an edge detection algorithm (such as a Canny edge detection algorithm), and then the coordinate range and intersection of each food outline are calculated to determine whether there are overlapping parts between different food outlines.

[0187] Since an occlusion relationship may exist in reality only when two ingredients are in the overlapping part between the outlines of the ingredients in the image, it should be understood that such an overlap may be a partial overlap, or it may be a situation where one ingredient is mostly occluded by another ingredient.

[0188] After the overlapping parts between the contours of multiple ingredients are determined, these overlapping parts are further analyzed, and factors such as the overall shape, relative position, and degree of overlap of the contours of the ingredients are combined to determine which of the multiple ingredients are mutually obstructed. For example, if a circular contour of an ingredient (such as an orange) and a rectangular contour of an ingredient (such as boxed milk) partially overlap, and a portion of the circular contour of the ingredient cannot be fully displayed, that is, it is determined that a portion of the circular contour of the ingredient is obstructed behind the rectangular contour of the ingredient, then it is determined that there is mutual obstruction between the orange and the boxed milk.

[0189] In another example, for the complex situation where multiple ingredients overlap, you can first analyze each pair of overlapping ingredients to determine all mutually occluded ingredient combinations in the entire image. After determining at least two mutually occluded ingredients, select a second target ingredient (with a discontinuous outline or a gap) from these mutually occluded ingredients as the occluded ingredient. This is because when an ingredient is obscured by another ingredient, its outline will be interrupted or missing at the obscured location. For example, among two mutually occluded fruits, if the outline of one of the fruits has an obvious depression or truncated portion, while the outline of the other fruit is relatively complete, the fruit with the outline problem is determined to be the second target ingredient, i.e., the occluded ingredient.

[0190] Through the above optional method, based on the actual presentation of the ingredients in the image, the obscured ingredients can be accurately found through the analysis of contour features, providing accurate objects for subsequent operations such as occlusion removal and ingredient information recognition, which helps to improve the accuracy and reliability of ingredient recognition.

[0191] In a possible implementation, the blocked food in the image is subjected to blockage removal processing and missing part completion processing to obtain a processed image, including:

[0192] Calibrate the occluded food in the image to obtain a calibrated image;

[0193] The calibrated image is input into a pre-trained occlusion removal model, so that the occlusion removal model is used to perform occlusion removal processing and missing part completion processing on the occluded food in the calibrated image to obtain a processed image.

[0194] Optionally, in the above example, the occlusion removal model is obtained by iteratively training a generative adversarial network model using multiple groups of training samples in advance, and each group of training samples includes: a first sample image with calibrated occlusions, and a second sample image corresponding to the first sample image with the missing part completed.

[0195] In the above implementation, the occluded food in the image is first calibrated for occlusion, mainly to clarify which parts of the image are occlusions that need to be processed, and the positions of the corresponding occluded food, so that the occlusion removal model knows accurately which areas to process.

[0196] Specifically, in this application example, some image processing techniques and tools can be used to implement the calibration operation. For example, image annotation software can be used or a code can be used in a programming environment to mark a specific area. In this application example, the occluder is calibrated in a visual way, such as using a marker point, a rectangular box, a polygonal box, or a marker of a specific color, so as to obtain a calibrated image. Figure 3 As shown, by using the marker points to calibrate the occluder (i.e. corresponding to Figure 2 The occluded food A) in the image is removed to obtain the calibrated image.

[0197] In the example of this application, the occlusion removal model is built based on the generative adversarial network (GAN) model. The GAN model consists of two parts: the generator and the discriminator, which continuously optimizes the model performance through mutual confrontation. Specifically, the task of the generator is to generate new images or data based on the input, making it as close to the real data as possible; the discriminator is responsible for judging whether the input data is real or generated by the generator, and through this adversarial mechanism, the generator is prompted to generate better quality images.

[0198] Moreover, in the example of the present application, before the occluder removal model is specifically applied to perform occluder removal processing and missing part completion processing, the initial GAN-based model is iteratively trained through multiple groups of training samples to obtain the occluder removal model. Each group of training samples contains two parts: a first sample image with occluders calibrated, and a second sample image corresponding to the first sample image with the missing part completed. During the training process, the first sample image is input into the initial GAN-based model, and the generator attempts to generate an image with occluders removed and missing parts completed based on the input image with occluders, and then the discriminator judges the image generated by the generator to determine whether it is similar to the corresponding second sample image (i.e., the real, completed image). By continuously iterating this process, the model adjusts the parameters of the generator according to the feedback of the discriminator, so that the image generated by the generator is closer and closer to the real completed image, thereby gradually optimizing the performance of the model so that it can accurately perform occluder removal processing and missing part completion processing on the newly input calibrated image.

[0199] After the calibrated image obtained through occlusion calibration is input into the pre-trained occlusion removal model, the occlusion removal model performs occlusion removal processing and missing part completion processing on the occluded food in the calibrated image according to the knowledge and capabilities learned during the training process. Figure 4a and Figure 4b As shown, the occluders in the calibration area are identified (i.e., corresponding to Figure 2 The occluded food A in the image is removed step by step, and the occluded food (i.e. the corresponding Figure 2 The known part of the occluded food B) in the image and the surrounding environment are used to complete the missing part of the occluded food, and finally the image is still the same. Figure 5 The processed image shown. Compared with the original image, the occluded food has its occluders removed and the missing parts are completed, making the occluded food more complete in the image, so as to accurately identify and obtain the food information of the occluded food.

Claims

1. A refrigeration device, characterized in that: include: A box body, wherein the box body is provided with a plurality of storage areas for accommodating food; An image acquisition device disposed in at least one of the storage areas; And, a control device, connected to the image acquisition device, is configured to: In response to the triggering of the image acquisition condition, an image in the storage area is acquired based on the image acquisition device, and the objects in the image include: one or more food ingredients; Determining the obscured food in the image according to the food outline of the food in the image; Performing an obstruction removal process and a missing part completion process on the obstructed food in the image to obtain a processed image; Identify the food information of the obscured food in the processed image.

2. The refrigeration equipment according to claim 1, characterized in that: The control device performs the step of determining the obscured food in the image according to the food outline of the food in the image, and is configured to: Using an image recognition algorithm to perform integrity analysis on the contour of the food in the image, the integrity analysis is used to analyze whether the contour of the food is discontinuous or has gaps; If the integrity analysis result indicates that the first target food exists in the image, it is determined that the first target food is the blocked food, and the food outline of the first target food is discontinuous or has a gap.

3. The refrigeration equipment according to claim 1, characterized in that: The object in the image includes a plurality of food ingredients; The control device performs the step of determining the blocked food in the image according to the food outlines of the plurality of food ingredients, and is configured to: determining overlapping portions between the contours of the plurality of ingredients; Determining, according to the overlapping portion, at least two ingredients among the plurality of ingredients that obstruct each other; A second target food among the at least two foods that obstruct each other is determined as the obstructed food, and a food outline of the second target food is discontinuous or has a gap.

4. The refrigeration equipment according to claim 1, characterized in that: The control device performs an obstruction removal process and a missing part completion process on the obstructed food in the image to obtain a processed image, and is configured to: Performing occluder calibration on the occluded food in the image to obtain a calibrated image; Inputting the calibrated image into a pre-trained occlusion removal model, so as to use the occlusion removal model to perform occlusion removal processing and missing part completion processing on the occluded food in the calibrated image, thereby obtaining the processed image; The occlusion removal model is obtained by iteratively training a generative adversarial network model using multiple groups of training samples in advance, and each group of training samples includes: a first sample image with calibrated occlusions, and a second sample image corresponding to the first sample image with the missing part completed.

5. The refrigeration equipment according to claim 1, characterized in that: The control device performs an obstruction removal process and a missing part completion process on the obstructed food in the image to obtain a processed image, and is configured to: Performing occluder calibration on the occluded food in the image to obtain a calibrated image; Based on the image and / or the model of the refrigeration device, determining the inner tank color of at least one of the storage areas; Inputting the liner color and the calibrated image into a pre-trained occlusion removal model, so as to use the occlusion removal model to perform occlusion removal processing and missing part completion processing on the occluded food in the calibrated image based on the liner color, to obtain the processed image; The occlusion removal model is obtained by iteratively training a generative adversarial network model using multiple groups of training samples in advance, and each group of training samples includes: a first sample image with calibrated occlusions, the sample liner color as a constraint condition, and a second sample image corresponding to the first sample image with the missing part completed.

6. The refrigeration device according to claim 1, characterized in that: The refrigeration equipment further comprises: a lamp assembly disposed in at least one of the storage areas, connected to the control device, and configured to: in response to the triggering of the image acquisition condition, turn on the lighting mode under the drive of the control device; The image acquisition device is configured to: when the lamp tube assembly operates in the lighting mode, under the control of the control device, photograph the storage area to obtain the image.

7. The refrigeration device according to claim 6, characterized in that: The control device is further configured to: In the process of controlling the image acquisition device to take pictures of the storage area, the lighting parameters of the lamp tube assembly are adjusted.

8. The refrigeration device according to claim 7, characterized in that: The control device adjusts the lighting parameters of the lamp assembly during the process of controlling the image acquisition device to take pictures of the storage area, and is further configured to: In the process of controlling the image acquisition device to take pictures of the storage area, characteristic information of the food in the storage area and internal environment information of the storage area are obtained, wherein the characteristic information is used to describe at least one of the following information of the food: color, transparency, and glossiness, and the internal environment information is used to describe at least one of the following information of the storage area: liner color, material, and spatial layout; Adjust the lighting parameters of the lamp assembly according to the characteristic information and the internal environment information.

9. The refrigeration device according to claim 7, characterized in that: The control device adjusts the lighting parameters of the lamp assembly during the process of controlling the image acquisition device to take pictures of the storage area, and is further configured to: In the process of controlling the image acquisition device to take pictures of the storage area, obtaining characteristic information of the food in the storage area and shooting information of the image acquisition device, wherein the characteristic information is used to describe at least one of the following information of the food: color, transparency, and glossiness, and the shooting information is used to describe at least one of the following information of the image acquisition device: shooting angle and shooting distance; Adjust the lighting parameters of the lamp assembly according to the feature information and the shooting information.

10. The refrigeration equipment according to claim 7, characterized in that: The control device adjusts the lighting parameters of the lamp assembly during the process of controlling the image acquisition device to take pictures of the storage area, and is further configured to: In the process of controlling the image acquisition device to take pictures of the storage area, obtaining characteristic information of the food in the storage area, shooting information of the image acquisition device and internal environment information of the storage area, wherein the characteristic information is used to describe at least one of the following information of the food: color, transparency, glossiness, the shooting information is used to describe at least one of the following information of the image acquisition device: shooting angle, shooting distance, and the internal environment information is used to describe at least one of the following information of the storage area: liner color, material, and spatial layout; The lighting parameters of the lamp assembly are adjusted according to the feature information, the shooting information and the internal environment information.

11. The refrigeration device according to any one of claims 1 to 10, characterized in that: The image acquisition conditions include at least one of the following: detecting a position change of one or more food ingredients in the storage area; detecting that one or more new food ingredients are added to the storage area; detecting that one or more food materials in the storage area are taken out; It is detected that the door of the storage area has executed a closing action.

12. A method for identifying food materials, characterized in that: include: In response to the triggering of the image acquisition condition, an image in the storage area is acquired based on an image acquisition device, and the objects in the image include: one or more food ingredients; Determining the obscured food in the image according to the food outline of the food in the image; Performing an obstruction removal process and a missing part completion process on the obstructed food in the image to obtain a processed image; Identify the food information of the obscured food in the processed image.

13. The method according to claim 12, characterized in that The step of determining the obscured food in the image according to the food outline of the food in the image comprises: Using an image recognition algorithm to perform integrity analysis on the contour of the food in the image, the integrity analysis is used to analyze whether the contour of the food is discontinuous or has gaps; If the integrity analysis result indicates that the first target food exists in the image, it is determined that the first target food is the blocked food, and the food outline of the first target food is discontinuous or has a gap.

14. The method according to claim 12, characterized in that The object in the image includes a plurality of ingredients; and determining the obscured ingredients in the image according to the contours of the plurality of ingredients comprises: determining overlapping portions between the contours of the plurality of ingredients; Determining, according to the overlapping portion, at least two ingredients among the plurality of ingredients that obstruct each other; A second target food among the at least two foods that obstruct each other is determined as the obstructed food, and a food outline of the second target food is discontinuous or has a gap.

15. The method according to any one of claims 12 to 14, characterized in that Performing an obstruction removal process and a missing part completion process on the obstructed food in the image to obtain a processed image, including: Performing occluder calibration on the occluded food in the image to obtain a calibrated image; Inputting the calibrated image into a pre-trained occlusion removal model, so as to use the occlusion removal model to perform occlusion removal processing and missing part completion processing on the occluded food in the calibrated image, thereby obtaining the processed image; The occlusion removal model is obtained by iteratively training a generative adversarial network model using multiple groups of training samples in advance, and each group of training samples includes: a first sample image with calibrated occlusions, and a second sample image corresponding to the first sample image with the missing part completed.

16. The method according to any one of claims 12 to 14, characterized in that Performing an obstruction removal process and a missing part completion process on the obstructed food in the image to obtain a processed image, including: Performing occluder calibration on the occluded food in the image to obtain a calibrated image; Based on the image and / or the model of the refrigeration device, determining the inner tank color of at least one of the storage areas; Inputting the liner color and the calibrated image into a pre-trained occlusion removal model, so as to use the occlusion removal model to perform occlusion removal processing and missing part completion processing on the occluded food in the calibrated image based on the liner color, to obtain the processed image; The occlusion removal model is obtained by iteratively training a generative adversarial network model using multiple groups of training samples in advance, and each group of training samples includes: a first sample image with calibrated occlusions, the sample liner color as a constraint condition, and a second sample image corresponding to the first sample image with the missing part completed.

Images